An app has no way to attach content to the message chat_server() sends. The module reads the input value and hands it straight to the client, so the only thing that reaches stream_async() is what the user typed.
Two common needs run into this. A RAG app searches its own documents for each question and wants the matching excerpts sent in the same turn as the question that produced them. An app whose state the model is acting on — a document, a dashboard, a canvas being edited — wants each message to carry that state as it stood when the message was sent.
The system prompt is not a substitute
A conversation has exactly one system prompt, and it is mutable: ellmer::Chat$set_system_prompt() drops the existing system turn and prepends a replacement. Rewriting it between messages therefore does not add context to the next turn — it retroactively replaces the context of every turn already in the history.
The result is a transcript that describes a world the prompt no longer shows. If the assistant said "I've added a summary section" at turn 3 and that section is gone by turn 10, the model sees both the claim and a prompt describing a document without it, and has nothing to reconcile them with. A RAG app hits the same wall from the other side: each question's excerpts overwrite the previous question's, so the model can no longer see the text it just answered from, and per-question evidence ends up sharing a slot with the app's standing instructions.
Content carried in the turn does not have either problem. Each message keeps the context it was sent with, and the history stays a truthful record.
There is a cost argument as well, though it is the smaller one. Prompt caching is a prefix match and system renders ahead of messages, so a system prompt rewritten between messages invalidates the whole conversation behind it. For Anthropic, ellmer marks the system prompt with cache_control and defaults to cache = "5m", so the re-send is billed at the cache-write rate — making a dynamic system prompt worse than no caching at all.
Why an app on chat_server() cannot do this
The module owns the send:
shiny::observeEvent(
session$input[[paste0(id, "_user_input")]],
label = "on_chat_user_input",
{
last_input(session$input[[paste0(id, "_user_input")]])
append_stream_task$invoke(
client, id, session$input[[paste0(id, "_user_input")]], controller = ctrl
)
}
)That is pkg-r/R/chat_app.R#L427. An app can observe the value afterwards through last_input(), but by then it has been sent. There is no way in from outside either: ellmer::Chat is unexported so the client cannot be subclassed, its instances are binding-locked (assigning to ch$stream_async fails with cannot change value of locked binding), and the returned module environment is sealed with lockEnvironment(ret).
Why this is not #10 again
Issue #10 asked for both input and response transforms and was closed as completed on a worked example: own the observer, change the text, then call chat$stream_async() yourself. That was correct for the API as it stood.
It is not available here. The chat_server() module is new (#264), it owns the stream_async() call, and it is where multi-conversation history, attachments and slash commands now live — so taking the #10 route today means giving up the whole module to gain one line.
The scepticism recorded in #10 was about the response side, where streaming, tool-call UI and bookmarking all pull against arbitrary transformation. None of that applies outbound, and this request leaves the response direction alone.
Proposed shape
A submit callback, registered the way mod$history$on_save() already is:
mod <- chat_server("chat", client)
mod$on_submit(function(contents) {
c(list(retrieved_context()), contents)
})The function would take and return the value in the shape input$<id>_user_input already has — a character string, or a list of ellmer::Content objects when attachments are enabled — and apply only to what is sent, leaving last_input() reporting what the user typed.
The contract would not be a new one for the package. A slash-command handler already receives a ContentSlashCommand and is expected to shape it before passing it to client$stream(); this asks for the same latitude on the path where the module does the sending.
Happy to open a PR if the shape looks right.
An app has no way to attach content to the message
chat_server()sends. The module reads the input value and hands it straight to the client, so the only thing that reachesstream_async()is what the user typed.Two common needs run into this. A RAG app searches its own documents for each question and wants the matching excerpts sent in the same turn as the question that produced them. An app whose state the model is acting on — a document, a dashboard, a canvas being edited — wants each message to carry that state as it stood when the message was sent.
The system prompt is not a substitute
A conversation has exactly one system prompt, and it is mutable:
ellmer::Chat$set_system_prompt()drops the existing system turn and prepends a replacement. Rewriting it between messages therefore does not add context to the next turn — it retroactively replaces the context of every turn already in the history.The result is a transcript that describes a world the prompt no longer shows. If the assistant said "I've added a summary section" at turn 3 and that section is gone by turn 10, the model sees both the claim and a prompt describing a document without it, and has nothing to reconcile them with. A RAG app hits the same wall from the other side: each question's excerpts overwrite the previous question's, so the model can no longer see the text it just answered from, and per-question evidence ends up sharing a slot with the app's standing instructions.
Content carried in the turn does not have either problem. Each message keeps the context it was sent with, and the history stays a truthful record.
There is a cost argument as well, though it is the smaller one. Prompt caching is a prefix match and
systemrenders ahead ofmessages, so a system prompt rewritten between messages invalidates the whole conversation behind it. For Anthropic,ellmermarks the system prompt withcache_controland defaults tocache = "5m", so the re-send is billed at the cache-write rate — making a dynamic system prompt worse than no caching at all.Why an app on
chat_server()cannot do thisThe module owns the send:
That is
pkg-r/R/chat_app.R#L427. An app can observe the value afterwards throughlast_input(), but by then it has been sent. There is no way in from outside either:ellmer::Chatis unexported so the client cannot be subclassed, its instances are binding-locked (assigning toch$stream_asyncfails withcannot change value of locked binding), and the returned module environment is sealed withlockEnvironment(ret).Why this is not #10 again
Issue #10 asked for both input and response transforms and was closed as completed on a worked example: own the observer, change the text, then call
chat$stream_async()yourself. That was correct for the API as it stood.It is not available here. The
chat_server()module is new (#264), it owns thestream_async()call, and it is where multi-conversation history, attachments and slash commands now live — so taking the #10 route today means giving up the whole module to gain one line.The scepticism recorded in #10 was about the response side, where streaming, tool-call UI and bookmarking all pull against arbitrary transformation. None of that applies outbound, and this request leaves the response direction alone.
Proposed shape
A submit callback, registered the way
mod$history$on_save()already is:The function would take and return the value in the shape
input$<id>_user_inputalready has — a character string, or a list ofellmer::Contentobjects when attachments are enabled — and apply only to what is sent, leavinglast_input()reporting what the user typed.The contract would not be a new one for the package. A slash-command handler already receives a
ContentSlashCommandand is expected to shape it before passing it toclient$stream(); this asks for the same latitude on the path where the module does the sending.Happy to open a PR if the shape looks right.