What Is a System Prompt?
Instructions set once at the top of a conversation, in a structurally distinct slot the model was trained to weight differently.
On this page
A conversation sent to a model is a list of messages, each tagged with a role. The system message comes first and frames everything after it.
system: You are a technical support assistant for a database product.
Answer only from the documentation provided. If the answer is
not there, say so and offer to escalate.
user: Why is my query slow?The same text pasted into the user message would behave differently, and the reason is worth understanding.
Why the slot matters
During instruction tuning, models are trained on conversations formatted with these role markers. They learn that system-role content sets persistent constraints while user-role content is the immediate request.
So the difference is not that system messages are magically privileged. It is that the model was trained on a great many examples where system content functioned as standing instruction, and it learned that pattern. The slot carries weight because of training, not because of enforcement.
That distinction matters practically: system instructions are strong tendencies, not guarantees. A sufficiently determined user message can pull against them. Treat the system prompt as a well-respected default, never as a security boundary.
What belongs there
Role and scope. What this assistant is for, and what it should decline.
Output conventions that apply every turn. Format, tone, length, language.
Persistent constraints. “Never recommend a competitor.” “Always include a source.” “Assume the reader is non-technical.”
Stable context. Product terminology, the current date, relevant policies.
What does not belong there
The current question. That is the user message.
Anything that changes per request. Retrieved documents, the specific task, variable input. These go in user messages — partly for clarity, and partly for a caching reason below.
Secrets. A system prompt is not hidden. It can frequently be extracted through determined questioning, and it should be assumed visible. Never put credentials or genuinely confidential instructions there.
The caching consequence
Because the system prompt sits at the very beginning of every request, it is the ideal candidate for prefix caching.
A stable system prompt means its keys and values can be computed once and reused across requests, which meaningfully reduces cost and latency. Change it per request and you forfeit that entirely.
This gives a concrete rule: keep the system prompt fixed, put everything variable in user messages. It is one of the cheapest cost optimizations available. See Cutting Your API Bill.
Length and placement effects
Two failure modes show up with long system prompts.
Instructions in the middle of a long system prompt get followed less reliably than instructions at either end — the same attention pattern that affects long contexts generally. Put the most important constraint first or last.
And in long conversations, system instructions compete with a growing transcript. Ten thousand tokens of conversation can dilute a short system prompt. Restating a critical constraint in the latest user message is a legitimate fix.
Multi-turn behavior
The system prompt is sent on every request, since the model has no memory between calls. Whatever framework you use is re-transmitting it each turn.
You can change it mid-conversation — nothing prevents it — but it invalidates the cached prefix from that point, and it can produce inconsistency since earlier turns were generated under different instructions.
What to remember
- The system prompt is a distinct role at the start of the message list, framing all subsequent turns.
- Its authority comes from training patterns, not enforcement — a strong default, not a security boundary.
- Put role, scope, format conventions, and stable context there; keep the task and variable input in user messages.
- Keeping it fixed enables prefix caching, which is a real cost saving.
- Middle-of-prompt instructions are followed less reliably, and long conversations dilute short system prompts.