Stateful vs stateless LLMs: why keeping context on the GPU changes the game.

Introduction

Large Language Models have transformed how we interact with technology. What has not changed is how most runtimes treat context: as something to be rebuilt from scratch on every single call.

Limits of stateless

A stateless runtime re-sends the entire conversation on every turn. The cost is not theoretical. It is paid in tokens, in time-to-first-token, and in the ceiling it puts on any agentic workload that needs to run for more than a handful of steps.

What stateful actually means

Keeping the KV cache resident on the GPU between turns. The model itself is unchanged and remains stateless; the runtime around it simply stops throwing away work it has already done.

Conclusion

Statefulness is a property of the runtime, not the model. It is also the property that decides whether agent workloads are economically viable at scale.