Stateful vs stateless LLMs: why keeping context on the GPU changes the game.

Introduction

Limits of stateless

What stateful actually means

Keeping the KV cache resident on the GPU between turns. The model itself is unchanged and remains stateless; the runtime around it simply stops throwing away work it has already done.

Conclusion

Statefulness is a property of the runtime, not the model. It is also the property that decides whether agent workloads are economically viable at scale.