The AI feature works in the demo. Now make it survive production

A model call that works on your machine is perhaps a fifth of an AI feature. The rest is everything that happens when the call is slow, when the answer is wrong, when the key is missing, when a thousand people arrive at once, and when someone types something adversarial into your input box.

Decide what happens when it is unavailable

Before writing the integration, answer this: if the model cannot be reached, what does the user see? There are only three honest answers — the feature degrades to something simpler, the feature disappears cleanly, or the page breaks. The third is the default if you do not choose, and it is the one that takes the rest of the page down with it.

Degrading well is usually worth the effort. A support assistant that falls back to a small set of reliable answers and says plainly that it is limited right now is far better than a spinner that never resolves.

Stream, and set a timeout you have tested

Streaming is not only a nicety. A long non-streamed generation will eventually hit a platform request timeout, and the failure is opaque. Streaming gets tokens moving immediately, which both feels faster and keeps the connection legitimately alive.

Separately: put a timeout on anything that can hang, and test it by actually making it hang. Promises that never settle are a specific and nasty category of bug, because nothing errors — the button simply stays in its loading state forever and the user assumes your product is broken. They are right.

A public endpoint that calls a model is a spending endpoint

The moment an unauthenticated page can trigger a model call, you have published a way for strangers to spend your money. This is not hypothetical and it does not require malice — a badly behaved script is enough.

• Cap input length and conversation history server-side. The client cannot be trusted to do it.
• Rate limit per IP. An in-memory limiter slows down casual abuse; a shared one (Redis, KV) is what holds under real traffic across multiple instances.
• Set a spend cap on the API key itself, as the backstop for everything you did not anticipate.
• Abort in-flight requests when the user navigates away. You are billed for tokens nobody will read.

Treat the user's text as data

Anything a visitor types will at some point contain instructions aimed at your system prompt. Constrain the model's scope explicitly, never give it authority it does not need, and assume that anything it can do, a determined visitor can eventually make it do. The strongest mitigation is not clever prompt wording — it is not granting the capability in the first place.

None of this is exotic. It is the ordinary discipline of shipping a network call that is slow, expensive and occasionally wrong — applied to a dependency that happens to be a model.