Stop Paying the Model to Say the Same Thing Ten Times: LLM Output Caching for Vibe Projects
LLM output caching stores answers the model already computed, so repeated questions get a stored answer instead of a fresh generation. This guide covers when caching pays off, exact vs. semantic caching, cache keys, hit-rate and cost math, expiry, privacy red lines, and multi-model strategy. All example products and figures are fictional.

Once a vibe project plugs into a large language model, the most expensive thing is rarely the hardest question — it is the repeated question. A user asks today how to add an internship to a resume, then asks the same thing tomorrow in different words; ten people in a study group take turns asking for the solution to the same problem, and the model has to generate the same passage ten times. Every generation costs tokens and takes seconds, and the bill and the waiting time creep up bit by bit. LLM output caching stores an answer the model has already computed, so the next time the same question — or one that means the same thing — comes in, the stored answer is served directly instead of computing it again.
Every example below is fictional: the product is "Qingning Assistant", an AI Q&A helper for study groups, and "A'nan" is its independent developer. All request counts, hit rates, and money saved are invented examples to explain the method, not real operating data from any product. Boundaries first: browser caching, HTTP cache headers, CDNs, and general server-side caching belong to our guide on caching strategy; per-token pricing, model selection, and cutting the cost of a single call belong to our guide on LLM inference cost engineering. This guide covers only one thing, the LLM output layer: whether to store an answer once it is computed, how to store it, when it must not be stored, and how to recover when storage goes wrong. It does not cover general HTTP or CDN caching, and it does not cover model pricing itself.

1. When caching is worth it: look at repetition first, then at cost
Caching is not something to build on the first day a model is connected. Let the product run for two weeks, then look at the request logs for three things: first, how many times the same question has appeared; second, how many tokens and how many seconds each answer costs; third, how much it costs when an answer is wrong. After two weeks, Qingning Assistant found that its twenty most-asked questions accounted for 40% of all requests, each answer cost on average more than 800 tokens and over two seconds, and the answers barely changed — that is the signal caching is worth it. The other way around, if every question is asked only once, a cache will never hit, and building one just adds a system to maintain for nothing.
Whether it is worth it also depends on whether the answer changes. Exam syllabuses, class timetables, and group rules stay the same for months once written, so caching them pays best; news, prices, and live match scores go stale within minutes, and caching them means serving expired answers. There is also a kind of answer that looks stable but hides change: "has this year's math exam syllabus changed?" is correct until the syllabus changes, and the moment it does, the cached old answer becomes misleading. The test is simple: an answer can live in the cache as long as its shelf life lasts, and if the shelf life cannot be stated, do not cache the question yet.
The third thing to check is the cost of being wrong. For study methods and problem-solving approaches, even a slightly stale cached answer is usually spotted by the user, who just asks a follow-up; but for registration deadlines, exam times, and payment methods, one wrong answer can derail something real. High-stakes answers like these should either not be cached at all, or cached for a very short time and recomputed the moment they expire. The first rule of caching is this: the money saved must outweigh the risk of serving a wrong answer, or it is better to ask the model every time.
2. Exact caching vs. semantic caching: do the cheap one first, then the clever one
There are two ways to cache LLM output, and they need to be told apart. Exact caching only recognises a question that is word-for-word identical: only when the user's input matches character for character does it count as a hit. Its advantage is simplicity and reliability — it never mistakes one question for another. What is stored under a question is looked up under that same question, so the answer is always the one the model gave at the time. The first version of Qingning Assistant's cache was exact caching and did only one thing: normalise the question first — trimming extra spaces, unifying case, dropping the trailing question mark and politeness words like "please tell me" — and only questions identical after normalisation counted as hits. In its first week, that version caught more than 30% of repeated questions.
Semantic caching recognises questions that mean the same thing but are worded differently. "How can I read English faster?" and "My English reading is too slow, what should I do?" use different words but mean the same thing, and semantic caching treats them as one question. The method is to turn the question into a vector, measure how similar the new question is to stored ones, and if the similarity passes a threshold, treat it as the same question and return the stored answer. The benefit is a higher hit rate; the danger is misrecognition. Two questions can be close in meaning but need different answers — "how should I revise for the math exam" and "how should I revise for the English exam" score very similarly yet have completely different answers, and one misrecognition means answering the wrong question.
Do not reverse the order: ship exact caching first, and only consider semantic caching once it runs smoothly. Exact caching carries no risk of misrecognition, and when something goes wrong it is easy to trace — the hit log records which question hit, and a wrong entry can be deleted on its own. Semantic caching needs three extra things before it is safe to ship: a similarity threshold, a correction entry point for misrecognitions, and a trial period. Do not set the threshold by gut feel; first run two weeks of real questions through it and see at what threshold the misrecognition rate becomes acceptable. Put the correction entry next to the answer so that when a user taps "this answer is wrong", that cache entry is voided. Run the trial for only a small share of users, and roll it out fully only after the misrecognition rate is confirmed low enough. For many small projects, exact caching is enough; semantic caching is the step to take only when the hit rate cannot climb any further.
3. Designing cache keys: put the question, the model, and the version in one key
The key to a cache entry is called the cache key, and a wrong design makes the cache serve the wrong person the wrong answer. The easiest mistake is using only the question as the key: the same question answered today by a cheap small model and tomorrow by a large model gives answers of different quality; the same question answered in Chinese or in English gives different answers; the same question with different background material in the prompt gives a different answer again. The key must contain all of these: the normalised question, the model name, the prompt template version, and the answer language. Only those four together make a key that cannot open the wrong door.
Should the key also record "who the answer is for"? It depends. For public knowledge Q&A in a study group, the answer is the same for everyone, so the key needs no user in it; but for a question like "give me three problems based on my mistake log", the answer is personalised, so the key must carry a user identifier — otherwise Zhang San's mistake-log answer gets sent to Li Si. The safer rule is: any answer that contains a user's personal data never goes into the public cache at all. It goes only into that user's own cache, or it is not cached at all. The first principle of key design is this: it is better to store a few extra copies and accept a lower hit rate than to let one person receive another person's answer.
Keys must also be deletable, searchable, and expirable. Every key should be readable: open the cache list and see from the key which question, which model, and which prompt version it belongs to. When debugging, given a wrong answer, you must be able to trace it back to its key, when it was stored, and how long it has left before expiring. Never put a user's raw input in full into a key — especially never a user's sensitive content as typed. Keys get written into logs, and sensitive content in a key means sensitive content in a log. The fix is to use a hash of the normalised question as the key instead of the original text; a hash cannot be reversed, so the log holds only an unreadable string.
4. Hit rate and cost math: work through one fictional set of numbers
Whether a cache is doing well comes down to two numbers: the hit rate and the money saved. The hit rate is the share of all requests the cache serves; the money saved is what those served requests would have cost if they had gone to the model again. Qingning Assistant worked through one fictional set of numbers: in one month, out of 10,000 requests, exact caching served 3,800, semantic caching caught another 2,400, and only the remaining 3,800 actually went to the model. The hit rate is 6,200 divided by 10,000 — 62%. At an average of ¥0.12 per model call, the 6,200 hits saved about ¥744. These numbers are invented, to show how the math is done; they are not real operating data.
When doing the math, do not count only the token money saved — count the cache's own cost too. A vector database, a cache server, and similarity computation each cost money and need maintenance. If a month saves ¥700-odd but the cache service itself costs ¥600 a month, the net saving is only about ¥100, and once maintenance time is counted, it may be better not to build it at all. Qingning Assistant's rule is: the cache's total cost must stay under 30% of the money saved, or it is not worth continuing; past that line, narrow the cache first — cache only the twenty most-asked questions rather than everything.
The hit rate is not a case of "the higher the better" either. A hit rate climbing above 90% usually means the threshold was set too loose and misrecognitions are creeping in, or the cache lifetime was set too long and expired answers are circulating. Qingning Assistant checks hit rate and correction rate once a week: a hit rate steady between 50% and 70% with fewer than 2% of users tapping "this answer is wrong" is healthy. If the hit rate suddenly jumps, check first whether someone loosened the threshold; if the correction rate climbs, tighten the threshold and clear suspicious entries. Numbers exist to manage the cache, not to show off.
5. Expiry: know an answer is stale before the user does
The most dangerous moment for a cache is not the day it is built, but the moment an answer quietly goes stale. An expiry policy gives every cache entry a death date: once the time is up, the old answer is void whether or not anyone asks, and the next question is computed again. How long the death date is depends on the answer's shelf life: stable answers such as group rules and study methods can live thirty days; timetables and exam times that change can live seven days; registration and payment information that can change at any time lives at most a day, and important items only a few hours. Set the death date a little short rather than letting a user receive an expired answer.
A death date alone is not enough — there must also be active invalidation. When a timetable changes, a syllabus is revised, or a rule is updated, the related cache entries must be cleared immediately instead of waiting for natural expiry. The method is to tag cache entries: each entry records which kind of question it answers and which document it is based on; when a document is updated, every entry tagged with it is voided at once. Qingning Assistant does it this way: when an admin changes the class timetable once, the system automatically clears every cache entry tagged "timetable", and the next time a user asks, the answer is computed from the new timetable.
Expiry also needs a fallback: after a cache entry is deleted, the next request has to ask the model again, and if the model happens to be failing, timing out, or erroring at that moment, the user sees a blank page. The safe approach is to compute the new answer first and replace the old one only after confirming the new one is fine; if the replacement fails, keep serving the old answer a little longer and add a line under it — "this answer is from a few days ago, please go by the latest notice" — rather than showing the user an error. The goal of an expiry policy is not a cache that is fresh forever, but one where stale answers are always found by the system first, never by the user.
6. The privacy red line: five kinds of prompt that never enter the cache
The biggest risk of caching is not technical, it is privacy. Once a model's answer enters the cache, another person may retrieve it; once cache keys and logs are recorded, the wrong person may see them. So some prompts should never enter the cache at all — this is a red line, not an optimisation. First, prompts containing a name, phone number, ID number, or address. Second, prompts containing grades, mistake logs, or other study records. Third, private documents, diaries, or chat logs pasted in by the user as-is. Fourth, questions about health, finances, or family circumstances. Fifth, questions from admin operations carrying internal data. These five kinds are never cached, no matter how often they repeat or how much money caching them would save.
Red lines must be guarded by mechanisms, not by good intentions. Qingning Assistant guards them in three layers: the first is entry filtering — before a request enters the cache, it passes a rules check, and anything matching sensitive words, a phone-number pattern, or an ID-number pattern skips the cache entirely. The second is separate buckets — public questions go into a public cache, questions carrying a user identifier go only into that user's own cache, and the two buckets are physically separate and never mixed. The third is regular sweeping — the cache is scanned on a schedule, and anything that looks like sensitive content is deleted and logged. If any layer finds something suspicious, the default action is not to cache, never to cache first and ask later.
Two things are easy to overlook. First, the cache service itself needs access control: only the application server may read and write it; a developer's laptop and the test environment must never connect directly to the production cache. Second, cache backups and logs must be treated as sensitive data too: backups must not contain users' private questions, and logs must not contain users' original text. Caching saves model money; a leak costs user trust, and a few hundred yuan saved will never cover that bill.
7. Caching with multiple models: one shared pool, or one cache each
With a single model, caching is simple; once two or more models are in use, questions arise: should an answer from the cheap model be recomputed by the large model? Can an answer from the large model be shown to a user on the cheap model? Qingning Assistant's answer is: the cache pool can be shared, but the key must name the model, and the answer must state its source. The same question answered by the small model and by the large model lives under different keys and never interferes; when a user asks, the cache is checked first under the model they are currently using, and only a miss goes to the model. An answer from the cheap model is never passed off as a large-model answer, and the answer states which model gave it, so the user can judge whether it is good enough.
When a model is upgraded, the cache must move with it. A new model's answers may differ in style and depth, so cache entries built up by the old model cannot simply carry over — serving old answers directly would make users feel the answer got worse after the upgrade. The fix is to put the model version in the key: when the model is upgraded, old keys naturally expire. In the first week after launch, let the new model run exact caching only, and only open up semantic caching gradually once the new answers' quality is confirmed. Downgrading works the same way: switching from a large model back to a small one means the cache is counted separately — never use a large-model answer to pad a small model's numbers.
Multiple models also offer a money-saving trick: check the cache first, then pick the model. A user's question goes through the cache first and is returned directly on a hit; only on a miss is a model picked by difficulty — easy questions go to the cheap model, hard ones to the large model. Do not reverse the order: picking the model first and then checking the cache means checking once per model, and the hit rate is split apart for nothing. The cache is a shared layer across models, and models are the execution layer behind the cache; once that division of labour is clear, the multi-model math works out.
8. Launch checklist: the full chain from request to money saved
Put the previous seven sections together, and every request follows the same chain: first check the cache, return directly on a hit, ask the model only on a miss, write the model's answer into the cache when it qualifies, and let expiry rules and privacy filtering guard both ends. Measure that chain with the same fictional numbers as before: in one month, out of 10,000 requests, exact caching served 3,800, semantic caching caught 2,400, and the remaining 3,800 actually went to the model — a 62% hit rate — and at ¥0.12 per call, about ¥744 was saved in a month. These numbers are invented, to show that every link in the chain saves something: exact caching catches word-for-word repeats, semantic caching catches reworded ones, and only the rest is worth paying for.

Before launch, there is a short checklist worth ticking off one by one. First, run for two weeks before deciding: if repeated questions are under 20% of requests, or an answer's shelf life cannot be stated, do not cache yet. Second, ship exact caching first and semantic caching later, set the threshold using real questions, and put the correction entry next to the answer. Third, put the normalised question, model name, prompt template version, and answer language into the cache key; answers carrying user data go only into a personal cache, and keys use hashes, not raw text. Fourth, give every entry a death date set by its answer's shelf life, clear entries by tag the moment source material is updated, and when a replacement fails, keep serving the old answer a little longer with a note rather than showing the user a blank page. Fifth, never cache the five kinds of sensitive prompt; entry filtering, separate public and personal buckets, and regular sweeping must all be in place, and cache access, backups, and logs must be treated as sensitive data. Sixth, share one cache pool across models with the model named in the key, let old caches expire naturally on a model upgrade, and check the cache first, then pick a model by difficulty. Seventh, review hit rate and correction rate weekly — a hit rate steady at 50–70% with a correction rate under 2% is healthy — and keep the cache's total cost under 30% of the money saved. With those seven done, LLM output caching is genuinely live, not just another black box hung on the product waiting to take the blame.
Comments (0)
Related articles
Flexbox lays things out in one direction at a time, and when it is done right, the nav bar, the card wall, and the cart rows all behave. This guide follows MDN's Basic concepts of flexbox, Aligning items in a flex container, and Controlling ratios of flex items along the main axis: the two axes, containers and items, direction and wrapping, free space, the three flex properties, alignment on both axes, and gaps and auto margins.
One order is one object: table, dishes, total, paid — all filed by label. This guide follows MDN's Working with objects, JSON, and Object references: object literals, dot vs bracket notation, adding and deleting properties, for...in with hasOwn, methods and this, getters and setters, references vs copies, Object.assign and friends, null-prototype objects, JSON's strict rules, and stringify and parse with their number-precision trap.
On 17 November 2026, Microsoft launched MAI Code 2 and Code 2 Flash: the larger model has 77 billion active parameters and a one-million-token context, averaging 63.2% across four benchmarks — close to GPT-6.1 Sol and Claude Opus 5.5 — at $0.309 per task. Read the three caveats first: promotional pricing, the lowest score of the three, and no long-horizon work.