When the Model Goes Down: Fallback and Degradation Strategies for AI Features in Vibe Projects

First, the boundaries: what this guide is not

Before we start, let us be precise about how this guide differs from four guides already published on this site, so you do not walk into the wrong article.

First, the existing rate-limiting guide is about limiting inbound requests so your system is not abused or flooded — it protects you from others. This guide is about the opposite direction: when the model provider you depend on goes down, throttles you, slows down, or returns garbage, how do you survive? One guards the entrance, the other guards an outbound dependency. Different problem domains entirely.

Second, the existing error-monitoring guide is about observing failures — getting alerts, tracing crashes, finding out something broke. That is "seeing the problem." This guide is about "the problem has already happened, users are still online, what do you serve them instead": fallback ladders, cached answers, queues, and fallback copy. Monitoring tells you there is a fire; degradation decides whether the house is still habitable while it burns.

Third, the existing feature-flags guide is about rollout switches — gradual rollouts, canary releases, emergency shutoffs as a release strategy. This guide is about runtime resilience: a switch is only one execution mechanism for the last rung of a fallback ladder. It does not answer the real questions: what should each degraded level output, who sees it, and when do you automatically climb back up?

Fourth, the existing last-hour launch runbook is about launch-day execution — the checklist and Go / No-Go decisions in the final hour before release. This guide is about runtime, after launch, on some night when a provider goes down. The timelines differ: one is launch day, the other is every night of operation.

So what this guide adds, which none of those four cover as a system, is: a degradation design for AI features — timeout and retry rules, a model fallback ladder, a decision table by feature type, the honesty boundary for serving cached answers, queue-and-notify as an alternative to blocking the user, degraded-mode UX copy, and a kill-switch, status page, and outage-night checklist.

Accept the reality: the model will go down

If your vibe project has an AI feature — a chat assistant, summarizer, classifier, code generator, agent loop — its failure is not a question of "if" but "when, and in what form." In practice there are four failure modes, and they cover nearly everything:

• Outage: the provider is down entirely, returning 5xx or unreachable. Every user is affected at once, and no correctness in your code helps.
• Rate limiting: 429 responses. You did nothing wrong; you hit a quota, concurrency, or burst limit. Part of requests fail, and naive retries make it worse — retrying without backoff helps the provider knock you down harder.
• Timeout: the request disappears into silence. The result is unknown: the model may already be running, may already have billed you, may already have written to your database. All you know is that no answer came back.
• Bad output: the request "succeeds" but the result is empty, truncated, malformed, off-topic, or fails JSON parsing. The HTTP status is 200, your monitoring is green, and the user sees garbage.

Each mode needs a different response. Treating them all as "it errored, retry once" is the most common design mistake. Retrying during an outage is wasteful, retrying immediately on a rate limit adds fuel to the fire, blindly retrying a timeout can double-bill or double-write, and retrying the same model on bad output will likely produce a different piece of garbage. Each section below is a targeted countermeasure for one or more of these modes.

One judgment call comes first: is your AI feature on the critical path? If AI is a bonus (an "intelligent summary" button), you can degrade boldly. If AI is the product itself (an AI writing tool), degradation can only protect the user's existing work — it must never fabricate a fake result and hand it over. Write this judgment down as the first line of your requirements doc before writing any code; it determines every tradeoff in the ladder below.

First line of defense: timeouts, retries, and idempotency

Everything else rests on this layer. If timeouts and retries are wrong, the whole ladder is built on air.

Set timeouts explicitly, in layers

Do not rely on runtime defaults — many HTTP clients default to "infinite" or several minutes, long after the user has closed the tab. Set three layers:

• Per-call timeout: set by feature. Chat: 20–30 seconds. Classification: 5–10 seconds. Background summarization can stretch to 60 seconds. Beyond that, the answer has no value to a waiting user.
• End-to-end timeout: the total budget from click to result, slightly longer than a single call to leave room for retries and fallback, but never unbounded.
• First-token timeout for streaming: if you stream output, a late first token and a mid-stream stall are two different failures. A first-token timeout (say 10 seconds) lets you trigger degradation early instead of leaving the user staring at a blinking cursor.

Here is a TypeScript-flavored timeout wrapper. It solves exactly one thing: any async call either returns within the budget or fails explicitly, leaving no pending case.

async function withTimeout<T>(
  task: Promise<T>,
  ms: number,
  label = "ai-call",
): Promise<T> {
  let timer: ReturnType<typeof setTimeout> | undefined;
  const timeout = new Promise<never>((_, reject) => {
    timer = setTimeout(
      () => reject(new Error(`${label} timed out after ${ms}ms`)),
      ms,
    );
  });
  try {
    return await Promise.race([task, timeout]);
  } finally {
    clearTimeout(timer);
  }
}

// Usage: a chat call that does not return in 25s is a timeout,
// and the fallback branch takes over.
const reply = await withTimeout(callModel(prompt), 25_000, "chat");

One subtlety: this wrapper only stops the caller from waiting. It does not cancel the work already running at the provider. If your client supports AbortSignal, pass the signal through; otherwise a timed-out request may keep consuming quota in the background. A timeout response is not the same as a cancelled task, and that distinction matters for billing and database writes.

Retry only what is worth retrying, with backoff

Write retry rules as an allowlist, not as "retry on failure":

• Worth retrying: 429 (rate limit), 502 / 503 / 504 (gateway and provider transient faults), connection resets, mid-stream stalls (where you can resume or cleanly resend).
• Never retry: 400 (the request itself is wrong; a hundred retries give the same 400), 401 / 403 (authentication problems), 4xx content-policy blocks, business validation failures.
• Retry at most once, carefully: timeouts. The result is unknown, so confirm idempotency (below) before retrying, or you risk double billing or double writes.

Cap retries at 3 attempts including the first, using exponential backoff with jitter: roughly 1s, then 2s, then 4s, each plus 0–50% random jitter. Jitter is not decoration: the moment an outage lifts, every client retrying at once creates a second shockwave, and jitter is what disperses it. If the provider's 429 response includes Retry-After, obey it instead of forcing your own backoff table.

const RETRYABLE = new Set([429, 502, 503, 504]);

function backoffMs(attempt: number): number {
  const base = Math.min(1000 * 2 ** attempt, 8000);
  return base + Math.floor(Math.random() * base * 0.5);
}

async function callWithRetry<T>(
  fn: () => Promise<T>,
  maxAttempts = 3,
): Promise<T> {
  let lastError: unknown;
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err: any) {
      lastError = err;
      const status = err?.status ?? err?.response?.status;
      const isLast = attempt === maxAttempts - 1;
      if (isLast || !RETRYABLE.has(status)) throw err;
      await new Promise((r) => setTimeout(r, backoffMs(attempt)));
    }
  }
  throw lastError;
}

One more point that is easy to miss: control the retry budget globally, not per request. During a mass failure, thousands of requests each retrying 3 times multiplies traffic by 3. A simple, effective rule: when the 5xx / 429 share exceeds a threshold (say 20% within a minute), automatically drop the global retry count to 1 or even 0, and restore it after recovery. That is a rudimentary circuit breaker, achievable in a few dozen lines without a heavy breaker library.

Idempotency keys: the safety belt for timeout retries

An idempotency key solves the hardest timeout problem: you do not know whether the first request actually executed. The client generates a unique key per user intent (a UUID), sends it with the request, and the server deduplicates on it — a second arrival with the same key returns the first result directly, without re-executing, re-billing, or re-writing.

• Key granularity is "one user action," not "one HTTP request." One user click on "generate report" shares one key across the whole flow; retries, refreshes, and double-clicks all carry the same key.
• Keys must be persisted server-side for at least 24 hours. Memory-only keys vanish on restart, which is the same as having no keys.
• Non-idempotent operations need keys; pure reads do not. The test is: would repeating this cause a side effect? If yes, add a key; if no, skip it.
• On key collision, return the first result — do not error. Returning 409 and asking the user to "try a different key" leaks an internal mechanism to the user. Hand back the original result as-is.

The minimal server-side shape is: on arrival, look up the key — hit, return the stored result; miss, execute, then store key and result together. The "in progress" state matters too: if the first request is still running when the second arrives with the same key, wait for the first to finish (or return "processing") instead of running a second copy in parallel. Many double-billing incidents trace back to exactly this: "completed-state dedup" without "in-progress dedup."

To put it plainly: an idempotency key is not an alternative to retries; it is the precondition for safe retries. Retrying a timeout without a key is gambling that the first attempt never ran. Retrying with a key is engineering.

Second line of defense: the model fallback ladder

When retries cannot save you, it is time to degrade. A fallback ladder is a pre-ranked list of alternatives, tried in order, each cheaper, simpler, and more deterministic than the last, until one succeeds — or until you reach the final rung: honestly telling the user that AI is temporarily unavailable.

The standard ladder has five rungs:

1. Primary model: the model you use normally — best quality, highest cost.
2. Cheaper model: a smaller model from the same provider, or a budget model from another provider. Slightly lower quality, but still genuinely generated. The precondition is that you have verified it works for your task — not every task survives a model downgrade, as the decision table below shows.
3. Cached answer: a good answer previously generated for the same or a highly similar input. Best for repetitive questions: FAQs, fixed-format summaries, repeat queries.
4. Static / template answer: pre-written content that needs no model: how-to guides, checklists, forms, rule explanations. Lowest quality, but 100% reliable.
5. Honest unavailable state: a clear message that AI is temporarily unavailable, plus an alternative path (retry later, queue the job, contact a human). This is the floor, not a failure — fabricating an answer is the failure.

The key implementation rule: step-down and step-up conditions must be written in code, not decided by a human in the moment. For example: after N consecutive primary-model failures, or when the 429 / 5xx share crosses a threshold, or when single-call latency exceeds budget, automatically drop one rung; after M consecutive primary-model successes or a passing health check, automatically climb back. Manual overrides exist only for extreme cases, and every override must be logged with who, when, and why.

type FallbackResult =
  | { kind: "generated"; text: string; model: string }
  | { kind: "cached"; text: string; cachedAt: string }
  | { kind: "template"; text: string; templateId: string }
  | { kind: "unavailable"; reason: string };

async function generateWithFallback(prompt: string): Promise<FallbackResult> {
  // Rung 1: primary model (with retries)
  try {
    const text = await callWithRetry(() => callPrimary(prompt));
    return { kind: "generated", text, model: "primary" };
  } catch { /* fall through */ }

  // Rung 2: cheaper model
  try {
    const text = await withTimeout(callCheaper(prompt), 15_000, "cheaper");
    return { kind: "generated", text, model: "cheaper" };
  } catch { /* fall through */ }

  // Rung 3: cached answer
  const cached = await lookupCache(prompt);
  if (cached) return { kind: "cached", text: cached.text, cachedAt: cached.at };

  // Rung 4: template answer
  const template = matchTemplate(prompt);
  if (template) return { kind: "template", text: template.text, templateId: template.id };

  // Rung 5: honest unavailability
  return { kind: "unavailable", reason: "AI provider unreachable" };
}

Notice one design choice in this code: each rung's failure falls through to the next instead of throwing. The caller then decides presentation by kind: generated renders normally, cached and template must carry labels (next section), unavailable renders the degraded-state UI (later section). Making degradation a return value rather than an exception is the most important philosophical line in this design: degradation is a normal branch of the flow, not an afterthought bolted onto error handling.

Decision table: not every feature fits the same ladder

Trim the ladder per feature type. Use this table against your own feature list:

Feature type	Cheaper model	Cached answer	Template answer	Final rung
Chat / Q&A	Good fit: downgrade first, quality stays acceptable	Good for FAQs; poor for personalized questions	Good fit: point to docs, FAQ, human support	Honest unavailability + queue to human
Classification / tagging	Good fit: small models usually classify well enough	Good fit when the same input repeats	Good fit: default category + "needs human review" flag	Store now, mark "pending classification", backfill later
Generation (articles, code, summaries)	Use with care: quality drops are obvious to users	Only for repetitive template-style generation	Good fit: give an outline, a template, a checklist	Never fabricate the full text; save the draft + queue regeneration
Agent loop (multi-step tasks)	Use with care: small models drift off-track over multiple reasoning steps	Poor fit: steps depend on context, cache rarely hits	Good fit: hand over the completed steps so far	Pause the loop, save progress, tell the user exactly which step was reached

Two rows deserve emphasis. First, generation features must never pass off a template as a generated result: the user asked for a summary of their material, and handing them a generic template without labeling is fabrication. The honest move is to give the outline and template and say plainly that full generation failed and this is a starting point. Second, the degradation unit for an agent loop is the step, not the whole task: if the loop finished 5 steps and step 6 failed, degradation means handing over the 5 completed steps and saving progress for resumption — not discarding the task, and not pretending all 6 steps finished.

One cross-cutting rule: degradation must not degrade safety. Cheaper models, caches, and templates must pass the same safety checks as the primary model (sensitive-data filtering, injection protection, output validation). The most dangerous second-order outage on a failure night is a fallback path that bypasses the main path's safety checks — content the primary model's policy would have blocked must not leak out through the template path.

Third line of defense: caching past good answers, with honesty as the boundary

Caching is the highest-value rung on the ladder — and the easiest to abuse. Its value is simple: many AI requests are repeats. The same question gets asked by different users, the same material gets summarized repeatedly, the same items get classified over and over. Store those past good answers, and during an outage you can serve them with almost no user-visible damage.

But caching tempts you to pass an old answer off as a new one. Whether serving a stale answer is acceptable comes down to three tests:

• Does the answer decay with time? Stable knowledge ("how to configure X") can be cached for weeks; time-sensitive answers (today's exchange rate, stock price, policy) become misleading after hours. Set a TTL per answer type; do not govern all types with one global TTL.
• Does the answer depend on personal input? Generic questions can be cached; answers containing a user's private data, orders, or codebase have low hit rates and high cross-user leakage risk. Personalized answers should either not be cached or be cached per user, never shared across users.
• Can the user tell old from new? This is the decisive test: whenever an answer comes from cache, label it with its time and provenance, e.g. "This is a reference answer generated 3 days ago; AI service is temporarily unavailable." The label is not decorative legal text — it is the honesty boundary itself. Users have the right to know they are not seeing a freshly generated result.

Here is the acceptable-versus-dishonest contrast, usable directly in copy review:

• Acceptable: an FAQ answer labeled "cached on [date], for reference only"; a historical summary labeled with its generation time; a classification result labeled "recovered by historical rules, to be re-run when the model recovers."
• Dishonest: presenting a cached answer as freshly generated; presenting yesterday's price, policy, or score as today's data; showing one user's answer to another user; dressing up a template answer as "generated for you by AI."

Three engineering notes round this out. First, cache keys should include the inputs that actually affect the answer — entities, parameters, versions — while stripping irrelevant wording, whitespace, and politeness, or hit rates will be dragged down by meaningless phrasing differences. Second, only validated good answers enter the cache: after the primary model returns, check format and basic quality (at minimum, sane length and structure) before storing. Bad output must never enter the cache, or one outage will poison every later degradation. Third, the cache itself must degrade: if the cache service is down, the ladder must skip straight to the template rung rather than stalling on cache — no link in the degradation chain may be a single point of failure.

Fourth line of defense: queue-and-notify instead of blocking the user

Not every AI task needs to finish in front of the user. Report generation, batch summarization, long agent tasks, media processing — anything the user can wait minutes for — degrades best not as a worse instant answer but as accept, queue, and notify when done.

The applicability test is one question: "Does the user need to see the result right now?" If the answer is no, do not fight the model outage inside the request-response cycle. The flow should be:

1. The user submits the task; the server immediately returns a task ID and a confirmation: "Queued, expected within X minutes." The page does not block.
2. The task enters a queue; workers consume in order. On provider failure, the task stays queued and retries with backoff instead of failing outright.
3. On completion (or final failure), the user is notified via in-app notification, email, or a status refresh on the original page.
4. When the user returns, they see the result, the progress, the failure reason, and a retry entry point.

This pattern also solves three problems the synchronous pattern cannot. First, peak shaving: bursts queue up instead of instantly exhausting quota. Second, retry-friendliness: queued tasks can wait for the provider to recover and run then; users need not sit on the page hammering retry. Third, observability: queue length, wait time, and failure rate are themselves health signals — a suddenly growing queue is often the earliest sign of trouble.

Queues have their own pitfalls to handle in advance. First, queues are not infinite: set a cap and a TTL, and when full, explicitly reject new tasks ("the queue is full, please try later") rather than silently accumulating until the system collapses. Second, task status must be queryable: users must always be able to see "queued / processing / done / failed" — a spinner alone is not a status page. Third, failure needs an endpoint: cap queue retries too (say 5), and on final failure notify the user, preserve their input, and offer one-click re-run rather than quietly dropping the task. Fourth, for paid or quota-gated operations, re-validate quota and permissions at execution time — having quota when the user queued is no guarantee of having it two hours later at execution.

A sizing note for vibe projects: you do not need heavy message-queue infrastructure early on. A database table as the task queue (status column + polling worker) carries you surprisingly far; migrate to a dedicated queue only when the queue itself becomes the bottleneck and you need multiple workers or latency guarantees. Premature heavy infrastructure is one of the most common over-engineering mistakes in vibe projects.

Degraded-mode UX copy: what to say, and what never to fake

Even perfect technical degradation fails if the copy is wrong. What users fear most during an outage is not "slowness" but "not knowing what happened, what they are getting, or what to do." Degraded-state copy must answer three questions: what happened, what can I get right now, and what should I do next.

Here are templates you can adapt directly:

• Chat downgraded to a cheaper model: "You are currently getting answers from a backup model, so quality may be slightly lower. We will switch back to the main model automatically once it recovers."
• Cache hit: "AI service is temporarily unavailable. Below is a reference answer generated 3 days ago, for reference only. You can regenerate the latest version once service recovers."
• Template fallback: "AI generation is temporarily unavailable. You can start with the checklist below, or leave the task with us and we will generate it as soon as service recovers."
• Fully unavailable: "AI features are temporarily unavailable. We are working on it. Your content is saved and will not be lost — you can continue once service recovers."
• Queue confirmation: "Task queued, expected within 10 minutes. We will notify you when it is done, and you can check progress here any time."

Three iron rules for this copy. First, plain language, no jargon: never write "upstream LLM provider returned 503." Users neither care about nor should need your architecture; "AI service is temporarily unavailable" is enough. Second, certainty, not vague promises: "almost fixed" is the worst promise you can make. If you cannot give a time, write "we will notify you when it recovers" — and actually notify. Third, never fake state: do not animate fake typing to pretend the AI is thinking, do not label a failed task as "AI is generating for you," and do not mark a template answer as "AI-generated." Users can accept "not available right now." They cannot accept discovering they were deceived.

A few UI details are worth doing. Degraded states should look distinct but not alarming — a light notice bar, not a red error screaming at the user, because degradation is "the system coping normally," not "you did something wrong." The notice belongs next to the answer, where users can see it — not buried in small print in a page corner. And most importantly, every degraded state must offer a way out: a retry button, a queue entry, a human contact, a docs link — at least one. A notice with no way out locks the user outside.

One last reminder: write degraded-state copy in advance and ship it in code, not on the night of the outage. Judgment is worst on outage nights, and copy written then tends toward two errors: over-promising ("fixed in 5 minutes" when it takes two hours) and blame-shifting ("this is the provider's problem"). Copy reviewed in calm times is what users deserve to see on a bad night.

Outage-night equipment: kill-switch, status page, runbook checklist

The five sections above are "design before the fact." This section is "handling during the fact." Three pieces of equipment should be ready before launch.

Kill-switch

A kill-switch is a master control that can immediately shut off an AI feature or force full degradation to templates / queueing. Design notes:

• The switch must be independent of the AI feature itself. If the switch lives in a database that is also down, you cannot press it. Use a config service, environment variable, or dedicated switch service — at minimum, the switch must be more available than what it controls.
• Switches must be graded: per-feature (chat only), per-level (force template level), global (all AI off). On an outage night, the last thing you want is a binary "all on or all off" choice.
• Switch operations must be logged: who flipped it, when, why, and who confirmed recovery. In a postmortem, the switch log is the most critical evidence chain.
• Switches must be rehearsed: flip each switch at least once before launch, confirm the degraded state actually appears, user data is unharmed, and recovery genuinely climbs back. An unrehearsed switch is no switch at all.

Note the distinction from the feature-flags guide: switches there govern release strategy; switches here govern survival strategy. The same switch infrastructure can be reused, but the semantics, permissions, and rehearsal must be designed separately for the outage scenario.

Status page

A status page is the most effective way to cut support load on an outage night, even if it is just a static page. The minimum viable status page needs three things: the current state of each feature (normal / degraded / unavailable), the outage start time and a brief description, and recovery information — or a "next update time." A "next update time" is more honest and more achievable than an "estimated recovery time": you may not know when recovery comes, but you can commit to "we will update progress in 30 minutes" — and actually do it.

Ownership of status-page updates must be assigned to a person: who updates it on outage nights, how often, through which channels (status page, in-app notice, email). A status page with no owner freezes at "investigating" an hour into the outage, which damages trust more than having no status page at all.

Outage-night runbook checklist

Print this checklist into the on-call handbook. When an outage hits, tick boxes in order instead of improvising from memory:

1. Confirm scope: one feature or all AI features? One provider or several? Check monitoring and queue length, not just user reports.
2. Check whether automatic degradation worked: did the ladder reach the level it should have? If not, manually flip the kill-switch to the right level.
3. Update the status page: affected features, start time, next update time. State only confirmed facts, not speculated causes.
4. Control retries and traffic: confirm the global retry budget has tightened; if needed, pause non-core AI jobs (batch, background) to preserve quota for core features.
5. Protect user data: confirm queued tasks, drafts, and in-progress agent state are intact; confirm no fabricated output was written into user data.
6. Notify and staff: notify the relevant people; arrange rotation. If an outage passes one hour, someone must take over — nobody should carry a whole night alone.
7. Verify recovery: after the provider recovers, validate primary-model output quality on a small slice of traffic before scaling back up. Do not flip everything back at once — a just-recovered service is at its most jittery.
8. Postmortem: within 24 hours, write it up: timeline, whether degradation worked, which parts were held together by humans, what should become automatic. A postmortem's output should be code or config changes, not "be more careful next time."

Step 7 deserves its own emphasis: recovery must also be designed. Much second-wave damage happens at the moment of recovery: full traffic flips back to the primary model, backlogged queue tasks flood out, and users retry en masse — three shocks stacked, knocking the just-recovered service down again. Recover in stages: 10% of traffic to validate quality, then 50%, then full; release backlogged queue tasks by priority, gradually, not all at once.

Pre-launch self-check: ten questions

The most useful way to close this guide is a self-check list. Before your AI feature ships, you should have a definite answer to each of these ten questions. The one you cannot answer is the one that will blow up first on outage night:

1. What is the timeout for each AI call? What is the first-token timeout for streaming?
2. Which error codes retry, how many times, with what backoff? Which errors never retry?
3. Which operations carry idempotency keys? Is key granularity per user action or per HTTP request? How long does the server keep them?
4. How many rungs does your fallback ladder have? What triggers step-down and step-up for each rung — automatic or manual?
5. Has the cheaper model been validated on your task? Is the quality drop acceptable to users?
6. What are the cache TTLs? Are personalized answers isolated? Are cached answers labeled with their time?
7. Which tasks use queue mode? Are queue caps, task TTLs, and final-failure notifications set?
8. Is degraded-state copy written, shipped in code, and does it offer users a way out?
9. Can the kill-switch shut things off per feature and per level? Has it been rehearsed? Are operations logged?
10. Who owns status-page updates? Are on-call rotation and recovery validation planned for outage nights?

Back to the opening line: AI feature failures are not accidents; they are the normal operating condition. Degradation design is not an optimization to add "when there is time" — it is part of the shipping bar for an AI feature. An AI feature without degradation design is at its most fragile on launch day. An AI feature with degradation design shows users, on a bad night, not a blank screen and an error, but an honest sentence, a usable alternative, and a promise of "we will notify you when it recovers." Write that promise into code before launch, and outage nights become survivable — including for you.