Rotating Keys and Rehearsing Leaks: A Key-Rotation Playbook for Third-Party APIs

When nothing goes wrong, a key looks like just a string. When something goes wrong, it turns out to be a pass that can run up a bill and walk off with data. This guide does not cover where keys should be stored or whether they may be handed to an AI; that belongs to the credential-management guide. It covers only two things: how to rotate keys regularly when nothing has gone wrong, and how to stop the bleeding within a fixed time when something has. The earlier guide covers storage; this one covers rotation and rescue.

1. Rotation is transfusion, not a password change

Many people understand rotation as deleting the old key, generating a new one, and changing one config value. In a solo project that approach has three traps. First, running jobs, scheduled tasks, or an older app version may still use the old key, so deleting it breaks them. Second, if the new key is misconfigured at the moment of change, there is no old key left to roll back to. Third, nobody remembers which services and environments were changed, so troubleshooting becomes a full sweep. The right mental model is transfusion: old and new keys coexist for a while, the new blood runs first and proves itself, and only then does the old blood retire, with no outage, a rollback path, and a checklist throughout.

The dual-active window is the core: for a period, the same service accepts both keys, the client moves part of its traffic to the new key, watches for anomalies, moves the rest, and then leaves the old key in a read-only observation period until its call volume hits zero before revoking it. Section six covers providers that do not support dual-active keys; do not abandon the whole rotation plan because one provider lacks it.

2. Inventory first, then cadence

Before rotating anything, build a key inventory with as few columns as possible: service, purpose, environment (production or staging), owner (you or an automation job), last rotation, next due date, and worst-case loss if leaked. That last column matters most: keys that can run up bills (payments, model calls) or read user data (databases, storage) go in tier one; read-only keys for public data go in the last tier. Tier one rotates every ninety days, tier two every six months, and the last tier gets an annual review; write the cadence into the calendar instead of relying on remembering. Also hunt shadow keys: keys you thought were deleted but still live in an old server's environment variables, an old CI config, or a forwarded chat message. Check the provider dashboard for recent call sources: a key the inventory calls retired but that was called last week is a shadow key. Register it before rotating anything.

3. Rotation in six steps: one key at a time

Step one, generate the new key and place it only in the key store, changing no running config; log who generated it, when, and for which service. Step two, canary: let one non-critical path (staging, or one low-frequency production job) run on the new key for twenty-four hours, watching success rate, latency, and billing for anomalies. Step three, full switch: move the remaining paths to the new key in batches, checking monitoring after each batch, never all at once. Step four, observation: keep the old key for seven to fourteen days, downgrading its permissions where possible (read-only, for example), and watching its call volume; only zero calls means truly unused. Step five, revoke the old key and tick it off the inventory, logging the revocation. Step six, one line of review: what changed, did anything break, what to watch next time, written into the inventory notes. Rotate one key at a time; rotating five at once means that when something breaks, you cannot tell which key caused it.

4. Leak severity: grade first, then act

When a leak happens, the worst move is to investigate while panicking. Spend one minute grading. Grade one: a production key reached a public repository, public chat, or public log; assume it has been taken. Grade two: the key reached a private but shared place (a shared document, a small group chat); assume it may have been seen by someone unrelated. Grade three: exposure limited to your own device (a flash on a shared screen); treat it as low probability but to be confirmed. Grade decides speed: grade one, revoke immediately without first checking call logs; grade two, revoke then check; grade three, check logs first and rotate if anything looks off. Reversing that order hands the attacker a longer window at your own expense.

5. The leak timeline: 15 minutes, 60 minutes, 24 hours

Zero to fifteen minutes: revoke or freeze the key, switch to a standby key to restore service, and record discovery time, discoverer, key name, and suspected channel. Do not wait to confirm theft; a grade-one leak is treated as stolen. Fifteen to sixty minutes: inspect provider call logs for unusual regions, time windows, volumes, and billing spikes; check whether other keys were exposed in the same repository or document, because leaks often arrive in batches. Sixty minutes to twenty-four hours: complete the switch to the new key, notify affected users if user data may have been read (state the scope and the actions taken plainly, without hedging), and write a postmortem: cause of exposure, detection delay, time to stop the bleeding, and next improvement. An old key left unrevoked past twenty-four hours is a door left open.

A copy-ready example: at 10:00 on a Tuesday you find a model-call key in a public repository commit. At 10:05 you revoke it in the provider dashboard and enable the standby key; service errors for three minutes, then recovers. At 10:30 the logs show two thousand calls from an unfamiliar region, about forty dollars extra. At 11:00 you confirm no other key was in the same commit. That afternoon the new key is fully switched and the postmortem written, with a single improvement: a pre-commit key-scanning hook. Without this timeline, the same incident easily eats a whole afternoon.

6. When the provider has no dual-active support

Some small providers accept only one key at a time: generating a new one instantly invalidates the old. Do not force the dual-active flow; use a short-window switch instead. Pick the lowest-traffic hour, validate the new key in staging first, prepare the production switch script including a one-click rollback plan where the provider allows it, then run a smoke test over the core flows immediately after switching, and only then call it done. Record the old key's call volume and billing baseline beforehand, and watch monitoring for thirty minutes after. Keep the window under fifteen minutes and warn users in advance about brief maintenance. No dual-active support is not a reason to skip rotation; it just means a shorter window and thicker preparation.

7. The drill: forty-five minutes, once a quarter

Do not let a real leak be the first rehearsal. The script is simple: prepare a dedicated drill key (genuinely usable, minimal permissions, read-only public data), deliberately commit it into a private test repository, then walk the section-five timeline: time the revocation, time the switch, time the log inspection, time the postmortem. Pass criteria: revocation and switch inside fifteen minutes, an anomalous call found in the logs, at least one actionable improvement in the postmortem, and the drill key revoked afterwards with nothing left in any inventory. Run it quarterly and record the elapsed time; dropping from forty-five minutes to under twenty means the process is actually in your hands. The easiest drill mistake is using a production key, or forgetting to revoke the drill key afterwards, which turns the drill key itself into a new leak. A drill key must be separately generated, separately labelled, and revoked the moment the drill ends.

8. Six anti-patterns to fix on sight

Deleting the old key during rotation manufactures an outage. Sharing one key across all services means one leak compromises everything; splitting by service and environment is a precondition for rotation. Committing keys to a repository and trusting that private means safe ignores collaborators, former contributors, and accidentally public screenshots. Investigating whether theft occurred before revoking hands the attacker time. Keeping rotation records only in your head means that six months later nobody can say which keys rotated and which did not. Finally, a drill performed for show, with no revoked drill key and no improvement item, is worse than no drill, because it provides false confidence.

9. Checklist for the inventory's front page

Does the inventory cover every service and environment, with no shadow key unregistered? Are tier-one keys inside the ninety-day cadence? Did the last rotation follow all six steps without skipping? Was the last drill within three months, and how long did it take? Is the leak timeline saved somewhere reachable offline, since a leak may be exactly when online documents are unreachable? All five ticked means key handling has moved from feeling to process. Any box unticked, fix the process before the next feature, because a key incident's billing and trust costs never follow a feature schedule.