Already Shipped a Poisoned Dependency? A 90-Minute Containment Playbook for Vibe Projects

First, the boundary: what this guide covers, and what it does not

This guide covers exactly one situation that has already happened: you have already installed a compromised dependency into your project and shipped it into production, and now you need to contain it. It starts at "already installed" and ends at "rebuilt, credentials rotated, users notified". It is not a prevention manual.

That boundary matters, because this site already publishes several neighbouring guides and they must not be confused with this one. First, the two published dependency supply-chain security guides are prevention: how to spot hallucinated package names, typosquatting and dependency confusion before you install, and how to build safe install habits. They answer "how do I avoid installing the wrong package?" This guide does not repeat them. It assumes you already installed the wrong one and answers "what do I do in the next 90 minutes?" Second, the published secrets-management guide covers everyday credential hygiene: where to store secrets, how to grade them, how not to hand them to AI tools. That is the normal system. This guide covers only the incident-day rotation: in what order, and from what machine, you replace secrets that may already have leaked. Third, the published general incident runbook covers the generic flow for every production incident: severity levels, stopping the bleeding, rollback, postmortem. It applies to outages, slow queries and bad config alike. This guide is the specialised version of that flow for one incident type, dependency poisoning, with steps that are more specific and an order that must not be shuffled. Fourth, the published AI-feature fallback and circuit-breaker guides cover what to do when a third-party service is unavailable: timeouts, retries, circuit breakers, degradation ladders. That is "the service is down, how does the product keep running?" This guide is "the service is up, but it is stealing things, what now?" The problems are different, and so are the responses. In short: the prevention guides govern before you install, the secrets guide governs everyday practice, the general runbook governs the framework for all incidents, the fallback guides govern unavailability. This guide governs only post-install containment, for the 90 minutes after you discover you already installed it.

First principle: isolate first, then clean up; stop the bleeding first, then assign blame

When a solo developer hears "a dependency version you use has been poisoned", the most tempting mistake is to upgrade immediately, as if moving to a new version fixes everything. It does not. Upgrading runs another install on the same possibly compromised machine, through the same possibly poisoned chain. The correct order is always: isolate first, so the malicious code stops and cannot reach new credentials; then hunt for the foothold, to learn what it changed and what it stole; and only then rebuild, rotate and verify from a clean machine. Isolation is not delay. It is what makes every later step count.

The second principle is: treat "may have leaked" as "has leaked". The most common payload in a poisoned package is a credential stealer: at install time or at program startup it reads environment variables, cloud credential files, SSH keys, browser credentials, and sends them out. You cannot prove it stole nothing just because you saw no exfiltration record — logs may have been wiped, traffic may have gone to a plausible-looking domain. If the poisoned version ever ran in an environment holding real credentials, handle it as a full leak. The reverse also holds: if the poisoned version only ever ran once in a credential-free sandbox and never touched production keys, you do not need the full rotation — but you must write down how you reached that conclusion and what evidence supports it.

The third principle is a time box. One person handling an incident is most at risk from unbounded investigation: digging through three months of logs, or debating root cause in a chat thread. Give yourself 90 minutes. After 90 minutes you must have a usable clean build, a completed credential rotation list, and a note to users. Lines of investigation you could not finish go into a follow-up list after the 90 minutes. They must not block stopping the bleeding.

Step 0: find out whether you are actually affected

Do not guess, and do not look only at package.json. That file states a range, such as ^1.2.0. What actually got installed lives in the lockfile. Lockfile, install logs and image tags must all agree before the verdict is complete.

Check the lockfile first. For Python, look at requirements.txt, Pipfile.lock, poetry.lock or uv.lock; for Node, package-lock.json, pnpm-lock.yaml or yarn.lock. Search for the package name plus the suspect version, not just the name:

# Python: is the poisoned version in the lockfile? (replace BAD-PACKAGE and 1.2.3)
grep -n "BAD-PACKAGE" poetry.lock requirements.txt uv.lock 2>/dev/null
grep -A3 'name = "BAD-PACKAGE"' poetry.lock | grep version
pip show BAD-PACKAGE 2>/dev/null | grep -E "Name|Version|Location"
pip freeze | grep -i "BAD-PACKAGE"

# Node: check both the lockfile and the installed tree
grep -n '"BAD-PACKAGE"' package-lock.json pnpm-lock.yaml 2>/dev/null
npm ls BAD-PACKAGE
cat node_modules/BAD-PACKAGE/package.json | grep '"version"'

Then check install logs and caches. CI logs record exactly which version was installed in a given build, and from which source; local caches may still hold the poisoned package. Three questions matter: when was the poisoned version installed, on which machine or CI job, and did that run have access to production credentials? Installed once on a laptop and installed during a production deploy are incidents of completely different severity.

# Traces in build logs and caches (adapt to your CI and package manager)
grep -rn "BAD-PACKAGE@" .github/workflows/ 2>/dev/null
pip cache list 2>/dev/null | grep -i "BAD-PACKAGE"
ls ~/.cache/pip 2>/dev/null | head

Finally, check image tags — the most commonly missed place. If you deploy with Docker, checking the package version in code is not enough: the poisoned version may be baked into an image, and the image tag may be floating. What latest points to today may not be what it pointed to yesterday. Check the digest, not just the tag name:

# A tag alone will mislead you; check the digest
docker images --digests | grep -i "your-app"
docker image inspect your-app:latest --format '{{.RepoDigests}} {{.Id}}'
docker history your-app:latest 2>/dev/null | head -n 30

There are only three possible verdicts, and you should write yours down. One: confirmed affected — the lockfile, logs or image contain the poisoned version. Two: confirmed unaffected — the lockfile version is clean, the image digest matches a clean build, and the install time falls outside the poisoning window, with evidence. Three: cannot confirm — logs are missing, the lockfile was never committed, or a floating tag makes history unrecoverable. Treat the third case as affected. When you record the verdict, attach the command output or log excerpt. "I checked" is not evidence.

The 90-minute checklist at a glance

This table is the skeleton of the whole guide. Minutes 0 to 15 are for isolation only, 15 to 45 for hunting the foothold only, and 45 to 90 for rotation, rebuild, verification and notification. Each phase has a checkable completion standard. Do not move on until it is met.

Time	Goal	Done when
0–15 min	Isolate: make the bad code stop	Deploys frozen, bad version pinned or blocked, affected service stopped or switched to a clean build, evidence preserved
15–45 min	Hunt the foothold: what it changed, what it stole	Startup hooks, unexpected files and outbound calls all checked, with a list; nothing left as "not checked"
45–90 min	Rotate, rebuild, verify, notify	Credentials rotated in order, clean image verified live, user note sent, follow-ups recorded

Minutes 0–15: isolate — do not just "upgrade in place"

First, freeze deploys. Stop automatic deployment and tell yourself, and any collaborator, that every deploy from now on needs manual confirmation. If a CI job is currently installing the suspect dependency, stop it rather than letting it finish. Do not immediately restart a running production service: a restart may trigger a malicious startup hook again, and may clear in-memory evidence. Decide first: can this service be stopped? If yes, stop it. If no, cut its sensitive outbound permissions first — for example, demote a read-write database account or revoke a cloud write role — but do not change credentials on the compromised host. The reason follows below.

Second, pin and block the bad version. Pin the lockfile to the last known clean version before the poisoning, using an exact version number, not a range. At the same time, block the poisoned version in your package manager or private registry, so a cache hit or a colleague's local install cannot bring it back. Python example:

# Pin to the clean version with an exact number; no ^ or ~ range
pip install "BAD-PACKAGE==1.2.2"
grep -n "BAD-PACKAGE" requirements.txt poetry.lock

Node example:

# Same rule: exact version, and check the lockfile moved with it
npm install "BAD-PACKAGE@1.2.2" --save-exact
grep -n '"BAD-PACKAGE"' package-lock.json

Third, preserve evidence. Copy the build logs, application logs, install logs and image digest list for the poisoning window into an incident folder, and keep a snapshot of the current file listing on the affected host or container. Do not rush to uninstall the package, delete the container or clear caches — once deleted, the foothold is gone for good. Preserving evidence takes minutes; recovering it later is impossible.

# Snapshot the current state before you touch anything
pip freeze > incident-pip-freeze.txt
npm ls --all > incident-npm-ls.txt 2>/dev/null
docker ps -a > incident-containers.txt
ls -la > incident-workdir.txt

Fourth, settle the internal wording baseline: do not publish an announcement yet, but write one internal sentence now — "A dependency version is suspected poisoned, isolation in progress, affected scope to be confirmed." Its job is to stop you from making promises under pressure before you have facts, and to give you a starting point when you write the user note at minute 90.

Minutes 15–45: hunt the foothold — startup hooks, unexpected files, outbound calls

A malicious dependency usually persists in only three ways. Check all three; do not skip one on a hunch. The first is startup hooks: code that runs automatically when a program starts, without you importing it. In Python the classic hooks are .pth files and sitecustomize modules, which the interpreter loads at startup. In Node they are install scripts and modified entry points: preinstall, postinstall, and a tampered startup file. List those locations directly and look for files you do not recognise:

# Python startup hooks: .pth and sitecustomize are the priority
python -c "import site; print(site.getsitepackages())"
ls -la $(python -c "import site; print(site.getsitepackages()[0])") | grep -E "\.pth|sitecustomize|usercustomize"
cat $(python -c "import site; print(site.getsitepackages()[0])")/*.pth 2>/dev/null

# Node install scripts and entry points
cat node_modules/BAD-PACKAGE/package.json | grep -A5 '"scripts"'
grep -rn "preinstall\|postinstall" package.json node_modules/BAD-PACKAGE/package.json 2>/dev/null

The second is unexpected files: files created or modified during the poisoning window. Check version control for uncommitted changes, then list files modified inside the window, then check system-level persistence: scheduled tasks, system services, shell startup files. On a solo developer's own machine, shell startup files and scheduled tasks deserve special attention, because many stealers write persistence there:

# Workspace and window-scoped file changes
git status --short
git diff --stat
find . -type f -newermt "2026-10-21 00:00" ! -newermt "2026-10-22 00:00" 2>/dev/null | head -n 100

# System-level footholds (check what applies; do not copy paths blindly)
crontab -l 2>/dev/null
ls -la ~/.bashrc ~/.zshrc ~/.profile 2>/dev/null
systemctl --user list-unit-files 2>/dev/null | grep enabled

The third is outbound records: where did it actually connect? Look at outbound requests in application logs, audit logs from your cloud provider, and unusual domains or regions during the window. The question is not "was there any outbound traffic?" but "was the destination a service you recognise?" An unfamiliar domain, a region you never use, or a burst of key-related calls at night are all signals. When the three checks are done, write them up as a list: what was found, what was not found, and what could not be confirmed because logs are missing. The "could not confirm" item is what forces the worst-case rotation scope.

Minutes 45–90: rotate from a clean machine, in an order that must not be shuffled

The first iron rule of rotation: do not generate new credentials on the compromised host, and do not log into a cloud console from it to revoke old ones. If the foothold is still alive, a new credential can be stolen the moment it is created, and the rotation is wasted. Use a clean machine: another computer you have confirmed never installed the poisoned version, or a freshly provisioned cloud host. Every revoke, generation and entry of a new credential happens there. The compromised host is only isolated, inspected, and finally rebuilt.

The rotation order follows "stop the bleeding before refilling; cloud control first, application secrets second". At each step, revoke the old credential at the provider first, then generate the new one. Generating first and revoking later leaves a window in which both work.

1. Cloud root access and IAM roles: revoke possibly leaked access keys first, and pull back write permissions and console sessions from suspicious roles. Do this first because anyone holding cloud access can work their way to everything else.
2. CI and code-hosting tokens: GitHub, GitLab, CI runner and container registry tokens and deploy keys. With these, an attacker can push another bad build into your repository and images.
3. Database and infrastructure credentials: database passwords, cache passwords, object-storage keys, queue credentials. After changing them, verify the old ones really fail to connect; changing config without verifying is not rotation.
4. Third-party API and model-service keys: payment, SMS, email and model-provider API keys. Work from highest production usage downwards.
5. Sessions and login state: user session cookies, OAuth refresh tokens, single-sign-on sessions. Many stealers specifically target browser cookies and sessions, so rotating API keys without invalidating sessions leaves the door open. Force a site-wide re-login if needed.
6. Signing and callback secrets: webhook signing keys, JWT signing keys, encryption keys. After rotating, update every verifier in step, or your own callbacks will break first — write the verification step down in advance.
7. Finally, personal account passwords: if your own email or code-hosting account was logged in on the compromised machine, change the password through the provider's flow and review login history and newly authorised applications.

Track rotation in a table: credential name, old one revoked or not, new one generated on a clean machine or not, which services updated, which services verified. Do not rely on memory — after 90 minutes you will not remember it all. Any row stuck at "generated but not yet updated" counts as unfinished.

Rebuild and verify: start from a clean image, do not reuse the old lockfile

Rebuilding is not "swap the bad package and deploy again". The correct rebuild starts from a clean base image, regenerates the lockfile from the pinned clean version, builds in a clean build environment, and deploys by digest. Do not reuse the old lockfile directly: if the poisoned version ever entered it, or its hashes were tampered with, reuse carries the problem forward. Likewise, do not reuse an old Docker tag: floating tags such as latest and main-latest have pointed at poisoned images, so a deploy must pin a digest and confirm that digest matches the clean build.

# Clean rebuild check order: remove the old environment, reinstall clean, verify
rm -rf node_modules .venv
npm ci
pip install -r requirements.txt
npm ls BAD-PACKAGE
pip show BAD-PACKAGE | grep Version

Verify at least five things: one, no leftover files in startup-hook locations; two, the lockfile version matches the expected clean version and the image digest matches the clean build; three, the service starts and passes a smoke test; four, the anomalous outbound destinations seen during the poisoning window no longer appear; five, the old credentials really fail — an old key tried once should be rejected, not still work. All five must pass before the rebuild counts as done. If any fails, go back to the corresponding step. Do not waive a check to meet the 90 minutes.

One real example: LiteLLM 1.107.0 (illustration only, not a news report)

On 21 October 2026, version 1.107.0 of LiteLLM, a Python LLM gateway library, was live on PyPI for about two hours before being removed; Docker Hub and GHCR tags v1.107.0, latest, main-latest and main-stable also pointed at the compromised image during that window. The release contained credential-stealing code and a remote-code-loading backdoor, and related startup-hook files could survive uninstalling the package, so uninstalling alone was not enough. The recommended clean version was 1.106.2. This example illustrates exactly the three points of this guide: floating tags spread a poisoned image to every deploy that did not pin a digest; startup-hook leftovers mean "deleted the package" does not mean "clean"; and a credential-stealing payload must be handled as "already leaked", not as "leaked only if we find exfiltration evidence". Use it as a mirror for your own project. Do not use it to infer your own exposure.

The note to users: answer four questions

Before minute 90 ends, send the user note. Users do not need technical detail, but they need four clear answers: what happened, how long the window was, what you have already done, and what they themselves need to do. For anything still uncertain, write "not yet confirmed, under investigation". Do not soothe with "probably no impact". A fill-in template:

[Incident note] During [dates and times] we found malicious code in
[package name and version] used by this project. We isolated and removed
the affected version at [time].
Completed: rebuilt a clean version, revoked and rotated possibly affected
credentials, verified old credentials no longer work.
What you need to do: [re-login / rotate your API key / check your billing].
Not yet confirmed: [items still under investigation and next update time].
Next update: [specific time].

If user data may have been affected, the note must say what users should do and what "done" looks like for them, not only "we are sorry". If you ultimately confirm user data was not affected, say how you confirmed it and on what evidence. After sending the note, file the timeline, the verdict evidence, the rotation table and the verification results in the incident folder. The next time this happens, your first move will not be panic; it will be opening that file.

Six common mistakes — check yourself against them

1. Rotating credentials from the compromised host: a new key can be stolen by a stealer that is still running the moment it is created. Rotate only from a clean machine.
2. Upgrading without isolating: moving to a new version leaves old startup hooks, unexpected files and stolen credentials untouched. It changes the packaging, not the compromise.
3. Trusting the old lockfile: the lockfile itself may be part of the poisoned chain. Regenerate and verify it during rebuild; do not reuse it blindly.
4. Forgetting floating Docker tags: the package version in code is clean, but the deployed latest tag still points at the poisoned image, and the next restart re-infects you. Deploy by digest, always.
5. Deleting evidence before investigating: uninstalling, clearing caches and deleting containers feels decisive, and it destroys the foothold and the timeline. Snapshot first, then act.
6. Declaring "all clear" too early: outbound logs unchecked, old credentials never verified dead, sessions never invalidated — and then an "all fixed" announcement. Until all five verification checks pass, do not write "fixed".

Closing: a one-page self-check list

Save this list as your incident template and tick it off directly next time. Each item needs an owner and a completion time; a tick without evidence does not count.

[ ] Lockfile / install logs / image digests checked; verdict written down
[ ] Deploys frozen; bad version pinned and blocked
[ ] Evidence snapshot kept (logs, file listing, container listing)
[ ] Startup hooks / unexpected files / outbound calls checked, with a list
[ ] Rotation done from a clean machine, in order; old credentials verified dead
[ ] Clean image rebuilt by digest and passed all five verification checks
[ ] User incident note sent: impact, actions taken, user actions, next update
[ ] Follow-ups and timeline filed; items for after minute 90 listed

The most dangerous part of dependency poisoning is not the two-hour poisoning window. It is every later moment of "we thought we were clean". The goal of a 90-minute containment is never to prove nothing happened. It is to replace "we thought" with "we confirmed" — through isolation, evidence, ordered rotation and a verifiable rebuild. After those 90 minutes, your project may not yet be back to how it was before the incident, but it will be in a state you can describe, verify and stand behind. For a solo developer, that is the most important outcome incident handling can deliver.