How Long Data Lives and Where It Goes: Data Retention for Vibe Projects

The bill tripled, and data a user asked to delete would not go away: a fictional lesson first

Here is a fictional teaching example. Lao Zhou built a ticketing system called CloudSail Tickets on his own with AI. Its first year went smoothly: customers filed tickets, support answered them, tickets closed. In the second year, two things arrived. First, the cloud bill: the database line had tripled in a month, with storage showing 2.1 TB. He looked inside and found that every operation log since day one, every login record, and every attachment from tickets closed three years ago were all still sitting on the most expensive hot storage, never cleaned up by anyone. Second, an email from a customer: "Please delete all of our chat records before 2024. Our legal team requires a retention period of two years." When Lao Zhou checked the code, he found chat records had no retention concept at all: deletion meant removing them one by one by hand. Worse, the backups still held full snapshots from three years ago — even if he deleted the live data, one restore would bring it all back.

It took him a month to clean up the mess: scripts to clear logs in batches, retention periods for each kind of data, expiry policies for backups, and a written retention statement for the customer. His most repeated sentence that month was: "If only we had set this on day one." Data retention is that kind of work: invisible while it is missing, and ten times more expensive to fix once it breaks — either the storage bill runs out of control, or legal comes knocking, or a user discovers you are still holding data they asked you to delete.

This guide is about setting a data retention policy for a vibe project: which data lives for how long, how to set the retention period, how data travels from hot storage to deletion, how a legal hold freezes deletion, what to do about old data in backups, how to delete cleanly when a user asks, and the five groups of tests to run before launch. All examples below are fictional teaching examples; the apps, people and data are invented to illustrate the approach.

1. Boundaries first: retention is not archiving, not backup, and not "user retention"

The word "retention" means at least three things in product work, so separate them before any meeting uses them interchangeably. The first is user retention: users staying and continuing to use the product. That belongs to growth and win-back, covered by our earlier guide on retention and win-back — it has nothing to do with this article. The second is data archiving: moving rarely accessed old data from hot storage to cheaper cold storage. Our earlier guide on data archiving and hot-cold separation covers how to move it, where to move it, and how to read it back. The third is what this guide covers: a data retention policy — how long each kind of data should exist in the system, and what happens when its time is up: delete it, anonymise it, or keep it.

The three form a chain: the retention policy decides "how long to keep it", archiving handles "where to put it more cheaply before it expires", and backup handles "can we get it back if we delete the wrong thing". The retention policy is the boss: without it, archiving does not know what to move, backup does not know what should expire, and deletion does not know where to start. In reverse, setting retention periods without landing the archiving and backup policies leaves the retention period as words on paper.

Draw the line against our earlier guide on account deletion and data erasure too: that one covers wiping one person's data in one go when they close their account. This guide covers the system's everyday retention rules — even if a user never closes their account, logs, tickets and analytics each have their own lifespan. Account deletion is "what to do when one person leaves"; retention is "how long all data lives". The deletion flow calls on the retention policy's deletion capability, but the two are not the same thing.

One sentence to remember: a retention policy answers "how long each kind of data may live here at most, where it goes when its time is up, and who may be an exception". Any retention document that cannot answer those three questions is just a title.

2. How to set the retention period: five rulers — measure first, then decide

Retention periods cannot be guessed, and a single "keep everything for 90 days" for the whole product will not do. Different data lives very different lives: login logs are fine at 90 days, financial invoices may need 10 years, and user chat records may only be kept for 2 years. Before setting a period, measure with five rulers.

The first ruler is the law. Many industries have minimum retention requirements for specific data: invoices, contracts and transaction records must be kept for 5 to 10 years in many jurisdictions, and medical and financial data have their own rules. Privacy law pulls the other way, setting maximum retention: personal data cannot be kept indefinitely without a legitimate reason — once its purpose is served, it should go. Caught between the two, a retention period is not "as long as we like" but an interval with a floor and a ceiling. Note: this guide is not legal advice; exact numbers depend on the current law of your jurisdiction and your lawyer's opinion. What this guide gives is a method for setting the rules, not the statutes themselves.

The second ruler is the contract. Enterprise customer contracts often include retention clauses: "the vendor shall retain operation logs for no less than 180 days", or "delete all customer data within 30 days after the project ends". Contract requirements override your own defaults — if you signed it, keep data as the contract says; if the contract says delete, delete on time. Before taking on an enterprise customer, extract the retention clauses from the contract and write them into the retention list.

The third ruler is business value. Ask one question: will anyone look at this data in 30 days? In 90 days? In a year? Ticket data is reopened often within 90 days after closing, and almost never after that; analytics data is most valuable in the two weeks after a release, useful for trends after a month, and barely touched after three months. The speed at which business value decays is the speed at which the retention period should shrink. A simple method: check access records for each kind of data over the past 90 days. Data nobody has touched for 90 days can have its retention period shortened.

The fourth ruler is cost. Hot, cold and backup storage differ in price by more than an order of magnitude. Keeping 1 TB on hot storage for a year and on cold storage for a year produce entirely different bills. Do the maths: if a kind of data is accessed fewer than three times a year, there is no reason for it to keep occupying hot storage — archive it to cold storage, or delete it outright. Cost is the best lever for pushing cleanup: calculate each kind of data's monthly storage cost and pin it next to the retention list. The numbers speak for themselves.

The fifth ruler is risk. The longer data lives, the greater the loss when it leaks, and the more work when someone asks for it to be deleted. Sensitive data in particular — personal identifiers, chat records, location data — gains a month of risk for every extra month of retention. Default retention for sensitive data should be short: better to start short and extend on demand than to start long and shrink later, because when you try to shrink it you discover references to that data scattered through the codebase, and nobody dares delete it.

3. A retention list you can copy: four kinds of data, four lifespans

Below is a retention list template you can copy directly. The numbers are example defaults — change them with the five rulers above. Each row must state five things: data category, retention period, action on expiry, owner, and exceptions.

Data category	Retention period (example)	Action on expiry	Notes
Business data (tickets, orders, projects)	2 years after closing / completion	Archive to cold storage, delete after 3 more years	Follow the contract when one applies
Operation and audit logs	180 days	Delete; aggregated statistics may be kept	Follow the longer rule when compliance requires more
Analytics and behaviour data	90 days raw data; 2 years aggregated data	Delete raw data, keep aggregated reports	Remove personal identifiers when aggregating
Chat records and user-generated content	2 years	Delete or anonymise	Delete immediately when a user asks
Backup snapshots	30-day rolling window	Expired snapshots destroyed automatically	Must not outlive the live data

Three points matter in this table. First, the retention clock starts at the business end point, not at creation: a ticket's retention starts the day it closes, an order's the day it completes, a chat record's from the last message. Counting from creation deletes data that is still in flight — a ticket open for three years would already be "expired" by creation date, even though it has not closed. Second, raw data and aggregated data get separate periods: raw analytics data is enough at 90 days, but daily-aggregated statistical reports can live for 2 years, because aggregation has removed personal identifiers — low risk, high value. Third, backup retention must not exceed live-data retention: if live chat records live for 2 years but backup snapshots hold data from 5 years ago, the retention policy is worthless.

The first thing to do once the list is set is to turn it into configuration in code, not words in a document. Each kind of data maps to one retention configuration: days to keep, action on expiry, execution frequency. Keep the configuration in one place — never scattered across a dozen scheduled jobs. Scattered configuration is the most common reason retention policies fail: one place is changed, another is forgotten, and in the end nobody knows which rule is actually in effect.

4. The life of a piece of data: hot, warm, cold, deleted — four stops, in order

Data passes through four stops from creation to disappearance, each with different storage cost and access speed. Hot storage holds data in active use: in the main database, readable at any time, most expensive per unit. Warm storage holds data accessed occasionally: on cheaper storage or a read-only replica, taking a few seconds to fetch. Cold storage holds data almost never read but not yet deletable: in archival storage, taking minutes to hours to retrieve, cheapest per unit. Deletion is the terminus: the data is gone for good, unrecoverable.

The journey through these four stops is driven by scheduled jobs, not by hand. A cleanup job runs every night: move data past its hot period to warm, move data past its warm period to cold, delete data past its retention period. Every run must be logged: how many rows moved, how many deleted, whether anything failed. The cleanup job itself needs monitoring: if deletions are zero for three days in a row, either the job is broken or the retention periods are too long — both deserve an alert.

The deletion step deserves special care, in two stages: soft delete first, then hard delete. Soft delete marks data as "deleted": invisible to users, but still in the database. Hard delete actually wipes it from disk. Leave a buffer between the two stages, usually 7 to 30 days: if a deletion turns out to be a mistake inside the buffer, the data can still be restored from its soft-deleted state; once the buffer passes, hard deletion runs and the data is truly gone. Do not confuse the buffer with the retention period: the retention period is "how long data should live", the buffer is "how long a mistaken deletion can still be undone".

Deletion must also handle references: when a ticket is deleted, what happens to its attachments, comments and operation logs? When an image is deleted, what happens to the articles referencing it? The rule: deletion cascades, but in tiers. Deleting the parent deletes its children; shared data referenced elsewhere is not deleted with it, but anonymised instead — the username becomes "Deleted user", the avatar becomes the default image, the content stays but can no longer be tied to a specific person. Anonymisation is not "masking a name with asterisks": a partly masked phone number or email can still identify a person. Real anonymisation removes or irreversibly replaces every field that can identify someone.

5. Legal hold: when someone says stop, the delete key must lock

The most commonly missed piece of a retention policy is the legal hold. Normally, data is deleted when its time is up. But once litigation, arbitration or a regulatory investigation is involved, the relevant data must stop being deleted at once, even if its retention period has already expired. The person calling the stop may be legal, a regulator, or your own lawyer — and once they call it, the deletion job must stop applying to that batch of data until the hold is lifted.

A legal hold must achieve three things. First, the hold freezes a defined scope, not the whole product: the scope must be written down — all data of one customer, operation logs for one time period, or all files of one project. The more precise the scope, the less it disrupts normal business. Second, the hold needs approval and a record: who initiated it, why, what the scope is, and when it will be lifted — all four must be filed. A spoken "don't delete that yet" does not count; there must be a hold record in the system. Third, the freeze must reach into backups: if live data is frozen, expired snapshots in backups must not be destroyed on schedule either, or the evidence would be deleted through the backup.

After a hold is lifted, the frozen data returns to the normal flow: data already past its retention period is deleted within 30 days of lifting; data not yet expired continues on its original period. Lifting must also be filed: who lifted it, when, and how much data was deleted afterwards. The biggest danger with legal holds is "frozen and never unfrozen": a batch frozen three years ago, the case long over, the hold record still there, the data never deletable, turning into "permanent data" nobody dares touch. Every hold record therefore needs a review date, which automatically reminds its initiator to review it: keep freezing, or lift it.

6. Old data in backups: the most common hole in a retention policy

Lao Zhou's second lesson happened here: the live data was deleted cleanly, but full snapshots from three years ago were still sitting in backups. Backups exist so that a mistaken deletion can be recovered — but if backups never expire, they become a second database, one outside the retention policy's control. Many teams' retention policies fail exactly this way: however good the live rules are, one restore brings back data from three years ago that someone asked to be deleted.

The fix is to give backups a retention period too, and backup retention must not exceed live-data retention. Rolling backups usually keep 30 days: one snapshot a day, with anything older destroyed automatically. The number 30 is not arbitrary: it must be longer than the deletion buffer (7 to 30 days), so a mistaken deletion can still be recovered from backup; and shorter than the live data's retention period, so backups never hold data the live system has already deleted. Long-term archival backups can be kept separately if the business genuinely needs them, but they must be written into the retention list with a reason and an expiry action — never "keep it just in case".

When a user asks for their personal data to be deleted, backups need handling too. The correct approach has two steps. First, live data is deleted or anonymised immediately. Second, data in backups expires naturally in the next backup cycle — within 30 days, as old snapshots are destroyed. Say this in the privacy policy: "Your live data has been deleted immediately; backup copies will be cleared within 30 days as backups expire." Do not promise that backup data is "also deleted immediately", because that would require rewriting every backup snapshot — extremely costly, and rewriting backups can itself introduce new risks. An honest 30-day window is far more credible than an "immediate" promise you cannot keep.

One more detail: recovery drills. Run a backup recovery drill at least once a year, and run the recovered data through the retention cleanup process again — do not leave recovered old data sitting in the production database. The drill's purpose is to prove the backup works, not to resurrect data from three years ago.

7. When a user asks for deletion: four standards for "deleted cleanly"

A user asking for their data to be deleted may be closing an account, exercising the right to be forgotten, or simply not wanting it kept. Whatever the reason, "deleted cleanly" has four standards, and missing any one of them means the job is not done.

First, the live primary data is gone: the user's profile, published content, chat records and order records, handled per the retention list's expiry actions. Second, the linked data is handled: usernames in comments become "Deleted user", delivery information in orders is anonymised, personal identifiers in operation logs are removed. Third, third-party copies are synced: if data was synced to an analytics service, a support system or an email provider, notify those third parties to delete in step, or at least confirm the contract includes a deletion duty. Many teams delete their own database and forget the data still sitting in three or four third-party services. Fourth, backups expire on schedule: per the previous section, they clear within 30 days as backups expire — and the user is told so.

The deletion request itself needs a process: first verify identity after receiving the request — one email from someone else must not delete another person's data. After verification, give a clear time commitment, such as "deletion completed within 15 days". When done, send the user a confirmation email stating which categories of data were deleted and when backup copies will clear. Keep a record of the whole process: who requested it, when, when it finished, which categories were deleted. The record is not for keeping the user's data; it is for proving you did delete it — if anyone asks later, you can show the record.

One easily missed point: deletion is not "returning the account to an unregistered state". After a user closes their account, can someone else re-register their username? Is their public content deleted with them, or kept after anonymisation? Decide both questions in the retention policy in advance, not on the spot when a user asks. The usual approach: hold the username for 90 days after closure so nobody can impersonate them; keep public content after anonymisation, because deleting it would harm other users' reading experience — but say so in the privacy policy in advance.

8. Landing SOP: from one list to a job that runs every day

Finally, a retention policy must land as an executable checklist. Here are the recommended launch steps, in order — do not skip ahead.

Step one, inventory: list every table and storage bucket in the system, and name an owner for each kind of data. The inventory will surface tables nobody knows who created and fields nobody knows what they are for — write them down as "to be confirmed"; do not guess. Step two, set the periods: using the five rulers from section two, fill in the retention period and expiry action for each kind of data, and write them into the retention list. Step three, write the configuration: turn the retention list into configuration in code, with the cleanup job executing from it. Step four, add monitoring: after the cleanup job runs each day, record how much was deleted, how much failed, and how much remains; alert if anything is abnormal for three days running. Step five, write the privacy policy: state the retention periods and the deletion process in the privacy policy, telling users how long each kind of data is kept and how to request deletion.

Watch four numbers for monitoring: how much data is deleted each day (a sudden zero means the job is broken); how much data is past its retention period but not yet deleted (a growing backlog means the job cannot keep up); the age of the oldest backup snapshot (older than 30 days means the expiry policy has failed); and the number of pending deletion requests (past the promised time means the process is broken). Review these four numbers weekly and put them in the weekly report — they need no daily watching, but they must not go unwatched.

A retention policy is not a document written once and forgotten; review it at least once a year: has the law changed, have contracts added new requirements, is any data no longer read by anyone, is any kind of data unreasonably expensive? Use the review to check legal hold records too: is there a freeze that should long since have been lifted? File the review's results: which retention periods changed, why, and who approved it.

9. Run five groups of tests before launch, then call it done

Group	What to test	Pass standard
List completeness	Check the retention list against the database, table by table	Every table and bucket has a retention period and expiry action; nothing "to be confirmed" for over 30 days
Deletion execution	Create expired test data and run the cleanup job	Expired data is deleted, unexpired data untouched; restorable inside the soft-delete buffer, truly gone after it — not found in the backend
Cascade and anonymisation	Delete a test user and check every related table	Child data is deleted with the parent, shared data is anonymised, no personal identifier of that user remains in logs
Backup expiry	Check the oldest backup snapshot's age; simulate one restore	Oldest snapshot no more than 30 days old; restored old data goes through cleanup again, never left in production
Deletion request	Walk a complete user deletion request end to end	Identity verified, completed within 15 days, confirmation email sent, fully filed; third-party deletion notified

Keep this acceptance checklist: the retention list covers every table and bucket; retention periods were set with the five rulers, with reasons on file; the cleanup job runs daily on its own, with monitoring and alerts; soft delete plus hard delete run in two stages, with the buffer verified; legal holds can freeze by scope, come up for review on schedule, and can be lifted; backup retention is no longer than live-data retention, and the backup window for user deletion is written into the privacy policy; the deletion-request flow has been walked end to end, fully filed. Only when all seven boxes are ticked is the retention policy done. Good data retention is invisible to users, leaves legal with nothing to pick at, and never lets the bill suddenly triple — the best retention policy is the kind nobody notices.