Move Without Losing Data: User Data Import and Migration for Vibe Projects

Why bad imports lose users on day one

When a user is willing to move from an old tool to your product, they have already decided in your favour. But if the move loses data, scrambles fields, or forces hours of manual repair, they will reach a different conclusion on day one: this product cannot be trusted. Data import is not a side feature. It is the first major test of trust. Too many vibe-coded projects treat import as "upload a file and hope it works," so users discover scrambled categories, wrong dates, and hundreds of duplicates — with nowhere that tells them what went wrong or how to fix it. Users rarely complain twice. They quietly go back to the old tool.

The deeper reason is that imported data is never clean. A user's old data has blank rows, merged cells, three spellings of the same person's name, two date formats mixed together, and category names that changed halfway through. An import feature that assumes clean data pushes all the cleaning work back onto the user — the very work they switched tools to escape. So the real goal of import design is this: let a user upload an imperfect file and still end up with a result they can check, repair, and roll back.

One boundary first: this guide is about users bringing their own business data into your product from old tools and table files. Changing database schema is a different, already-published topic, and generating fake test data is another. This guide covers neither. It covers the full journey of real user data, from upload to verification.

Four import sources, four temperaments

The first source is table files, usually CSV or Excel, and the most common one. Its temperament is freedom: headers may read "Customer Name," "Name," or the equivalent in another language; dates may appear as 2026/10/1 or 10-01-2026; numbers may carry currency symbols and thousand separators. Table import must do header mapping and format cleaning. It must never assume users filled in your template.

The second source is exports from old tools — a Notion, Trello, or Airtable export, often a zip archive or JSON file. Its temperament is nesting: a task carries comments, attachments, and subtasks, and the export may flatten the hierarchy or drop fields. Before building this path, list honestly what survives and what is lost, and tell the user. Silently dropping part of their data is worse than refusing the import.

The third source is a direct API connection: the user authorises your product to pull data from the old service. Its temperament is pagination and rate limits: large datasets time out, and API changes break field mappings. API imports must run as resumable, batched jobs — never as a single synchronous request.

The fourth source is pasted text: the user copies a table from a web page and pastes it in. Its temperament is the least controlled, but it is the friendliest path for small batches, and a good "move ten rows first" trial channel before a full import. Start with table files, which cover the most users, then expand to API connections.

Six design principles

First, preview before writing. After upload, the system should parse the first rows and show a mapping preview: file columns on the left, system fields on the right, with matched, unmatched, and skipped rows all visible. The user confirms the mapping before anything is written. This single step prevents most import errors.

Second, imports must be verifiable. Do not end with a bare "Import successful." Produce a report: rows read, rows succeeded, rows skipped, rows failed; failed rows listed with their original line numbers, original content, and reasons; successful data reachable in one click. Users need this report to answer one question: did all of my data actually arrive?

Third, imports must be reversible. Every import run gets a batch number recording which records were created or changed. If the import was wrong, the user can undo the whole batch and return to the pre-import state. An import without batching is surgery without a way back.

Fourth, re-importing must not duplicate data. Users often re-run an import because the first result disappointed them, or retry after a network failure. Deduplicate on business keys: a customer keyed by email or phone, a task keyed by its old system's unique ID. Re-importing the same file should yield zero new records and a set of skips, not doubled data.

Fifth, errors must be explained per row, in plain language. Never show users a raw "Validation failed on row 42." Write "Row 42: the date '2026-13-40' could not be read — please use a format like 2026-01-15." Include the original row content in the report so users can locate the problem without reopening the source file.

Sixth, large imports must run asynchronously and stay trackable. Beyond a few hundred rows, do not make users watch a loading page. Turn the import into a background job with a progress bar, an estimated remaining time, and a completion notice. If users close the page and return, the import record should still show the job's status and its report.

A seven-step import flow, from upload to verification

Step one: upload and format detection. Accept CSV, Excel, and pasted text. On upload, detect encoding and delimiters first: Chinese files often arrive in GBK encoding, Excel files carry merged cells, and CSV files carry quoted commas. Handle these in the parser rather than erroring and asking users to convert formats themselves. State the size limit plainly — for example, fifty thousand rows per run — and guide larger imports toward batching or the API path.

Step two: field mapping. The system guesses mappings from headers — "Customer Name," "Name," and equivalents all pointing at the customer name field — and leaves uncertain fields blank for the user to choose. The mapping screen must support three actions: change a mapping, set a default value, and mark a column as ignored. Defaults are powerful: if the file has no status column, default everything to "pending" instead of failing every row.

Step three: preflight validation. Before writing anything, validate every row lightly: required fields present, dates parseable, numbers in a sane range, business keys not duplicated within the file. Report results in three tiers: rows ready to import, rows with warnings that can still import, and rows that must be fixed. Users can fix the must-fix rows, or choose "import the passing rows and generate a fix-up file for the rest."

Step four: a small trial run. Never start with the full file. Run the first fifty rows through a real write, and show the trial report to the user. The trial exists so mapping and rule mistakes surface at a scale where cleanup costs almost nothing. Only after the trial passes does the user press "import all." That extra click is worth it.

Step five: full import with batch records. Write in batches, each stamped with the batch number, source filename, mapping snapshot, and operator. Use transactions or segmented commits so a mid-run failure never leaves half-old, half-new dirty data. Show live progress — rows processed, succeeded, failed — and let users leave and come back.

Step six: report and repair. When the run finishes, generate a report, and let failed rows be downloaded as a fix-up file that keeps the original content plus a failure-reason column. Users repair that file and re-import it; business-key deduplication prevents duplicates. Keep the report in the import history permanently, not as a flash message that disappears.

Step seven: verification and rollback. Give users a verification checklist: does the total match, do the distributions across a few key categories look right, do three spot-checked records have correct fields? If not, one click on the batch number undoes the import and restores the pre-import state. Log the rollback itself, so an accidental click is traceable. Only when all seven steps complete is an import truly closed.

Designing the field-mapping table

The mapping table deserves the most interface care in the whole feature. First, make automatic guesses transparent: show "auto-matched" where the system guessed, and let users correct a guess with one click, rather than hiding the guess and making users wonder what the system assumed. Second, build a synonym dictionary: pre-load common header variants for each field, and keep adding from user corrections, so guess accuracy improves with use.

Third, treat dates, numbers, and enums as special cases. Dates need multi-format recognition, and the preview must show parsed results so users can spot mis-parsed dates at a glance. Numbers need currency symbols and thousand separators stripped, with decimal places declared. Enum values such as task status need value mapping: "in progress," "doing," and local equivalents in a user's file should all be mappable to one system status. Fourth, save mappings as templates: next time users import a similar file, they reuse the last mapping instead of reconfiguring everything. Keep templates per source type — a "customer table template" and an "order table template" — not one vague global template.

Resumable API imports: checkpoints, not restarts

Once table import works, the next step is usually direct API import: after authorisation, the system pulls data from the old service. The key difference from file import is uncontrollable timing: the old service may rate-limit, return tens of thousands of pages, or fail midway. Building API import as one synchronous request guarantees a page timeout with nobody knowing how many rows actually landed.

Split API import into three stages — fetch, transform, write — each with its own cursor. Fetch page by page, persist each page of raw data, and record the next-page cursor. Transform row by row under the mapping rules, quarantining failures without blocking the batch. Write in batches, updating progress after each committed batch. If any stage interrupts, the next run resumes from the last successful cursor instead of starting over. The resume button belongs in the import history, prominent, needing no re-authorisation and no re-mapping.

Respect rate limits and quotas from the start. Back off exponentially on rate-limit responses instead of retrying immediately and getting the account blocked; control write concurrency so a flood of writes does not overwhelm your own database. Show three live numbers in the import record — fetched, transformed, written — so users can see the job is alive. Watch for field drift too: when the old service renames a field, the mapping template should flag "this field matched last time but not now" and ask the user to confirm, rather than silently dropping the column.

Finally, handle authorisation credentials with care: encrypt them, scope them to read-only access for the import job, and either retain or revoke them after the job according to the user's choice — stated plainly. Never ask for broader permissions than reading data. Tell users exactly whether the credential is kept or revoked, and where they can revoke it manually. When users hand you their old tool's data, the boundaries of trust must be explicit; a vague authorisation note damages trust more than a failed import.

The first week after import: do not let new data rot

Import completion is not the finish line; the following week decides whether the user stays. First, mark imported records with a "recently imported" filter or view, so users can find and review them quickly in week one and fix them in bulk. Second, prompt a verification pass within three days — a homepage note such as "You imported 4,985 records last week and have reviewed 120" — rather than never mentioning the import again. Third, learn from the corrections users make during verification: which fields get bulk-edited, which categories get merged. Those corrections are the training data for the next version of the mapping templates. An import feature's reputation is not built on import day; it is built in the week users discover that what they moved actually works, and that mistakes can actually be fixed.

Anti-patterns to avoid

• Writing to the database on upload, with no preview or confirmation: one wrong file selection, and dirty data is already in.
• Reporting only a failure count, with no row-level reasons: users learn that 37 rows failed but not which rows or why, so they re-import everything blindly.
• No batch concept, no undo: a wrong import can only be cleaned up record by record, and users simply give up.
• Duplicate data on re-import: without business-key deduplication, one retry doubles the data and collapses trust.
• Raw technical errors as user messages: users cannot act on "foreign key constraint." They need the row, the field, and the fix.
• Synchronous large imports that hang and time out: users cannot see progress, and a refresh may cause duplicate writes.

Pre-launch checklist

• Upload shows a mapping preview, and nothing is written before user confirmation.
• Preflight validation has three tiers — ready, warning, must-fix — and failed rows download as a fix-up file.
• A fifty-row trial run passes before full import is offered.
• Every import has a batch number and a mapping snapshot, and supports one-click batch undo.
• Business-key deduplication is implemented; re-importing the same file adds zero records.
• Error reports are per-row, in plain language, with original row content and fix suggestions.
• Large imports run as background jobs with progress, completion notices, and a searchable import history.
• Date, number, and enum cleaning rules are visible in the preview.
• Mapping templates can be saved and reused, separated by source type.
• The feature has been tested with a genuinely dirty file: blank rows, duplicate keys, two date formats, and unexpected category names.

A worked example: an expense app

Imagine a user exporting five thousand rows from an old expense app as CSV, with headers for date, category, amount, and note. Dates mix 2026/1/5 and 01-05-2026 formats; categories include both "dining" and the user's own "eating out" and "takeaway." The flow runs like this: upload auto-maps the four columns; preflight flags three unparseable dates and twelve amounts carrying currency symbols; a fifty-row trial prompts "should 'eating out' and 'takeaway' both map to dining?" and the user confirms the merge; the full import reports 4,985 successes and 3 failures; the user downloads the fix-up file, repairs three rows, re-imports with zero duplicates; verification shows a sensible dining share, and the import is done. At no point did the user open a database or hand-edit the source file. That is what an import feature should feel like.

One closing note: after launch, the most valuable feedback comes from users' real dirty files. Aggregate import failure reasons — which fields fail most, which formats appear most, which mappings get corrected most — and let those numbers set the next round of improvements. Import is not a one-off feature but a pipeline that gets sharper with use: more templates, better guesses, smoother moves. That is how a product earns its reputation, one migration at a time.

A note on privacy during migration

Migration files often contain other people's personal data: customer names, email addresses, phone numbers, order histories. Treat the uploaded file as sensitive from the moment it arrives. Store it encrypted, restrict access to the importing user and the background job, and delete the original file once the import and its fix-up cycle are complete — keeping only the report summary, not the raw rows. If the import fails midway, the retained raw file should still expire automatically after a short window, such as seven days. Tell users this retention rule in the import screen itself, not buried in a privacy policy, because the users doing the migrating are the ones accountable to their own customers for how that data is handled.