A 2 GB Video Fails at 99% and Starts Over? Chunked Upload and Resumable Transfer in Practice

A 2 GB Video Fails at 99% and Has to Start Over? Chunked Uploads Exist for Exactly This

Almost everyone building a content or tool vibe project hits the same wall: a user tries to upload a video of several hundred megabytes, a design source file, or a data package. The progress bar reaches 99%, the network hiccups, the upload fails, and everything resets to zero. The user swears once and never comes back. The problem is not the user's network or the server's bandwidth. It is the upload method itself: one HTTP request carrying the whole file means any single interruption throws away all progress. Chunked uploading changes this "all or nothing" into "only lose one small piece", and resumable uploading lets an interrupted transfer continue from the piece that broke instead of starting over.

This guide shows how to build a working chunked upload with resume support for a vibe project, covering chunk design, the frontend implementation, server-side merging, instant upload, failure handling, and security. The boundary first: our existing file upload guide covers basic uploads without letting image uploads overwhelm the server; the file storage and CDN guide covers presigned direct uploads and the three-layer storage architecture; the image processing pipeline guide covers what happens to images after upload. This guide covers only chunking, resuming, and merging for large files, and does not repeat basic upload content. All examples below are fictional teaching examples; app names, numbers, and code snippets are simplified versions constructed for explanation.

1. When Do You Need Chunking? Do the Math First

Chunking is not free. It brings chunk records, a merge step, cleanup jobs, and more failure modes. Using it for small files is asking for trouble. Do three sums before building.

Scenario	File size	Recommended approach
Avatars, illustrations, ordinary attachments	Under 10 MB	A single upload is enough; skip chunking
Short videos, design files, data packages	10 MB–500 MB	Chunked upload with resume support
Long videos, backup archives, datasets	Over 500 MB	Chunked upload + instant upload + background merge
Weak networks, mobile, cross-border uploads	Any size	Smaller chunks, lower concurrency

Before deciding, think through the extra cost chunking brings. It adds at least three things that need long-term maintenance: upload session records on the server, temporary storage for chunks, and a scheduled job that cleans up orphaned data. On a one-person team, that maintenance belongs in the schedule — not just the days spent writing code. If the largest file in your product stays under 50 MB, or users almost always upload on stable broadband, loosening the single-upload timeout and adding one whole-file retry is often cheaper than chunking. Chunking is for large files and weak networks, not the default answer for every upload.

Two signals mean you have waited long enough. First, support messages or feedback mention "upload failed" or "it vanished halfway", concentrated on large files. Second, server logs show many upload requests that get close to finishing and then abort — bandwidth spent, not one file landed. Either signal alone is enough to start.

2. Designing the Chunk Protocol: Five Decisions, Made Once

Before writing code, settle the protocol. It is small, but five decisions made wrong will haunt you with patches everywhere.

First, chunk size. The common choices are 5 MB or 10 MB. Chunks too small explode the request count — a 2 GB file cut into 1 MB pieces needs 2,000 requests, and handshakes alone drag the speed down. Chunks too big shrink the benefit of resuming, since one break costs a large re-upload. Start with 5 MB, drop to 2 MB for weak networks, raise to 10 MB for intranet or dedicated lines. Make chunk size configurable; do not hard-code it.

Second, chunk numbering. Use consecutive integers starting from 0. The number is what merge ordering relies on, and what resume uses to decide which chunks are already there. Do not use timestamps or random values as chunk identifiers — sorting and reconciliation both get painful.

Third, an upload session identifier. One upload task needs an uploadId, issued by the server at initialization and carried by every chunk request after that. The uploadId doubles as the directory name or key prefix where the server stores temporary chunks. When the upload completes or expires, the server cleans up temporary data by uploadId.

Fourth, verification. Every chunk needs a checksum, usually MD5 or SHA-256. The frontend computes the chunk hash and sends it with the request; the server verifies it after receiving the chunk and asks for a re-upload of that one chunk if it does not match. The whole file needs a total hash too, verified once after merging, to catch the case where every chunk is fine but the assembled file is wrong. Compute hashes in a Web Worker, not on the main thread.

Fifth, a status query endpoint. Before resuming, the frontend must ask the server "which chunks have you already received for this uploadId", and only send the missing ones. This endpoint is the heart of resumable upload. Without it, there is no way to resume. The server just returns the list of received chunk numbers for the uploadId.

3. Frontend: Slicing, Concurrency, Progress, Pause

The core of the frontend is the File object's slice method, supported natively by browsers, no library needed. Cut the file into chunks, upload them, and control concurrency. Three to five concurrent uploads is the usual range: too many compete for bandwidth and get slower, too few leave speed on the table. Compute the progress bar from "bytes in chunks the server has confirmed / total file bytes", not from "requests sent", or failed retries will make the bar go backwards or past 100%.

// Minimal skeleton of chunked upload on the frontend (simplified example)
const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB
async function uploadFile(file) {
  const { uploadId } = await initUpload({ name: file.name, size: file.size });
  const total = Math.ceil(file.size / CHUNK_SIZE);
  const done = new Set(await queryUploaded(uploadId)); // resume: ask which chunks exist
  const queue = [];
  for (let i = 0; i < total; i++) {
    if (done.has(i)) continue;
    queue.push(i);
  }
  await runWithConcurrency(queue, 4, async (index) => {
    const chunk = file.slice(index * CHUNK_SIZE, (index + 1) * CHUNK_SIZE);
    await uploadChunk(uploadId, index, chunk); // retries 3 times internally on failure
  });
  await completeUpload(uploadId, total);
}

Pause, cancel, and resume are three faces of the same mechanism. Pause means stop sending new chunk requests and let in-flight ones finish. Cancel is pause plus telling the server to delete the temporary chunks. Resume means re-initializing the same task, asking which chunks were received, and sending only the missing ones. The key detail is surviving a page refresh: store the uploadId, file name, file size, and total hash in localStorage, so when the user picks the same file again, the total hash matches an existing upload task and it resumes directly instead of creating a new one.

Three experience details matter. First, do not lock the page during upload; let the user browse elsewhere, with the upload state in a persistent small floating bar. Second, validate locally before a large upload starts: file type, size limit, a disk-space hint — reject what fails locally instead of erroring halfway through. Third, be honest in the progress display: show uploaded bytes, total bytes, current speed, and estimated remaining time together. Users can tolerate slow; they cannot tolerate not knowing how long is left.

4. Server: Receiving, Merging, and Cleaning Up

The server has four jobs: initialize, receive chunks, merge, and clean up. At initialization it generates the uploadId, records the file name, total size, chunk count, and total hash, and returns them to the frontend. When receiving a chunk, it stores it by uploadId and number, verifies the chunk hash, and returns an error for a re-upload of that one chunk if the hash does not match — do not keep a bad chunk around. At merge time it concatenates all chunks in number order, verifies the total hash, and only then moves the file to permanent storage and updates the database record. Cleanup is the last step and the easiest to forget: delete temporary chunks after merging, and delete them when an upload task expires too.

There are two ways to store chunks temporarily. One is object storage, using the uploadId as a key prefix, each chunk a small object; at merge time the server reads them back and concatenates them, or uses the object storage service's own multipart upload capability directly. The other is the server's local disk, fine for a small team starting out, but watch disk space and multi-instance deployment: with several service instances, chunks can land on different machines and the merge step will not find them. Local disk is fine for a single machine; once you scale to multiple instances, switch to object storage.

Merging is the most fragile part of the whole flow, so write it as an operation that must be idempotent and retryable. The merge request can be sent twice by the frontend because of a network problem, and the server must guarantee a duplicate merge never produces two files. If merging breaks halfway, it must be able to continue from the break or redo safely, rather than leaving a half-written file marked as successful. The safest pattern is to merge to a temporary location, verify the total hash, then atomically move it to its permanent location, and only then update the database status.

5. Instant Upload: If the File Is Already on the Server, Do Not Send It Again

Instant upload comes free with chunked upload. At initialization the frontend sends the file's total hash; the server checks whether a file with that hash already exists, and if so returns success immediately — the frontend is done in seconds. This is especially useful for team collaboration, where the same asset gets uploaded repeatedly by different people and never needs sending twice.

Two boundaries need stating. First, instant upload must not bypass permission checks: a file uploaded by user A can only be instantly uploaded by user B when B has permission to access it — a matching hash must not hand someone else's file over. Second, instant upload must not bypass content review: whatever review flow a file goes through on first upload, an instantly uploaded file goes through too; "this file was reviewed before" is not a reason to skip it. Think of instant upload as "skip the transfer", not "skip every check", and neither trap will catch you.

6. Failure Handling: Retries, Expiry, and Consistency

Chunked upload has far more failure modes than a single upload, and they need handling on three layers. The first layer is a single chunk failing: network jitter, a server 500, a timeout. The frontend retries these — three attempts with exponential backoff — and only reports an error to the user after all three fail. A retry re-sends only the failed chunk, never the whole file.

The second layer is session expiry: the user uploads halfway, goes to lunch, and comes back to find the uploadId expired. The fix is a reasonable session lifetime, say 24 hours; after expiry the frontend re-initializes, but chunks already uploaded can be recovered by total hash and reused instead of forcing a restart from zero. Put the lifetime in the product copy so users know, instead of leaving them to guess.

The third layer is orphaned data: a cancelled upload, a crashed browser, or a failed merge all leave temporary chunks on the server that nobody wants. The fix is a scheduled cleanup job that scans once a day, deletes temporary chunks past their lifetime with no corresponding permanent file, and logs how much it cleaned. A chunked upload system without a cleanup job will eventually have its disk filled by orphaned chunks.

7. Security and Cost: Do Not Let the Upload Endpoint Become Someone Else's Free Cloud Drive

The upload endpoint is one of the easiest to abuse, and chunked upload makes abuse more expensive because it supports large files. Four checks are all required. First, authentication: the initialization endpoint must verify login state and upload permission — an anonymous user should not even be able to get an uploadId. Second, size and type limits: a total size cap, a chunk size cap, and a file type allowlist, all verified again on the server — frontend validation is only an experience optimization. Third, rate and quota limits: throttle upload speed and daily upload volume per user so one person cannot saturate the bandwidth. Fourth, content scanning: a merged file goes through virus scanning or content review before being marked usable.

On cost, chunked upload mainly adds request count and temporary storage. Request count can be pushed down with a slightly larger chunk size, and temporary storage is controlled by the cleanup job. Also mind how object storage is billed: some services charge per request, and chunks too small can make request fees exceed storage fees. Before launch, work out "what does one complete 2 GB upload cost" and write it in the internal docs — do not discover it from the bill.

8. A Complete Walkthrough: From the Upload Button to Playable

Let us string the previous seven sections together into one complete flow. This is a fictional teaching example: the fictional app "MicroLesson Box" is a small online course site, and an instructor is uploading a 1.8 GB recorded lesson over ordinary home broadband, with one network outage in the middle.

Step one: the instructor picks the file, and the frontend validates locally — it is an MP4 within the size limit — then computes the total hash and initializes with the server, receiving an uploadId. The server checks the total hash, finds nobody has uploaded this file before, creates the upload task, and returns the uploadId and a 5 MB chunk size. Step two: the frontend cuts the file into about 360 chunks and sends them at a concurrency of 4, with the floating bar showing uploaded bytes, speed, and remaining time live. At chunk 200 the home network drops; the frontend retries the failed chunk three times, still fails, stops, and shows "Network disconnected — will resume automatically when it returns". Step three: when the network recovers, the frontend first asks the server which chunks it has, confirms the first 199 are all there, and only sends chunk 200 onward — nothing already uploaded is sent again. Step four: with all 360 chunks in, the frontend sends the merge request; the server concatenates in number order, verifies the total hash, moves the file to permanent storage, triggers the transcoding job, and finally marks the lesson "playable". The instructor clicked upload once, the outage needed no attention, and even a page refresh would have resumed.

The easiest step to overlook in this story is step three: "ask first, then send". Many first implementations simply restart from zero after an outage. It appears to work, but a user on a weak network uploading a 2 GB file ends up sending it three or four times; both experience and cost lose to the small extra endpoint that asks which chunks were received and only sends the missing ones. The entire value of resumable upload lives in that small endpoint.

9. Pre-Launch Checklist

Walk this checklist item by item before shipping chunked upload. One, chunk size, concurrency, and retry count are all configurable. Two, after a network drop and a page refresh, picking the same file resumes. Three, the pause, cancel, and resume buttons have all been tested for real. Four, merging is idempotent — a duplicate submission never produces a duplicate file. Five, the total hash is verified after merging, and a failure never marks the upload successful. Six, the expiry cleanup job is configured and orphaned chunks get deleted on schedule. Seven, instant upload passes permission and review checks and bypasses neither. Eight, upload speed, quota, and file type limits are enforced on the server. Nine, the upload bar stays visible after navigating pages, and the progress numbers are honest. Ten, a real 2 GB file has been run through the full flow once on a weak network.

Chunked upload is not deep magic. Its entire value is one sentence: it turns large-file uploading from "one shot decides everything" into "each failure only loses one small piece". Settle the five protocol decisions once, write the frontend around slicing, concurrency, progress, and pause, make the server solid on initialize, receive, merge, and clean up, and add the three layers of failure handling — retries, expiry, orphan cleanup. A one-person team can ship a usable version in a week. Start with 5 MB chunks, 4 concurrent uploads, and 3 retries, then tune from real data after launch — far cheaper than chasing perfect parameters before shipping.