A product team is about to ship a feature that lets users export their entire account as a large file. How would you design the upload-and-download path so a multi-gigabyte file never streams through your application servers?

system-design · Senior level · software-engineering

What the interviewer is really asking

Probes whether the candidate keeps large-file transfer off the application tier using pre-signed/direct-to-object-storage uploads and signed download URLs, handles multipart/resumable transfer and access control, rather than proxying gigabytes of bytes through stateless app servers.

What to say

• State the anti-pattern you're avoiding: routing a multi-gigabyte file through the app servers ties up a request thread and memory for the whole transfer, caps you at the slowest client, and makes autoscaling miserable. The app tier should broker access, not carry the bytes.
• Use direct-to-object-storage transfer with pre-signed URLs: the client asks the app for a short-lived pre-signed PUT (upload) or GET (download) URL scoped to one object and a few minutes, then transfers bytes straight to/from S3/GCS. The app only ever handles the small control request, never the payload. For large uploads use multipart upload so parts go in parallel and a failed part retries without restarting; for downloads, serve through a CDN with range requests so clients can resume.
• Keep access control and integrity on the broker: the app authorizes the request, sets the object key, ties it to the user, and records the export job; it validates content-length and type limits in the pre-sign policy. For generated exports, do the heavy work async (a worker writes the file to object storage and the user gets a signed link when it's ready), not inline in the request.

What to avoid

• Have the client POST the whole file to an app endpoint that buffers it and forwards it to storage, holding a worker and gigabytes of memory for the duration of the transfer.
• Hand out a permanent or public object URL instead of a short-lived signed one, leaking access to anyone who sees the link.
• Generate a huge export synchronously inside the HTTP request and stream it back, so the request times out and the app server is pinned for minutes per export.

Example answers

Strong: For uploads I'd have the client request a pre-signed PUT URL from the app, scoped to a specific key under that user's prefix and valid for a few minutes. The bytes go directly from client to S3 — my app never touches them — and for multi-gig files I'd use multipart upload so parts upload in parallel and a dropped part retries on its own. The app just records the upload job and the resulting object key against the user.

Weak: I'd add an endpoint that accepts the file upload, holds it in memory or a temp file, and then writes it to S3 from the server. It's straightforward and the server controls the whole flow.