Save media downloads atomically in Python before processing

Write a media download to a separate temporary file, validate it, and only then replace the path that downstream workers read. This gives your application a clear handoff: readers of the final path see a finished version instead of a file growing one chunk at a time.

The distinction matters when a transcription worker, thumbnail generator or importer starts as soon as a file appears. Creating video.mp4 at the beginning of a transfer can wake that consumer before the download finishes. A filename and a remote job's completed status do not prove local readiness.

Keep remote completion and local readiness separate

Tornado's job-status reference describes the job state and delivery information, including s3_url when available. Keep the job ID while your application fetches that delivery. A failed local write is not, by itself, a reason to create another remote job.

Model the local transfer separately: receiving, validating, ready or failed. Consumers should use only a committed path or an explicit ready record. They should not scan every temporary file and infer readiness from an extension.

For interrupted HTTP transfers, see resuming a media download. That guide covers which remote bytes may be appended. This guide covers a different boundary: making a complete local artifact visible to the next stage.

Stage beside the destination

Create the temporary file in the final path's directory, under an application-owned location. Python's NamedTemporaryFile documentation explains how to obtain a unique temporary filename and control cleanup. Using a unique attempt file avoids two writers sharing one .part path.

Keep staging and destination on the same filesystem. The os.replace reference states that replacement can fail across filesystems and that a successful rename is atomic. It also replaces an existing file when permissions allow. That overwrite behavior must be intentional.

Use one writer per destination, or assign a separate destination to each immutable revision. Atomic replacement does not choose which of two competing revisions is correct. Do not accept arbitrary client-supplied paths into this helper; resolve your own artifact identifiers into an application-owned directory first.

A bounded Python handoff example

The helper below accepts an iterator of byte chunks, an expected positive byte count and a validation callback. It deliberately contains no HTTP client. Your downloader must first validate the response and establish which representation and expected size those chunks belong to.

The callback receives a closed temporary file and must raise an exception if it is unacceptable. It must only inspect the file, not modify or move it. A successful return permits replacement.

import os
from pathlib import Path
from tempfile import NamedTemporaryFile


def commit_media(chunks, destination, expected_bytes, validate):
    if type(expected_bytes) is not int or expected_bytes <= 0:
        raise ValueError("Expected size must be a positive integer")
    destination = Path(destination)
    temporary = None
    try:
        with NamedTemporaryFile(
            mode="wb", dir=destination.parent,
            prefix=".media-", suffix=".part", delete=False,
        ) as output:
            temporary = Path(output.name)
            received = 0
            for chunk in chunks:
                if not isinstance(chunk, bytes):
                    raise TypeError("Chunks must be bytes")
                received += len(chunk)
                if received > expected_bytes:
                    raise ValueError("Transfer exceeds expected size")
                output.write(chunk)
            if received != expected_bytes:
                raise ValueError("Transfer is incomplete")
            output.flush()
            os.fsync(output.fileno())
        validate(temporary)
        os.replace(temporary, destination)
        temporary = None
        return destination
    finally:
        if temporary is not None:
            temporary.unlink(missing_ok=True)

The destination directory must already exist. The size check bounds bytes written by this function, but not memory allocated by the caller: provide reasonably sized chunks rather than loading an entire video first. Unknown-size transfers need a different, explicitly bounded policy; do not pretend that an unknown size is zero.

The helper closes the file before validation and replacement. It flushes the Python buffer before calling fsync, following the Python file synchronization guidance. This is not a complete power-loss durability protocol: it does not synchronize the parent directory or establish the guarantees of your actual filesystem and storage stack.

Validate what your consumer needs

An equal byte count detects a truncated or oversized transfer under the agreed representation. It does not prove that a file is a usable video. For example, an unexpected error response can have a plausible length.

Where a trusted expected checksum exists, compare it before replacement using the approach in SHA-256 media verification. Add the media checks your importer requires, such as parsing the container or checking streams. The container and codec guide explains why an extension alone is insufficient.

Do not mark an artifact ready if the validator exits unsuccessfully, times out or cannot inspect the file. Keep validation failures distinguishable from network and disk failures. For a known good synthetic byte fixture, a callback can compare the staged bytes with the expected bytes; that tests this handoff mechanism without claiming to validate a real codec.

Exercise the failure boundaries

Run the exact helper against temporary test directories before integrating it. Useful cases are:

Local tests for this article exercised successful creation and replacement, short and oversized input, an interrupted iterator, invalid arguments, a rejecting validator and a simulated replacement failure. They also inspected the final path during staging and validation. These were synthetic local filesystem tests, not a production download, a concurrent filesystem stress test or a power-loss test.

The finally block handles ordinary exceptions, not every possible process termination. A killed worker can leave its temporary file behind. Cleanup itself can also fail, for example because permissions changed; monitor and reconcile abandoned attempts instead of assuming every failure frees space. The temporary disk planning guide covers capacity and ownership considerations.

Commit the downstream record deliberately

After this helper succeeds, record the artifact revision and enqueue the consumer. A crash between file replacement and the database update can leave a complete file without a ready record. Conversely, writing the record first can expose a nonexistent or unfinished file. A rename and a database transaction are not one atomic transaction.

Use stable artifact identifiers, an idempotent reconciliation process and a revision-aware consumer. Our duplicate event and local outbox guide covers reliable event delivery; it does not make filesystem writes part of a database commit.

On retry, inspect the recorded revision and validate any existing final artifact before deciding whether another transfer is needed. Preserve the distinction between acquiring the media remotely, downloading it locally and successfully processing it downstream.

Build your first Tornado media workflow, then add this local handoff wherever a consumer reads files from disk. Start with one owned destination and an interrupted-transfer test before increasing worker concurrency.