Limit media download size while streaming in Python

Enforce a media download size limit by counting the bytes your application actually accepts while streaming. Use an announced size to reject obviously oversized transfers early, but keep a running limit even when that size is missing. Stop before writing the chunk that would exceed your policy, and leave the artifact unready.

This guide implements a small Python binary-copy boundary. It applies to the file your worker receives, not to Tornado billing units or a server-side upload limit.

Define which bytes the limit covers

Choose the boundary before choosing a number. A limit can describe bytes received over HTTP, bytes produced after HTTP content decoding, bytes stored in a file or space occupied by all concurrent jobs. Those quantities need not be identical.

The HTTP Content-Length field describes message content length under the protocol's rules. It is not a reservation of local disk space. If your client decodes an HTTP content encoding, the bytes exposed to your copy loop may differ from the encoded content length. Check the client's stream behavior before comparing counts.

For the example below, the policy covers the bytes returned by a binary reader and passed to a fresh temporary output. It does not cover decoder buffers, filesystem overhead or other workers. If you need a limit on decoded media frames or samples, that belongs to the media-processing stage as well.

Keep job success separate from local admission

Tornado's job-status reference describes job identity, status and delivery fields. Save the job ID and identify the delivered artifact before starting your local transfer. A successful remote job does not mean your worker has admitted the file within its own storage policy.

If the artifact exceeds your application's limit, record a local size-policy outcome. Do not overwrite the remote job state with a guessed failure or automatically recreate the job. A repeated download of the same artifact will not make it smaller.

The temporary disk planning guide covers admission across concurrent work. The check here is the enforcement point inside one admitted transfer.

Copy with a bounded read and a running counter

Use a blocking binary source whose read(n) returns at most n bytes and returns b"" only at end of stream. Use an empty binary destination reserved for this attempt. Python's binary I/O reference describes these stream operations.

class DownloadTooLarge(ValueError):
    pass


def copy_with_limit(source, destination, max_bytes):
    if type(max_bytes) is not int or max_bytes < 0:
        raise ValueError("max_bytes must be a non-negative integer")
    total = 0
    while True:
        requested = min(64 * 1024, max_bytes - total + 1)
        chunk = source.read(requested)
        if not isinstance(chunk, bytes) or len(chunk) > requested:
            raise TypeError("source must obey the binary read contract")
        if not chunk:
            return total
        if len(chunk) > max_bytes - total:
            raise DownloadTooLarge("artifact exceeds local byte limit")
        if destination.write(chunk) != len(chunk):
            raise OSError("incomplete destination write")
        total += len(chunk)

The extra byte in the requested amount distinguishes an artifact exactly at the limit from one exceeding it. When the counter reaches the limit, the next read requests one byte: EOF means the copy fits; another byte means rejection. The offending chunk is not written. Earlier chunks may already be present, so an exception is not a rollback.

This is a synchronous copy helper, not an HTTP client. It does not fetch a URL, manage authentication or validate media. A reader that violates the bounded-read contract can allocate excessive memory before the helper sees the result; the type check does not make an arbitrary adapter trustworthy.

Handle the boundary cases deliberately

A file exactly at the limit is allowed by this size policy. An empty file is also allowed, including with a zero-byte limit; your media validator should reject emptiness when a real audio/video artifact is required. Keep those two decisions explicit rather than hiding format validation inside a size counter.

A short read is not necessarily EOF. The loop continues until it receives the empty byte string. Non-blocking streams that return None need an adapter with appropriate waiting behavior; this example rejects them instead of pretending the transfer finished.

The helper refuses partial destination writes rather than retrying them. Treat any write exception or mismatch as a failed attempt. The caller owns stream closure, flushing, durability and temporary-file cleanup. Only publish a final path after all required steps succeed, using the atomic handoff guide.

Use headers for early decisions, then count anyway

If a trustworthy parsed size already exceeds the policy at the same byte boundary, reject before downloading the body. If no size is available, stream under the running limit. Do not read the entire response into memory just to call len() afterward: by then the allocation has already happened.

A size that fits does not prove the response is media. Combine the limit with response screening and container validation. Likewise, a stream may stay below the limit while never ending, so configure read and attempt timeouts too.

For resumed transfers, this counter would cover only bytes copied by this invocation. Account for bytes already retained and validate the artifact revision before appending. Do not reset a full-file limit for every resumed segment. The download resume guide explains the additional range and identity checks.

Test the counter independently of the network

The exact published helper was exercised locally with in-memory binary streams and a temporary disk file. Checks covered files below, at and above the limit, multiple reads, short reads, invalid limits, a source contract violation, a partial write and a read exception.

These tests establish copy behavior with synthetic inputs. They do not validate an HTTP library's buffering, decompression limits, filesystem exhaustion, parallel admission or a live Tornado delivery. Test those separately with your real stream adapter. A 64 KiB read request is a sample implementation choice, not a measured throughput optimum or a bound on total process memory.

Record the artifact ID, configured limit and local failure category. Keep partial output out of the ready queue and release its storage reservation after cleanup. Build your first Tornado media workflow, then enforce the consumer's size policy at the transfer boundary so an unexpected artifact cannot silently consume its entire storage budget.