A media worker needs enough temporary storage for the files that coexist at its busiest stage, not just the final output. Before starting another download, account for its input, intermediate files, output and space already promised to active work. Keep a reserve for unrelated filesystem growth and stop admitting work when the remaining budget is insufficient.
This is a capacity-planning task for your application. Tornado can deliver media for downstream processing, but a completed remote job does not reserve disk space on your worker. The approach below helps you decide when to fetch and process that delivery locally.
Measure the peak, not only the finished file
Walk through one representative job and list what remains on disk at each stage. For example, your downloader may keep a partial file while receiving bytes. Your processor may retain the input while producing a separate output. Your uploader may need that output until remote storage confirms receipt. Failed attempts may leave additional files behind.
A synthetic planning example makes the difference clear:
| Stage | Files kept locally | Illustrative occupied space |
|---|---|---|
| Download finished | Input | 2 GiB |
| Transform running | Input, scratch files, growing output | Up to 6 GiB |
| Upload running | Input and finished output | 4 GiB |
| Cleanup finished | No local media retained | 0 GiB |
These are invented sizes for explaining the calculation, not measurements of Tornado or a recommended sizing ratio. Your actual processor may stream, copy, expand or retain different files. Measure the workflow you run, including failed attempts, before choosing a per-job allowance.
Do not use billable API volume as a disk estimate. Billing and physical storage answer different questions; the usage guide explains that distinction. Likewise, a delivered file's byte count cannot describe intermediates that your later processing creates.
Inspect the filesystem your worker actually writes to
Python's shutil.disk_usage reference returns total, used and free bytes for a path. Inspect the existing staging directory, rather than assuming that your home directory, container root and media volume share a capacity pool.
If input and output use different volumes, budget each volume separately. A single sum across independent disks can hide a full destination. Check applicable quotas and runtime limits too: reported filesystem free space does not necessarily equal the amount your application is permitted to consume.
For Kubernetes users, the local ephemeral storage documentation describes storage accounting and eviction. It also distinguishes memory-backed emptyDir usage from local ephemeral storage. This is a deployment-specific consideration, not a claim about Tornado's infrastructure.
Calculate a conservative admission count
Use a current free-space snapshot and subtract two things: your operational reserve, and the additional bytes still expected from jobs already admitted. Bytes those jobs have already written are already reflected in free space; subtracting their entire original allowance again would double-count that portion.
Divide the remaining budget by the estimated peak additional allocation of one new job. The following pure Python helper calculates that count for a homogeneous workload. All sizes are integer bytes. It deliberately rejects zero-sized job estimates and booleans, which Python otherwise treats as integers.
def available_slots(free_bytes, reserve_bytes, outstanding_bytes, job_peak_bytes):
values = (free_bytes, reserve_bytes, outstanding_bytes, job_peak_bytes)
if any(type(value) is not int or value < 0 for value in values):
raise ValueError("Use nonnegative integer byte counts")
if job_peak_bytes == 0:
raise ValueError("A new job needs a positive allocation estimate")
budget = max(0, free_bytes - reserve_bytes - outstanding_bytes)
return budget // job_peak_bytes
GiB = 1024 ** 3
# Synthetic snapshot: 20 GiB free, 4 reserved, 4 still promised.
assert available_slots(20 * GiB, 4 * GiB, 4 * GiB, 6 * GiB) == 2
assert available_slots(19 * GiB, 4 * GiB, 4 * GiB, 6 * GiB) == 1
assert available_slots(8 * GiB, 4 * GiB, 4 * GiB, 6 * GiB) == 0
assert available_slots(3 * GiB, 4 * GiB, 0, 6 * GiB) == 0
The four assertions were run locally with synthetic values. Additional checks covered exact byte boundaries and invalid inputs. This validates arithmetic, not a live storage limit or a production scheduler.
Treat the result as a planning decision, not a reservation. Two processes can read the same free-space value and both decide to start. A real scheduler needs coordinated claims and accounting updates, plus monitoring while work runs. The concurrency and backpressure guide explains where admission belongs in the wider pipeline.
If jobs have different sizes, calculate each proposed job's allowance instead of applying one average to everything. Unknown sizes should enter a deliberately bounded lane or wait for better information. They should not become zero-byte estimates. No estimate prevents an unexpected expansion, so stop safely when actual usage exceeds the allocation.
Release capacity only when files are gone
Keep each attempt's temporary files in an application-owned directory associated with its job and attempt IDs. Record whether a worker still owns that attempt. Cleanup must not remove files another process is reading, uploading or resuming.
On success, confirm the durable handoff before removing the only local copy. On failure, decide whether a partial file is useful for recovery or should be cleaned up. Bound the retention of abandoned attempts so a succession of failures cannot consume the whole volume. Do not assume that marking a task finished also deletes its files or immediately restores capacity.
After cleanup, reconcile accounting with current filesystem observations. Keep outstanding reservations conservative until the corresponding work has ended. A cleanup process that only subtracts an estimated number from a database can report capacity that the filesystem has not actually recovered.
Separate remote completion from local readiness
Use Tornado's job status reference to follow the remote job and obtain its delivery information. Keep the remote job ID while your application waits for local capacity. A full staging disk is not evidence that the remote job failed, and it is not by itself a reason to submit the source again.
Queue the downstream handoff separately. Before a delayed worker fetches the file, confirm that its access path is still usable. The expired-link guide helps distinguish an access problem from a missing artifact. Where the downstream service accepts a supported direct handoff, evaluate whether an unnecessary local copy can be removed from your design; confirm the consumer's actual behavior before relying on streaming.
Start with one complete media workflow. Measure the largest set of coexisting files, exercise an interrupted attempt, and confirm cleanup before raising concurrency. Then use the capacity calculation as one gate alongside CPU, memory, downstream rate limits and queue length.