Webhooks Deep Dive: Real-Time Job Notifications

Polling for job status wastes resources and adds latency. Tornado webhooks deliver real-time notifications the moment your jobs complete, enabling truly async workflows.

Setting Up Webhooks

Include a webhook_url in your job request:

curl -X POST "https://api.tornadoapi.io/jobs" \
  -H "x-api-key: sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://youtube.com/watch?v=...",
    "webhook_url": "https://your-app.com/webhooks/tornado"
  }'

Webhook Payloads

Job Completed Successfully

{
  "type": "job_completed",
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "Completed",
  "url": "https://youtube.com/watch?v=...",
  "s3_key": "videos/my-video.mp4",
  "s3_url": "https://cdn.example.com/videos/my-video.mp4?signature=...",
  "file_size_bytes": 104857600
}

Job Failed

{
  "type": "job_completed",
  "job_id": "550e8400-...",
  "status": "Failed",
  "url": "https://youtube.com/watch?v=...",
  "error": "Video unavailable"
}

Progress Webhooks

For long downloads, enable progress webhooks to track each processing stage:

{
    "url": "https://youtube.com/watch?v=...",
    "webhook_url": "https://your-app.com/webhook",
    "enable_progress_webhook": true
}

You'll receive updates at each stage:

Handling Webhooks

Node.js (Express)

app.post('/webhooks/tornado', (req, res) => {
  const { type, job_id, status, s3_url, error } = req.body;

  if (type === 'job_completed') {
    if (status === 'Completed') {
      console.log(`Job ${job_id} completed: ${s3_url}`);
      // Process the file
    } else {
      console.log(`Job ${job_id} failed: ${error}`);
    }
  }

  res.status(200).send('OK');
});

Python (Flask)

@app.route('/webhooks/tornado', methods=['POST'])
def handle_webhook():
    data = request.json

    if data['type'] == 'job_completed':
        if data['status'] == 'Completed':
            print(f"Job {data['job_id']} completed: {data['s3_url']}")
        else:
            print(f"Job {data['job_id']} failed: {data['error']}")

    return 'OK', 200

Best Practices

• Respond quickly — Return a 2xx status within 30 seconds. Process data asynchronously if needed.
• Handle duplicates — We retry failed webhooks 3 times. Use job_id for idempotency.
• Verify the source — Consider IP whitelisting for production environments.
• Use HTTPS endpoints — Always use HTTPS for your webhook URLs to protect payload data in transit.
• Log everything — Store webhook payloads for debugging and audit trails.

Webhooks vs. Polling: When to Use Each

For production AI video pipelines processing terabytes of content, webhooks are essential. They enable fully asynchronous architectures where your system can submit thousands of download jobs and process results as they arrive, without wasting resources on polling.

Real-World Architecture: AI Video Processing Pipeline

Here's how teams building long-form to short-form AI tools typically use webhooks with Tornado:

User submits YouTube URL
        ↓
Your API creates Tornado job (with webhook_url)
        ↓
Tornado downloads video (anti-bot bypass, direct cloud upload)
        ↓
Webhook fires → Your server receives notification
        ↓
AI model analyzes video from S3/R2 bucket
        ↓
Short clips generated and published

This architecture handles massive throughput because each step is decoupled. Tornado can download at TB/hour while your AI pipeline processes videos at its own pace, with webhooks connecting the two.

Webhook Security

For production environments, consider these additional security measures:

• Shared secret validation — Verify webhook signatures to ensure they come from Tornado
• IP allowlisting — Restrict your webhook endpoint to Tornado's IP range
• Rate limiting — Protect your endpoint from unexpected bursts
• Timeout handling — Process webhook data asynchronously to avoid timeouts

Next Steps

• Use batch downloads with webhook notifications for playlist processing
• Automate with n8n using webhook triggers
• Back to the quickstart guide if you're just getting started