Built an async multi-stage pipeline — raw upload → S3 → SQS → FFmpeg workers → parallel platform publishers — that transcodes and distributes 4K content to YouTube, TikTok, Instagram and three others within 15 minutes of upload.
Content creators needed uploads published across six platforms within 15 minutes of upload, including full 4K transcoding. The naive approach of a sequential, monolithic pipeline couldn't meet the latency requirement and had a single point of failure for every platform.
Decomposed the pipeline into isolated async stages: raw upload to S3 triggers an SQS message, picked up by containerised FFmpeg workers on ECS spot instances (over-provisioned 30% during business hours). A separate publishing layer runs one queue consumer per platform, each with its own retry logic and dead-letter queue. Status updates flow to the creator UI via Server-Sent Events.
Content now consistently publishes within 12 minutes of upload — beating the 15-minute SLA by 20%. Platform reliability improved to 99.4% — up from 87% with the previous vendor. Creators report eliminating approximately 3 hours of manual cross-posting per week.
