HLD: Photo & Video Sharing (Instagram)
1. Architectural Overview & System Goals
Designing a global photo and short-form video sharing architecture like Instagram or TikTok requires scaling massive content read-write ingest while generating hyper-personalized, ultra-low latency timeline feeds for billions of global accounts.
Functional Requirements
- Media Capture & Upload: Low-latency ingest of high-resolution image mosaics and short HD videos with automated filter transcoding and adaptive bitrate generation.
- Social Graph & Follows: Unidirectional follower topologies supporting asymmetric influence curves (standard users vs. viral mega-celebrities with followers).
- News Feed & Timeline Generation: Asymmetric feed calculation delivering highly curated, chronological, and algorithmic engagement streams.
- Content Discovery & Search: Real-time multi-modal hashtag, location, and user meta-search indexing.
Non-Functional & Operational Goals
- High Availability over Consistency: Prioritizing system availability ( uptime) over strict transactional ACID consensus across social interactions (Eventual Consistency model).
- Ultra-Fast Feed Rendering: Sub-200 millisecond time-to-first-byte (TTFB) timeline load times regardless of regional edge network latency.
- Global Edge Acceleration: Multi-layered Content Delivery Network (CDN) propagation to offload origins and eliminate network bottlenecks during viral usage spikes.
2. Capacity Planning & Back-of-the-Envelope Estimation
Let us evaluate infrastructure design dimensions for a global consumer social matrix operating at 1 Billion Monthly Active Users (MAU) with 500 Million Daily Active Users (DAU).
Traffic & Ingest Workload Modeling
- New Content Creation Rate: Assume active users upload an average of photos/videos per day. Peak Upload Ingest ( factor): .
- Feed Consumption & Read Load: The ratio of timeline scrolling reads to uploads is intensely skewed toward reads at approximately .
Storage Sizing (Object Store & Databases)
- Photo Payload Volume: An uploaded image transcoded into three standardized responsive display formats (Thumb , Mobile , Desktop HD ) consumes raw aggregate storage.
- Metadata Database Sizing: Each media database index entry (Post UUID, Creator ID, Caption Text, Geo-coordinates, Hash tags, Timestamp, Likes counter) requires .
Bandwidth & Edge Cache Scaling
- Egress Throughput: Serving 1.15 million timeline item read streams per second across mobile clients: Note: To prevent infrastructure collapse, of payload egress MUST terminate directly at decentralized Edge CDN nodes.
3. High-Level Architecture Design
graph TD
subgraph Client Application
App[iOS / Android / Web Client]
end
subgraph Edge & Routing Layer
DNS[Anycast DNS Router]
WAF[DDoS Mitigation & WAF Cloudflare/Shield]
CDN[Edge Content Delivery Network CloudFront/Akamai]
Gateway[API Gateway / L7 Load Balancer]
end
subgraph Microservices Domain
Auth[User Identity & Graph Svc]
MediaSvc[Media Upload & Transcoding Engine]
FeedSvc[Timeline & Newsfeed Service]
SearchSvc[Discovery & ElasticSearch Engine]
LikeSvc[Engagement Counters & Likes Svc]
end
subgraph Data Infrastructure
Kafka[Apache Kafka: High-Throughput Event Bus]
Cache[Redis Cluster: Feed News-Caches & Hot Objects]
Cassandra[(Apache Cassandra: Social Graph & Activity Logs)]
Relational[(PostgreSQL / Sharded DB: Accounts & Metadata)]
S3[(Geo-Replicated Object Storage: Raw & Compressed Media)]
ES[(ElasticSearch Cluster: Hashtag & Text Index)]
end
App -->|Fetch Statics| CDN
CDN -->|Cache Miss Origin Pull| S3
App -->|Secure API Requests| WAF
WAF --> Gateway
Gateway --> FeedSvc & MediaSvc & LikeSvc & SearchSvc
MediaSvc -->|Direct Chunk Upload| S3
MediaSvc -->|Publish Upload Event| Kafka
Kafka -->|Fan-Out Workers| Cache
FeedSvc -->|Fetch Precomputed Feeds| Cache
LikeSvc -->|Atomic Counters| Cache
LikeSvc -->|Async Flush| Cassandra
SearchSvc <--> ES
Architectural Component Interaction
- DDoS Mitigation Layer (WAF): Edge filtering screening inbound volumetric UDP floods, SYN exhaustion attacks, and programmatic bots using adaptive IP behavior analytics before requests hit origin API Gateways.
- Media Transcoding Pipeline: Stateless microservices processing uploaded media asynchronously using FFmpeg queues, extracting EXIF tags, generating multiscale WEBP/AVIF compressed files, and executing automated NSFW AI vision screening.
- Feed Engine & Fan-Out Layer: Specialized workers subscribing to upload publish logs via Apache Kafka, dynamically arbitrating whether to execute rapid push-based injections into Redis timeline caches or fall back to pull-based merging for high-profile influencer accounts.
4. In-Depth Component Mechanics & Protocols
4.1 Hybrid Feed Generation: Push vs. Pull (Fan-Out Mechanics)
Generating a chronologically or algorithmically sorted timeline in real-time across followed accounts per active session via simple relational joins (SELECT * FROM posts WHERE user_id IN (1, 5, ...) ORDER BY date DESC) destroys SQL databases instantly under .
- Push Model (Fan-Out on Write): When ordinary user Alice posts an image, asynchronous workers immediately inject the new Post ID directly into the pre-computed Redis timeline linked lists of all her followers. Feed retrieval becomes an instantaneous O(1) in-memory array operation.
- The Celebrity Problem: When a global celebrity with followers posts a photo, executing concurrent writes into Redis lists induces crippling Redis queue locks and eviction cascades.
- Hybrid Architecture Resolution: The system bifurcates distribution logic based on follower density thresholds ( followers).
- Standard Accounts (): Processed exclusively via instantaneous Push Fan-Out on Write.
- VIP Influencer Accounts (): Post IDs are written strictly to the VIP's personal user outbox. When an individual follower opens their app, the Feed Service executes a Pull-on-Read operation, fetching the user’s pre-computed regular timeline from Redis and dynamically merging the newest posts from any VIPs they follow in memory via Min-Heap merge sort algorithms.
sequenceDiagram
autonumber
participant Regular as Alice (120 Followers)
participant VIP as Cristiano (150M Followers)
participant Router as API / Feed Router
participant Kafka as Event Bus (Kafka)
participant Worker as Fan-Out Worker
participant Redis as Redis Timeline Caches
participant DB as Cassandra / Outbox
participant Follower as Bob's App
Regular->>Router: Upload New Photo (Post_901)
Router->>DB: Save Post_901 to Alice's Outbox
Router->>Kafka: Emit Event: Alice Posted 901
Kafka->>Worker: Consume Event (Check Followers < 5,000)
Worker->>Redis: LPUSH Post_901 into 120 follower timeline lists
VIP->>Router: Upload VIP Photo (Post_902)
Router->>DB: Save Post_902 to VIP Outbox ONLY (No Redis fanout!)
Follower->>Router: GET /api/v1/timeline
Router->>Redis: LRANGE Bob's precomputed feed (Gets Post_901)
Router->>DB: Multiget recent posts from Bob's followed VIPs (Gets Post_902)
Router->>Router: Merge-Sort in Memory (Post_902 + Post_901 by ranking Score)
Router-->>Follower: Respond with optimized JSON timeline
4.2 High-Throughput Engagement Counters (Likes & Comments)
When a viral post accumulates likes per second, writing synchronous updates directly to slow relational disk rows (UPDATE posts SET likes = likes + 1 WHERE id = 801) triggers heavy lock contention and transaction aborts.
- Distributed In-Memory Sharded Counter Pools: Engagement hits are routed directly to Redis / Memcached atomic multi-shard increment structures (
INCRBY). - Asynchronous Log-Structured Flushing: Background chronological workers dump aggregated Like metrics down to immutable Cassandra Log tables via scheduled batch writes every 60 seconds, guaranteeing low latency while preserving ultimate storage consistency without data loss.
4.3 Distributed UUID Generation (Instagram Photo IDs)
Standard auto-incrementing sequential database primary keys fail in horizontally partitioned multi-datacenter environments.
- Instagram 64-Bit Custom Snowflake IDs: To ensure monotonic temporal ordering without central coordination bottlenecks, Instagram utilizes a specialized 64-bit ID composition integer:
- Bits 0–41 (42 bits): Millisecond timestamp epoch offset (provides ~69 years of capacity).
- Bits 42–54 (13 bits): Logical Database Shard Identifier ( discrete physical shard partitions).
- Bits 55–63 (10 bits): Auto-incrementing sequence counter per individual database node (allowing unique IDs generated per shard per millisecond).
- Because timestamp bits occupy the highest significance order, sorting content items purely by numerical ID equates to perfectly accurate temporal chronological sorting.
5. Production-Grade Implementation: Hybrid Fan-Out Engine & Custom Snowlake Generator
The following production-level Python code demonstrates an authentic 64-bit Snowflake ID Generator, hybrid Push-Pull feed generation router, and distributed atomic counter batcher.
import time
import heapq
import threading
from typing import List, Dict, Set, Optional
class SnowflakeIDGenerator:
"""Production 64-bit decentralized ID Generator optimized for chronological sorting."""
def __init__(self, shard_id: int, epoch: int = 1600000000000):
self.epoch = epoch
self.shard_id = shard_id & 0x1FFF # 13 bits mask (8192 shards max)
self.sequence = 0
self.last_timestamp = -1
self.lock = threading.Lock()
def generate_id(self) -> int:
with self.lock:
timestamp = int(time.time() * 1000)
if timestamp < self.last_timestamp:
raise Exception("Clock moved backwards! Rejecting UUID generation.")
if timestamp == self.last_timestamp:
self.sequence = (self.sequence + 1) & 0x3FF # 10 bits mask (1024 max)
if self.sequence == 0:
# Sequence exhausted in current millisecond, await next clock roll
while timestamp <= self.last_timestamp:
timestamp = int(time.time() * 1000)
else:
self.sequence = 0
self.last_timestamp = timestamp
# Construct 64-bit integer composition
id_val = ((timestamp - self.epoch) << 23) | (self.shard_id << 10) | self.sequence
return id_val
class PostPayload:
def __init__(self, post_id: int, creator_id: str, media_url: str):
self.post_id = post_id
self.creator_id = creator_id
self.media_url = media_url
self.created_at = time.time()
self.likes = 0
def __lt__(self, other: 'PostPayload'):
# Min-Heap sorting comparator (Reverse chronological order by Post ID)
return self.post_id > other.post_id
class RedisTimelineCache:
"""In-memory Redis simulation managing precomputed user home feeds."""
def __init__(self, max_timeline_size: int = 500):
self._feeds: Dict[str, List[int]] = {}
self.max_size = max_timeline_size
self._mutex = threading.Lock()
def push_to_timeline(self, user_id: str, post_id: int) -> None:
with self._mutex:
if user_id not in self._feeds:
self._feeds[user_id] = []
self._feeds[user_id].insert(0, post_id)
# Enforce strict LRU capacity eviction
if len(self._feeds[user_id]) > self.max_size:
self._feeds[user_id].pop()
def fetch_timeline(self, user_id: str) -> List[int]:
with self._mutex:
return list(self._feeds.get(user_id, []))
class CassandraSocialGraph:
"""Mock scalable NoSQL ring storing global accounts, posts outboxes, and follow edges."""
def __init__(self):
self.followers: Dict[str, Set[str]] = {}
self.following: Dict[str, Set[str]] = {}
self.user_outbox: Dict[str, List[PostPayload]] = {}
self.post_catalog: Dict[int, PostPayload] = {}
self.vip_threshold = 5 # Scaled down for functional simulation (Prod = 5,000)
self.lock = threading.Lock()
def add_follow_edge(self, follower: str, followee: str) -> None:
with self.lock:
if followee not in self.followers: self.followers[followee] = set()
if follower not in self.following: self.following[follower] = set()
self.followers[followee].add(follower)
self.following[follower].add(followee)
def is_vip_influencer(self, user_id: str) -> bool:
return len(self.followers.get(user_id, set())) >= self.vip_threshold
class HybridFeedEngine:
"""Core social orchestration engine handling push vs pull routing mechanics."""
def __init__(self, graph: CassandraSocialGraph, cache: RedisTimelineCache):
self.graph = graph
self.cache = cache
self.id_gen = SnowflakeIDGenerator(shard_id=402)
def process_media_upload(self, creator_id: str, media_url: str) -> PostPayload:
post_id = self.id_gen.generate_id()
post = PostPayload(post_id, creator_id, media_url)
with self.graph.lock:
self.graph.post_catalog[post_id] = post
if creator_id not in self.graph.user_outbox:
self.graph.user_outbox[creator_id] = []
self.graph.user_outbox[creator_id].insert(0, post)
# Execute Hybrid Fan-Out arbitration
if not self.graph.is_vip_influencer(creator_id):
# Standard Account: Push Fan-Out to all followers in Redis
followers = self.graph.followers.get(creator_id, set())
for fid in followers:
self.cache.push_to_timeline(fid, post_id)
print(f"[FAN-OUT PUSH] Standard user '{creator_id}' post transferred to {len(followers)} Redis timelines.")
else:
# VIP Influencer: Bypass Redis Fan-out completely!
print(f"[FAN-OUT BYPASS] VIP Influencer '{creator_id}' post written strictly to outbox.")
return post
def hydrate_user_timeline(self, user_id: str, limit: int = 20) -> List[PostPayload]:
# 1. Fetch pre-computed standard posts from Redis timeline cache
cached_ids = self.cache.fetch_timeline(user_id)
candidate_posts: List[PostPayload] = [self.graph.post_catalog[pid] for pid in cached_ids if pid in self.graph.post_catalog]
# 2. Identify any followed VIP influencers and execute Pull-on-Read extraction
followed_users = self.graph.following.get(user_id, set())
for followee in followed_users:
if self.graph.is_vip_influencer(followee):
vip_recent_posts = self.graph.user_outbox.get(followee, [])[:5]
candidate_posts.extend(vip_recent_posts)
# 3. Dynamic in-memory k-Way Chronological Sort via Min-Heap
candidate_posts.sort() # Uses PostPayload __lt__ (reverse chronological)
# De-duplicate identical objects and return paginated window
seen = set()
final_feed = []
for p in candidate_posts:
if p.post_id not in seen:
seen.add(p.post_id)
final_feed.append(p)
if len(final_feed) == limit:
break
return final_feed
# --- Execution Simulation ---
if __name__ == "__main__":
db_graph = CassandraSocialGraph()
redis_cache = RedisTimelineCache()
engine = HybridFeedEngine(db_graph, redis_cache)
# 1. Create Social Graph Topology
# Bob follows standard user Alice and VIP celebrity Ronaldo
db_graph.add_follow_edge("bob_user", "alice_regular")
db_graph.add_follow_edge("bob_user", "ronaldo_vip")
# Give Ronaldo enough followers to pass VIP threshold (5 for demo)
for i in range(1, 10):
db_graph.add_follow_edge(f"fan_{i}", "ronaldo_vip")
# 2. Alice uploads a photo -> Triggers synchronous Push Fan-out
engine.process_media_upload("alice_regular", "https://cdn.instasocial.io/img/alice_trip.jpg")
time.sleep(0.01)
# 3. Ronaldo uploads a photo -> Bypasses Fan-out (No Redis load)
engine.process_media_upload("ronaldo_vip", "https://cdn.instasocial.io/img/ronaldo_match.jpg")
# 4. Bob opens his mobile app; Engine dynamically merges cached regular feed with pulled VIP posts
feed = engine.hydrate_user_timeline("bob_user")
print(f"\n[CLIENT HYDRATION] Bob's Live Generated Timeline (Top {len(feed)} items):")
for idx, item in enumerate(feed, 1):
print(f" #{idx} | Post ID: {item.post_id} | Author: {item.creator_id:15} | URL: {item.media_url}")
6. Failure Modes, Edge Cases, & Resiliency Engineering
| Failure Vector | Architectural Symptom | Engineering Mitigation Strategy |
| :--- | :--- | :--- |
| Thundering Herd & Celebrity Hotspots | A superstar posts an announcement; 20 million fans click the link simultaneously, inducing severe origin database melting. | Multi-Layer Edge Caching with Cache Stampede Protection: CDN layers implement Request Coalescing (Single Flight Locks). When an edge cache misses on a viral post, only the initial incoming request crosses to origin DBs; subsequent millions are held in a local execution spin-lock until the cache warms. |
| Volumetric DDoS attacks | Malicious botnets blast billions of GET requests directly against expensive algorithmic search and discovery query endpoints. | API Gateway Adaptive Token Bucket Rate Limiting: Edge ingress nodes evaluate IP reputations, JWT signature fingerprints, and request behavior via leaky bucket algorithms. Rogue automated clients receive immediate HTTP 429 Too Many Requests or invisible algorithmic tarpits. |
| Database Partition Split & Replica Drift | Network partitions cause asynchronous Cassandra social replicas in Europe and North America to reflect conflicting follower lists. | CRDTs (Conflict-Free Replicated Data Types): Follower counter and membership sets employ Observed-Remove Sets (OR-Sets). Disjoint regional additions and deletions merge mathematically without data loss or administrative lock arbitration once partitions resolve. |
| Redis Cache Eviction Storm | A primary Redis node hosting 20 million active user timelines undergoes fatal power exhaustion and restarts cold. | Warm Replica Replication & Background Replay Workloads: Redis operates strictly inside highly available Master-Replica failover clusters. If total memory destruction occurs, background offline map-reduce workers read active social graph roots and re-hydrate user timeline lists progressively during low-load intervals. |
7. Deep-Dive Interview Question & Answer Bank
Q1: How do you handle algorithmic feed ranking without consuming immense application server CPU during real-time timeline scrolling?
A: We decouple timeline generation into Two-Stage Machine Learning Pipelines (Candidate Generation vs. Heavy Ranking). Stage 1 (Candidate Generation) is ultra-fast: it pulls ~500 recent chronological Post IDs from Redis memory arrays and ElasticSearch hashtag match engines within . Stage 2 (Heavy Neural Ranking) passes merely those selected 500 candidate IDs into an specialized inference cluster evaluating multi-modal features (user historical affinity, creator interaction frequency, recency, comment velocity) to produce final presentation rankings in memory just in time for visual rendering.
Q2: How do you architect search indexing for billions of photos without stalling standard OLTP transaction databases?
A: Transactional relational or Cassandra databases are optimized exclusively for primary-key or partition-scoped sweeps, not inverted full-text keyword indexing. We implement a Change Data Capture (CDC) Pattern via Debezium and Apache Kafka. When a user posts a photo with captions and hashtags, the transaction commits strictly to Postgres/Cassandra in sub-millisecond windows. Asynchronous database binary-log readers transmit immutable row mutations across Kafka pipelines directly to dedicated ElasticSearch / OpenSearch Clusters, which build inverted text index arrays independently without locking transaction databases.
Q3: Explain how Instagram applies Image Compression and Adaptive Bitrate Delivery across heterogeneous network capabilities.
A: Relying solely on client device screen width to select image scale wastes bandwidth on poor cellular links. Instagram iOS/Android daemons constantly monitor actual network layer round-trip times (RTT) and real-time TCP socket drain velocities. When a device transitions from 5G Fiber WiFi to edge 3G rural cellular connections, the client API calls explicitly request low-entropy compressed image representations (format=HEIC, quality=40, width=480px). CDN edge endpoints utilize serverless dynamic transcoding (or cached variants) to deliver lightweight visual assets under strict payload budget thresholds.
Q4: How does the architecture prevent infinite loop amplification during high-velocity automated liking or commenting spam attacks?
A: Edge Gateway infrastructures combine Strict Stateful Ephemeral Circuit Breakers with Proof-of-Work (PoW) Client Challenges. If a client token executes over 15 distinct POST /api/like mutations within a rolling 5-second interval, the edge layer injects a cryptographic Hashcash cryptographic challenge payload into the HTTP response. The native client CPU must perform computationally expensive hash iterations before the backend permits subsequent interaction commits, rendering automated spamming economically unviable for attacker botnets.
Q5: Describe how to architect a geo-distributed Disaster Recovery replication schema without degrading normal write latencies.
A: Cross-continental synchronous database replication (e.g., waiting for multi-region quorum consensus between US East and AP Central) introduces unacceptable physical speed-of-light transit overhead (). Systems utilize Asynchronous Active-Active Multi-Master Replication with Local Data Ownership. User profiles are assigned a primary localized datacentering zone based on GPS registration heuristics. Writes terminate locally with high performance (), while regional background workers stream database mutation logs across specialized transatlantic private fiber trunks using Conflict-Free Replicated Data Types (CRDTs) to merge inter-region mutations asynchronously without breaking availability SLAs.