HLD: Chat System (WhatsApp)

1. Architectural Overview & System Goals

Designing a global-scale real-time messaging platform akin to WhatsApp or Telegram requires balancing low-latency communication, strict data consistency, high availability, and extreme end-to-end security. At peak load, modern messaging architectures handle hundreds of millions of concurrent connections, delivering billions of messages daily with sub-hundred-millisecond delivery guarantees.

Functional Requirements

• 1-on-1 & Group Messaging: Low-latency delivery of text, media (images, videos, audio), and documents.
• Presence & Read Receipts: Real-time indicator updates for online/offline presence, last seen, message delivery statuses (Sent, Delivered, Read), and typing indicators.
• Offline Message Queuing: Reliably buffering and delivering messages when disconnected users return online.
• End-to-End Encryption (E2EE): Cryptographically assuring that no intermediaries (including backend infrastructure) can inspect message contents.

Non-Functional & Operational Goals

• Ultra-Low Latency: Under 100ms end-to-end message routing latency for active connected parties.
• High Availability & Fault Tolerance: Nine-nines ($99.999\%$) availability for connection termination and message queuing.
• Connection Scalability: Supporting upwards of 500 million concurrently active persistent TCP/WebSocket connections across geographically distributed edge nodes.

2. Capacity Planning & Back-of-the-Envelope Estimation

Let us model a large-scale enterprise messaging ecosystem with 500 Million Daily Active Users (DAU).

Traffic & Throughput Modeling

• Average Daily Usage: Each active user sends an average of $40$ messages per day.
• Total Daily Messages: $500\text{M} \times 40 = 20\text{ Billion messages/day}$.
• Average Write QPS: $\text{Average QPS} = \frac{20 \times 10^9 \text{ messages}}{86,400 \text{ seconds/day}} \approx 231,481 \text{ msgs/sec}$
• Peak Write QPS: Assuming a $2.5\times$ diurnal scaling factor during high-usage evening window: $\text{Peak QPS} = 231,481 \times 2.5 \approx 578,700 \text{ msgs/sec}$
• Fan-out & Read Receipts: Each message triggers 2 status updates (Delivered, Read). Accounting for presence broadcasts and typing signals, internal signaling traffic is approximately $4\times$ message QPS ($\approx 2.3\text{M packets/sec}$).

Memory & Connection Footprint

• Persistent Connection Tracking: Assuming each connected device maintains a long-lived stateful TCP socket to an edge connection server.
• Memory per Connection Structure: Maintaining SSL state, user buffers, and heartbeats requires roughly $\sim 10\text{ KB}$ per open connection in optimized C++/Erlang runtime (e.g., Erlang OTP on FreeBSD or Netty on Linux).
• Total Active Connection Memory: $\text{Total RAM} = 500 \times 10^6 \times 10\text{ KB} = 5\text{ TB total socket buffer footprint}$
• Connection Server Sizing: A high-performance instance with 64GB RAM and tuned file descriptor limits (ulimit -n 1000000) can handle $\approx 500,000$ concurrent connections. We require a minimum array of $1,000$ edge Gateway instances globally.

Storage & Database Sizing

• Message Metadata: Assume average message payload size (including encrypted text, IDs, sender, timestamps) is $500\text{ Bytes}$.
• Daily Ingest Volume: $20 \times 10^9 \times 500\text{ B} = 10\text{ TB/day}$ of text metadata.
• Media Attachment Sizing: Assume $10\%$ of messages involve attachments ($1\text{ MB}$ average compressed media). $\text{Media Daily Storage} = 2 \times 10^9 \times 1\text{ MB} = 2\text{ Petabytes/day}$ Note: Media is aggressive deduplicated via cryptographic hashes before blob store persistence.

3. High-Level Architecture Design

graph TD
    subgraph Clients
        MobileA[Alice's Device]
        MobileB[Bob's Device]
    end

    subgraph Edge & Transport Layer
        DNS[Geo-DNS & Anycast Routing]
        LB[L4 / L7 Edge Load Balancers]
        WSG[WebSocket / TCP Connection Gateway]
    end

    subgraph Core Services
        Dispatch[Message Router & Dispatcher]
        Session[Session / Presence Service]
        GroupSvc[Group Messaging Service]
        Auth[Identity & E2EE Key Vault]
    end

    subgraph Storage & Queuing
        Redis[Redis Cluster: User Presence & Session Map]
        Kafka[Apache Kafka: High-Throughput Message Log]
        Cassandra[(Apache Cassandra: Offline & Chat History)]
        S3[(Object Store: Media Attachments)]
    end

    subgraph Push & Notification Layer
        APNS[Apple Push Notifications APNs]
        FCM[Google Cloud Messaging FCM]
    end

    MobileA -->|TLS / WebSocket| WSG
    MobileB -->|TLS / WebSocket| WSG
    WSG <-->|RPC / gRPC| Dispatch
    Dispatch -->|Lookup Target Routing| Redis
    Dispatch -->|Store & Forward| Kafka
    Kafka -->|Persist History| Cassandra
    Dispatch -->|Target Offline?| FCM
    Dispatch -->|Target Offline?| APNS
    MobileA -->|Upload / Download Encrypted Media| S3

Architectural Component Interaction

1. Connection Gateway Layer (WSG): Edge microservice clusters responsible for terminating TLS SSL connections, managing WebSocket heartbeats (PING/PONG), protocol decapsulation, and translating stateful client frames into stateless internal RPCs (gRPC).
2. Session & Presence Service: Highly partitioned in-memory cluster (Redis Cluster or etcd) mapping an active user ID to their exact Gateway instance host ID ({userId: "user_981", host: "wsg-us-east-1a-04", status: "ONLINE"}).
3. Message Dispatcher: Stateless engine evaluating message destinations. If the recipient is online, it emits an asynchronous RPC directly to the target’s hosting WSG. If offline, it pushes an event to push notification queues and stores an offline delivery package.
4. Group Messaging Engine: Evaluates group recipient membership lists, executes dynamic routing fan-out via message queues (Apache Kafka), and guarantees causal consistency across distributed members.

4. In-Depth Component Mechanics & Protocols

4.1 Real-Time Protocol & Connection State

While traditional HTTP request-response paradigms suffer from high protocol header overhead and tcp handshaking latencies, chat systems utilize custom binary framing over persistent transport protocols (e.g., WhatsApp’s modified XMPP over TCP or secure WebSockets).

• Heartbeating & Keep-Alive: To prevent carrier NAT (Network Address Translation) timers from dropping silent connections, devices exchange empty framing packets every $30–60$ seconds.
• Optimized Reconnection Strategies: If WiFi transitions to cellular data, sessions perform quick cryptographic resume via Session-IDs and sequence number syncs rather than renegotiating full TLS handshakes.

4.2 End-to-End Encryption (The Signal Protocol)

To ensure zero knowledge storage, messages are encrypted at the client boundary using the Signal Protocol (Double Ratchet Algorithm).

• Pre-Key Bundles: Upon registration, each client pushes cryptographic identity keys, signed pre-keys, and a pool of one-time ephemerals to the identity server.
• Session Initialization (X3DH): When Alice first messages Bob, her device downloads Bob's Pre-Key bundle from the repository, mathematically generating a shared root secret without requiring Bob to be concurrently online.
• Double Ratchet Engine: Every transmitted message rotates the symmetrical conversation key, guaranteeing forward secrecy and compromise resilience.

sequenceDiagram
    autonumber
    participant Alice
    participant Gateway as Connection Gateway
    participant Router as Message Router
    participant DB as Cassandra / Offline Queue
    participant Bob

    Alice->>Gateway: Encrypted Message (MsgId=402, Target=Bob)
    Gateway->>Router: gRPC RouteMessage(402, Bob)
    Router->>DB: Query Session State (Bob Online?)
    alt Bob is Online on Gateway-2
        Router->>Gateway: Relay directly to Bob's Gateway
        Gateway->>Bob: Push via active WebSocket frame
        Bob->>Gateway: Ack Delivery (ACK_Delivered_402)
        Gateway->>Alice: Push Delivery Receipt to Alice
    else Bob is Offline / No Active Socket
        Router->>DB: Write to Offline Message Box (Cassandra)
        Router->>Gateway: Dispatch FCM/APNS Wakeup Payload
        Note over Bob: Device connects later
        Bob->>Gateway: Establish WS Socket & Request Sync
        Gateway->>DB: Pull unread message payloads
        DB->>Bob: Deliver Offline Messages (MsgId=402)
        Bob->>Gateway: Flush Delivery Acks
        Gateway->>DB: Mark messages delivered & clean offline box
    end

4.3 Offline Storage & Database Selection (Apache Cassandra)

Why Cassandra over relational PostgreSQL for core message logs?

• Write Optimality: Chat systems represent high-volume append-only workload profiles. Cassandra’s Log-Structured Merge (LSM) tree architecture ensures constant-time $O(1)$ disk sequential writes via commit logs and memtables without B-Tree lock contention.
• Data Sharding & Partitioning Strategy: To read conversation timelines efficiently, messages are co-located by composite keys: $\text{Primary Key} = (\text{Partition Key: } \text{conversation\_id}, \text{ Clustering Key: } \text{message\_timestamp}, \text{message\_id})$
• This model guarantees that an entire user conversation lives on a single Cassandra partition node, turning inbox hydration into rapid sequential disk sweeps.

5. Production-Grade Implementation: Real-time Dispatcher & State Engine

The following Python reference architecture demonstrates an enterprise-grade message dispatcher using thread-safe state concurrency, asynchronous network routing, offline message buffering, and robust exponential backoff delivery attempts.

import time
import uuid
import logging
import threading
from typing import Dict, List, Optional, Any

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")

class UserSession:
    """Represents a stateful user TCP/WebSocket session connection."""
    def __init__(self, user_id: str, gateway_host: str):
        self.user_id = user_id
        self.gateway_host = gateway_host
        self.connected_at = time.time()
        self.is_active = True

class MessagePayload:
    """Encapsulates an E2EE binary messaging frame."""
    def __init__(self, sender_id: str, receiver_id: str, encrypted_body: str):
        self.message_id = str(uuid.uuid4())
        self.sender_id = sender_id
        self.receiver_id = receiver_id
        self.encrypted_body = encrypted_body
        self.timestamp = time.time()
        self.delivery_status = "SENT"

class CassandraOfflineStore:
    """Mock persistence ring representing an LSM-Tree backed Cassandra cluster."""
    def __init__(self):
        self._store: Dict[str, List[MessagePayload]] = {}
        self._lock = threading.Lock()

    def enqueue_offline_message(self, receiver_id: str, message: MessagePayload) -> None:
        with self._lock:
            if receiver_id not in self._store:
                self._store[receiver_id] = []
            self._store[receiver_id].append(message)
            logging.info(f"[CASSANDRA] Message {message.message_id} persisted in offline queue for {receiver_id}.")

    def fetch_and_purge(self, receiver_id: str) -> List[MessagePayload]:
        with self._lock:
            messages = self._store.pop(receiver_id, [])
            if messages:
                logging.info(f"[CASSANDRA] Purged {len(messages)} offline messages for active hydration: {receiver_id}.")
            return messages

class RedisPresenceCluster:
    """Distributed Redis cache controlling real-time gateway presence lookup."""
    def __init__(self):
        self._sessions: Dict[str, UserSession] = {}
        self._mutex = threading.Lock()

    def register_session(self, user_id: str, gateway_host: str) -> None:
        with self._mutex:
            self._sessions[user_id] = UserSession(user_id, gateway_host)
            logging.info(f"[REDIS] Registered session for {user_id} on gateway {gateway_host}.")

    def terminate_session(self, user_id: str) -> None:
        with self._mutex:
            if user_id in self._sessions:
                self._sessions[user_id].is_active = False
                del self._sessions[user_id]
                logging.warning(f"[REDIS] Terminated session for {user_id}.")

    def locate_user(self, user_id: str) -> Optional[str]:
        with self._mutex:
            session = self._sessions.get(user_id)
            return session.gateway_host if session and session.is_active else None

class RealtimeMessageDispatcher:
    """Core routing engine executing intelligent dispatching, offline buffering, and receipt signaling."""
    def __init__(self, presence_cluster: RedisPresenceCluster, offline_store: CassandraOfflineStore):
        self.presence = presence_cluster
        self.storage = offline_store
        self.max_retries = 3

    def dispatch(self, message: MessagePayload) -> bool:
        logging.info(f"[DISPATCHER] Processing message {message.message_id} from {message.sender_id} -> {message.receiver_id}")
        
        # 1. Query Real-Time Presence Ring
        target_gateway = self.presence.locate_user(message.receiver_id)
        
        if target_gateway:
            # Attempt active WebSocket transport delivery with exponential backoff
            if self._relay_to_gateway(target_gateway, message):
                message.delivery_status = "DELIVERED"
                self._emit_read_receipt(message)
                return True
            else:
                logging.error(f"[DISPATCHER] Active relay failed to gateway {target_gateway}. Falling back to offline persistence.")
        
        # 2. Target Offline or Unreachable -> Commit to DB & Fire Push Notification
        self.storage.enqueue_offline_message(message.receiver_id, message)
        self._trigger_push_notification(message.receiver_id, message)
        return False

    def on_user_reconnect(self, user_id: str, gateway_host: str) -> None:
        """Triggered upon socket reconnection to flush state."""
        self.presence.register_session(user_id, gateway_host)
        pending_messages = self.storage.fetch_and_purge(user_id)
        for msg in pending_messages:
            logging.info(f"[HYDRATION] Flushing offline message {msg.message_id} -> {user_id} via {gateway_host}")
            msg.delivery_status = "DELIVERED"
            self._emit_read_receipt(msg)

    def _relay_to_gateway(self, gateway: str, message: MessagePayload) -> bool:
        """Simulates RPC transit attempt with retries."""
        for attempt in range(1, self.max_retries + 1):
            try:
                # Simulate RPC network boundary delivery
                logging.info(f"[RPC] Transmitting frame {message.message_id} to {gateway} (Attempt {attempt}/{self.max_retries})")
                time.sleep(0.05) # Simulated wire latency
                return True # Success
            except Exception as exc:
                logging.warning(f"[RPC ERROR] Failed transit: {exc}. Retrying in {2**attempt}s...")
                time.sleep(2 ** attempt)
        return False

    def _trigger_push_notification(self, user_id: str, message: MessagePayload) -> None:
        logging.info(f"[PUSH NOTIFICATION] Dispatched encrypted APNs/FCM wakeup signaling payload to {user_id}.")

    def _emit_read_receipt(self, message: MessagePayload) -> None:
        sender_gateway = self.presence.locate_user(message.sender_id)
        if sender_gateway:
            logging.info(f"[RECEIPT] Emitted '{message.delivery_status}' ACK for {message.message_id} back to {message.sender_id} via {sender_gateway}.")

# --- Execution Simulation ---
if __name__ == "__main__":
    redis_presence = RedisPresenceCluster()
    cassandra_store = CassandraOfflineStore()
    dispatcher = RealtimeMessageDispatcher(redis_presence, cassandra_store)

    # 1. Register active sessions
    redis_presence.register_session("alice_mdx", "gateway-eu-central-01")
    
    # 2. Alice sends message to Bob (who is currently offline)
    msg1 = MessagePayload("alice_mdx", "bob_mdx", "E2EE_Ciphertext_A9F8B2C1")
    dispatcher.dispatch(msg1)

    # 3. Bob turns on mobile internet & connects
    time.sleep(0.5)
    dispatcher.on_user_reconnect("bob_mdx", "gateway-us-east-02")
    
    # 4. Immediate real-time exchange while both active
    msg2 = MessagePayload("bob_mdx", "alice_mdx", "E2EE_Ciphertext_D8C7A3E4")
    dispatcher.dispatch(msg2)

6. Failure Modes, Edge Cases, & Resiliency Engineering

| Failure Vector | Architectural Symptom | Engineering Mitigation Strategy | | :--- | :--- | :--- | | Gateway Node Crash | 500,000 active connections immediately severed; massive reconnection storm occurs. | Exponential Jitter & Edge Anycast: Client SDKs enforce randomized reconnect jitter ($t = \text{base} \times 2^\text{attempt} \pm \text{random}$). Anycast DNS routes traffic to surviving global nodes. | | Out-of-Order Delivery | Asynchronous network hops cause message $N+1$ to arrive before message $N$. | Monotonic Client-Side Sequencing: Each message includes a client-generated Lamport timestamp or incremental sequence ID. Receiving SDKs buffer out-of-order packets and reassemble sequentially. | | Split-Brain Session Maps | Network partition causes Redis cluster to show user connected to two different Gateways. | Epoch Fencing Tokens: When a session is initialized, it generates a monotonically increasing fencing token. Gateways discard routing commands carrying outdated epoch stamps. | | Group Fan-out Amplification | A viral group containing 10,000 members triggers database connection exhaustion during active broadcast. | Kafka Partition Sharding & Fan-out on Read: Large communities transition to a read-driven pull model or break fan-out lists into parallelized Kafka messaging buckets to eliminate backend blocking. | | Duplicate Message Dispatch | Network timeouts during WebSocket ACK transits prompt clients to retransmit identical message packets. | Idempotent Dedup Window: Connection Gateways cache recent client deduplication UUIDs ({sender_id}:{client_msg_id}) in an LRU memory cache for 5 minutes to intercept retries. |

7. Deep-Dive Interview Question & Answer Bank

Q1: How do you prevent message loss if the target recipient reconnects right as their gateway node fails during offline database purge?
A: We implement Two-Phase Acknowledgment between the client device and the backend storage ring. The Gateway initiates an offline message read by locking the payload records in Cassandra with a visibility timeout, rather than deleting them immediately upon dispatch. The offline database messages are only physically purged or marked with a tombstone after the receiving client's device returns a cryptographically valid L7 acknowledgment packet over the WebSocket (ACK_CONFIRMED). If the gateway or device dies mid-flight, the visibility lock expires, and the subsequent connection attempt safely redelivers the payload without loss.

Q2: In large group chats ($100,000+$ users in Telegram channels), standard push fan-out ($1 \times N$ database writes) destroys underlying databases. How do we architect around this?
A: We discard naive write-time fan-out in favor of a hybrid Fan-Out on Read (Pull-Based Channel Architecture). Instead of replicating a message $100,000$ times across individual user inboxes, the payload is written exactly once into an immutable Channel Message Log partition in Cassandra. User sessions track only a single lightweight marker: their Last Read Offset ($O_U$). When active users open the channel, their client executes a simple ranged query against the centralized channel partition: SELECT * FROM channel_log WHERE channel_id = 'C1' AND sequence_num > O_U LIMIT 50. Push notifications for super-groups are downgraded to coalesced background signals rather than individual message wakes.

Q3: Why doesn't WhatsApp store user conversation history natively on backend server disks forever like Telegram or Slack?
A: This represents a core design architecture choice centered around End-to-End Privacy, Regulatory Compliance, and Infrastructure Cost Optimization. By treating backend architectures strictly as high-speed routing relays (Store-and-Forward) and deleting ciphertext immediately upon confirmed client receipt, the enterprise completely circumvents multi-petabyte historical database maintenance costs, simplifies GDPR/data privacy adherence, and eliminates server-side encryption back doors. Historical storage relies entirely on encrypted client local storage (SQLite on iOS/Android) backed up directly to user cloud vaults (iCloud / Google Drive).

Q4: How do you balance presence service load when 500 million active users continuously emit live typing and online indicators every second?
A: Typing and presence signals are treated as Transient Ephemeral State and completely decoupled from durable storage engines (Cassandra/DBs). They are routed exclusively via volatile in-memory publish-subscribe meshes (such as Redis Pub/Sub or Apache Kafka with zero-disk retention). To prevent signaling network storms during active typing, client SDKs apply strong throttle debounce mechanisms—broadcasting a typing event at most once every 3 seconds per conversation window and relying on localized timers on the recipient’s device to auto-expire the visual indicator after 5 seconds of silence.

Q5: Describe how media attachment processing operates without exhausting WebSockets or application gateway bandwidth.
A: WebSocket channels are optimized exclusively for low-latency, lightweight binary framing ($\le 64\text{ KB}$). Attempting to funnel $10\text{ MB}$ HD video attachments through TCP sockets would create severe head-of-line blocking for subsequent concurrent text messages. Instead, client devices request a Pre-Signed Upload URL from the application API and upload the symmetrically encrypted media blob directly to an Edge Object Store (e.g., AWS S3 via CloudFront accelerating multipart uploads). Once the store responds with a cryptographic Hash ID and download token, the client transmits an ordinary lightweight JSON textual message over the WebSocket containing the metadata ({media_id: "sha256_e82a", key: "enc_key", type: "IMAGE"}). The recipient downloads and reconstructs the media directly via static CDN routes.