System Design Cheatsheet: Numbers, Patterns & Back-of-the-Envelope Math

1. Latency Numbers Every Programmer Should Know

When architectural bottlenecks emerge during high-scale system design, foundational computer architecture dictates performance limits. Evaluating whether a design will satisfy SLA targets requires an instinctive understanding of execution latencies across modern processors, memory arrays, physical disk subsystems, and network boundaries.

+-------------------------------------------------------------------------+
|                  Visual Latency Comparison Scale                        |
|                                                                         |
|  L1 Cache        [=] 0.5 ns                                             |
|  Mutex Lock      [===] 25 ns                                            |
|  Main Memory     [=======] 100 ns                                       |
|  SSD Random Read [================================] 150,000 ns (150 us)  |
|  HDD Disk Seek   [==============================================] 10 ms |
|  CA to Netherlands Packet Round-Trip [==========================] 150 ms|
+-------------------------------------------------------------------------+

| Operation / Hardware Interaction | Absolute Latency | Human Scaled Analogy (If 1 ns = 1 second) | | :--- | :--- | :--- | | L1 Cache Reference | 0.5 ns | 0.5 seconds (One heartbeat) | | Branch Mispredict (Processor Pipeline Stall) | 5 ns | 5 seconds | | L2 Cache Reference | 7 ns | 7 seconds | | Mutex Lock / Unlock (Uncontended) | 25 ns | 25 seconds | | Main Memory Reference (DRAM Read) | 100 ns | 100 seconds (~1.5 minutes) | | Compress 1 KB Bytes with Zippy | 3,000 ns (3 µs) | 50 minutes | | Send 2 KB Bytes over 1 Gbps Commodity Network | 20,000 ns (20 µs) | 5.5 hours | | SSD Random Read (NVMe Flash Array) | 150,000 ns (150 µs) | 1.7 days (A weekend) | | Read 1 MB Sequentially from RAM | 250,000 ns (250 µs) | 2.8 days | | Read 1 MB Sequentially from SSD | 1,000,000 ns (1 ms) | 11.6 days | | HDD Magnetic Disk Seek (Rotational Latency) | 10,000,000 ns (10 ms) | 16.5 weeks (~4 months!) | | Read 1 MB Sequentially from Mechanical HDD | 20,000,000 ns (20 ms) | 33 weeks | | Send Packet Datastore Round-Trip (Datacenter Intra-Region) | 500,000 ns (0.5 ms) | 5.8 days | | Send Packet Round-Trip US West to Europe (Cross-Atlantic) | 150,000,000 ns (150 ms) | 4.7 years! |

Critical Architectural Takeaways

1. Memory vs. Disk: Reading from main memory is 1,500x faster than a random SSD read, and 100,000x faster than a rotational HDD disk seek. Avoid disk I/O in latency-sensitive paths by layering RAM caches (Redis / Memcached).
2. Sequential vs. Random I/O: Sequential reads from an SSD or HDD are vastly faster than random sector reads. Log-Structured Merge (LSM) databases (Cassandra, Kafka) exploit this by utilizing pure sequential append-only operational log formats.
3. Network Physics: Network round-trip latencies across geographic oceans are constrained by the speed of light in fiber-optic mediums. To deliver sub-50ms user interface responses globally, deploy Content Delivery Networks (CDNs) and regional Anycast Edge Routing Proxy tiers.

2. Capacity & Back-of-the-Envelope Mathematical Equations

When sizing infrastructure architecture during design reviews or technical interviews, utilize standardized mathematical approximations to estimate processing throughput, network bandwidth, storage requirements, and memory caching envelopes.

       +-------------------------------------------------------------+
       |             Essential Math Approximation Table              |
       +-------------------------------------------------------------+
       |   Seconds in a Day        ≈ 86,400  (Round to 100,000)      |
       |   Power of 2: 2^10 (1 KB) ≈ 1,000 bytes                     |
       |   Power of 2: 2^20 (1 MB) ≈ 1,000,000 bytes (1 Million)     |
       |   Power of 2: 2^30 (1 GB) ≈ 1,000,000,000 bytes (1 Billion) |
       |   Power of 2: 2^40 (1 TB) ≈ 1 Trillion bytes                |
       +-------------------------------------------------------------+

1. Processing Throughput & QPS Sizing

To size backend processing clusters, calculate average and peak Queries Per Second (QPS) from predicted Daily Active Users (DAU) and interaction volumes:

$\text{Average QPS} = \frac{\text{Daily Active Requests}}{86,400 \text{ seconds}}$

$\text{Peak Burst QPS} = \text{Average QPS} \times \text{Peak Multiplier } (M)$ (Where industry standard multiplier $M$ typically ranges between $2.0\times$ and $5.0\times$ depending on real-time event concentration or seasonal traffic spikes).

2. Network Ingestion Bandwidth ($MB/s$)

To size API link adapters, edge load balancers, and internal switch fabrics:

$\text{Bandwidth Ingestion (MB/s)} = \text{Peak QPS} \times \text{Average Request Payload Size (KB)}$

3. Persistent Data Storage Expansion

When estimating database persistent storage capacity over a multi-year architecture timeline, include explicit overhead factors for metadata indexing structures ($I \approx 20\%$) and fault-tolerant replication configurations (typically $R = 3$ replicas across availability zones):

$\text{Annual Storage Requirement} = (\text{Daily New Records} \times \text{Record Size}) \times 365 \times \text{Replication Factor } (R) \times (1 + \text{Index Ratio } I)$

4. Memory Caching Pool Sizing (The 80/20 Pareto Principle)

In enterprise distributed application workloads, approximately $20\%$ of hot catalog records drive $80\%$ of all system read traffic. To protect legacy transactional backend SQL databases from overload during traffic peaks, dimension edge memory caching layers to hold at least $20\%$ of daily read volume in high-speed DRAM:

$\text{Cache RAM Size} = \left(\text{Daily Read Operations} \times \text{Average Payload Size}\right) \times 0.20$

3. Core Architectural Building Blocks & System Patterns

Modern cloud microservices rely on standardized architectural blueprints to achieve scalability, structural reliability, and fault isolation.

graph TD
    Client[Client Browser / App] -->|HTTPS Anycast DNS| CDN[Cloud CDN & Geo Edge]
    CDN -->|Cache Miss / API Invocation| LB[Layer 7 API Gateway & Load Balancer]
    
    LB -->|Consistent Hash Routing| Pool[Microservice Engine Pool]
    
    Pool <-->|Cache-Aside / Write-Through| Cache[(Distributed Cache Cluster: Redis)]
    Pool -->|Async Ingest / Event decoupling| Kafka[Log-Structured Stream: Apache Kafka]
    Pool <-->|ACID Read/Write| PrimaryDB[(Primary Relational Cluster: Aurora/PG)]
    
    PrimaryDB -->|Asynchronous Replication| ReadReplica[(ReadOnly Replicas)]
    Pool -->|Read Heavy Traffic| ReadReplica
    
    Kafka -->|Stream Processing Workers| Workers[Background Async Consumer Agents]
    Workers -->|Write Projections| NoSQL[(Wide-Column Store: Cassandra)]

1. DNS Resolution & Edge Load Balancing

• Anycast Routing & Geo-DNS: Maps domain names to physical server clusters situated physically closest to the client's geographic location, routing requests across optimized backbone networks rather than public Internet hops.
• Layer 4 vs. Layer 7 Load Balancing:
  • L4 (Transport Layer - TCP/UDP): Balances incoming packets using pure kernel network routing rules (IP + Port hashes) without inspecting HTTP payload body bytes. Exceptional throughput ($> 1,000,000$ QPS), utilized in edge infrastructure architectures (AWS Network Load Balancers / IPVS).
  • L7 (Application Layer - HTTP/HTTPS/gRPC): Terminates SSL encryption handshakes, parses HTTP request headers, and evaluates URI paths (/api/v1/orders vs /api/v1/users), allowing granular intelligent routing, custom authentication interception, and automated API shedding (Envoy, NGINX, AWS ALB).

2. Caching Topologies & Eviction Algorithms

• Cache-Aside (Lazy Loading): The application client queries the cache first; upon a cache miss, it reads directly from the primary persistent database, populates the cached entry, and serves the user. Ideal for read-heavy workflows with low mutation frequency.
• Write-Through / Write-Behind (Write-Back): In write-through setups, database modifications write synchronously to both cache memory and database disk arrays before confirming success. In write-behind topologies, writes ingest instantly into high-speed memory cache lines, and asynchronous worker queues flush mutations down to disk in the background—maximizing throughput at the risk of minimal transactional data loss during unexpected server power events.
• Eviction Algorithms (LRU, LFU, W-TinyLFU): While Least Recently Used (LRU) ejects items that haven't been read in the longest time period, it suffers from caching thrashing when occasional sequential batch scanning scripts traverse the entire database. Modern high-performance caching platforms deploy W-TinyLFU (Window Tiny Least Frequently Used), combining admission frequency bloom filters with windowed decay registers to protect high-value historical cache entries against sequential scanning pollution.

3. Asynchronous Messaging & Stream Ingestion

• Message Queues (RabbitMQ / AWS SQS): Direct worker task distribution utilizing independent FIFO queue processing. When a consumer node processes a message, the record clears entirely from storage queues. Ideal for transactional job processing and scheduled batch compute tasks.
• Log-Structured Streams (Apache Kafka / AWS Kinesis): Append-only immutable immutable storage logs organized by logical event topics and physically divided across numbered partitions. Records persist for long configurable retention durations (e.g., 7 days) regardless of consumption status. Multiple independent consumer groups replay the event stream independently at self-governed cursors, enabling real-time telemetry analytics, auditing ledgers, and Change Data Capture (CDC) synchronization pipelines.

4. Sharding & Partitioning Topologies

When dataset capacities exceed single-machine storage limitations, database tables must be horizontally partitioned across multiple physical database server nodes:

• Range-Based Sharding: Partitions rows according to strict numerical or alphabetical key thresholds (e.g., Server A processes IDs 1 - 1,000,000; Server B handles 1,000,001 - 2,000,000). Simple to query for range scans, but highly prone to structural Hotspotting where current active insertions flood the single highest range node while historical database nodes sit idle.
• Hash Sharding: Evaluates a mathematical hashing function against an entity partition key to distribute rows evenly across nodes: $\text{Node Index} = \text{Hash}(\text{Entity ID}) \pmod{N}$ Eliminates writing hotspots by dispersing sequential insertions uniformly across the database server cluster, at the cost of complicating cross-partition sequential range query scans.
• Consistent Hashing: Traditional modular hashing breaks down during infrastructure resizing: adding or removing a database node ($N \to N+1$) invalidates almost every active hash mapping, triggering massive database data migrations. Consistent hashing projects server node IDs and data resource keys onto a shared $360^\circ$ cyclical mathematical ring space ($0 \text{ to } 2^{128}-1$). When a server adds or drops, only data records assigned immediately adjacent to that specific node's angular coordinate require network migration, preserving stability across the rest of the cluster topology.

4. Reliability & Resilience Engineering Patterns

High-availability distributed architectures must assume underlying cloud networking tiers, remote third-party APIs, and hardware components will fail continuously. System resilience requires defensive architectural protection patterns.

stateDiagram-v2
    [*] --> Closed: Normal Operations
    Closed --> Open: Failure Rate Exceeds Threshold (%)
    Open --> HalfOpen: Cooldown Timer (Retry-After) Expired
    HalfOpen --> Closed: Probe Request Succeeds (Health Recovered)
    HalfOpen --> Open: Probe Request Fails (Still Degraded)

1. Circuit Breaker Pattern

Inspired by domestic electrical wiring safety breakers, software circuit breakers wrap downstream network integration calls inside an stateful monitoring interface.

• Closed State (Normal Operations): Traffic passes through directly to target downstream services. The breaker monitors execution exceptions and HTTP timeouts in real-time.
• Open State (Protective Trip): If network execution failures surpass a strict tolerance percentage threshold within an observed rolling sliding window (e.g., $> 50\%$ errors over 10 seconds), the circuit trips OPEN. Subsequent calls terminate instantly with fallback default payloads or rapid operational exceptions—sparing degraded backend servers from cascading queue saturation.
• Half-Open State (Recovery Probing): Once a designated cool-down period elapses, the circuit transitions into a test Half-Open posture, admitting a single experimental probe request. If the verification request succeeds, normal operations resume and the breaker resets to Closed. If it fails, the circuit snaps back to Open immediately.

2. Bulkhead Isolation

Derived from marine ship engineering where waterproof hull bulkheads prevent a single localized leak from flooding the entire vessel, software bulkheads compartmentalize resources across independent tenant or execution pools. Instead of maintaining one shared global database connection pool or thread executor across an entire application, partition infrastructure into dedicated, isolated thread pools assigned directly to distinct domain functional verticals (e.g., Billing Thread Pool vs. User Recommendation Thread Pool). If an external recommendation service stalls and exhausts its dedicated connections, core checkout and billing transactions execute seamlessly without encountering resource starvation.

3. Exponential Backoff with Jitter Mathematics

When transient network interruptions break communication lines, traditional retry algorithms that re-invoke calls immediately at static intervals trigger Retry Storms—an unintentional self-inflicted DDoS attack where thousands of reconnecting client instances amplify backend CPU loads simultaneously.

Robust distributed architectures govern retry loops utilizing Exponential Backoff combined with Random Jitter:

$t_{\text{wait}} = \min\left(t_{\text{max}}, t_{\text{base}} \times 2^{\text{attempt}}\right) \times \left(0.5 + \text{random}(0, 0.5)\right)$

Where:

• $t_{\text{base}}$ defines initial baseline wait duration (e.g., $100\text{ms}$).
• $\text{attempt}$ tracking successive execution failure loop iterations.
• $t_{\text{max}}$ caps ultimate retry ceiling ceilings (e.g., $30\text{ seconds}$).
• Jitter Formulation: Adding stochastic randomization ($0.5 + \text{random}(0, 0.5)$) desynchronizes competing concurrent client retry loops, dispersing reconnecting traffic storms safely across a smooth statistical distributions curves.

5. End-to-End Enterprise Architecture: Hyperscale Ride-Hailing Platform

To observe how these discrete building blocks converge in production, below is an architectural structural layout for a high-concurrency real-time ride-hailing geo-dispatch engine (analogous to Uber or Lyft).

graph LR
    subgraph Clients
        P["Passenger Mobile Client"]
        D["Driver Mobile Client (WebSockets)"]
    end

    subgraph Edge & Security
        DNS["Anycast Geo DNS"]
        CDN["Cloud CDN Layer"]
        WAF["Web Application Firewall (DDoS Def)"]
        API["L7 Reverse Proxy & API Gateway (Envoy)"]
    end

    subgraph Core Domain Microservices
        DS["Dispatch & Geo Engine"]
        BUS["Trip Management Service"]
        PAY["Billing & Payments Worker"]
    end

    subgraph Persistence & Streaming Layer
        RED["Redis Cluster (Active Driver Locations)"]
        KAFKA["Apache Kafka (Streaming Location Telemetry)"]
        ACID["PostgreSQL Multi-AZ Primary (Rides & Payments)"]
        WIDE["Apache Cassandra (Historical Ride Trace Metrics)"]
    end

    P -->|HTTPS Request| DNS --> CDN --> WAF --> API
    D -->|Persistent TCP WebSockets| API
    
    API -->|Routing| DS
    API -->|Routing| BUS
    API -->|Routing| PAY

    DS <-->|O(1) Geospatial Indexes / GeoHash| RED
    D -->|Stream Location UDP/TCP| KAFKA
    KAFKA -->|Consume Stream| DS
    KAFKA -->|Archive Telemetry| WIDE
    
    BUS <-->|ACID Transactions| ACID
    PAY <-->|Payment State| ACID

Architectural Flow Decomposition

1. Edge Route Defense: Mobile clients terminate network sessions against global Anycast edge proxies. Static UI elements serve directly from CDN edge caching layers. Web Application Firewalls (WAF) inspect protocols to suppress malicious payload injection before terminating SSL encryption at the Envoy Layer 7 proxy tier.
2. Real-Time Telemetry Ingest: Active drivers transmit GPS geo-coordinates every 4 seconds via lightweight WebSockets or UDP telemetry links. These updates inject straight into Apache Kafka partitioned topic queues, absorbing concurrent ingest storms without blocking primary database engines.
3. Low-Latency Geospatial Matching: Dispatch Worker instances consume real-time driver coordinates from Kafka streams and project location updates into an in-memory Redis Spatial Cluster. Redis utilizes specialized GeoHash geospatial structures to calculate adjacent available driver capacity in $< 2\text{ms}$ when a passenger hits request.
4. Reliable Polyglot State Separation: Once an algorithmic driver-to-rider match concludes, trip lifecycle events and billing debits write strictly into an authoritative, redundant PostgreSQL ACID primary array, preventing double-booking anomalies or incorrect billing calculations. Simultaneously, high-volume historical telemetry trails and complete route tracing coordinates flush sequentially into wide-column Apache Cassandra storage engines optimized for scalable log-structured append operations and long-term diagnostic pattern analysis.