# ML System Design Case Studies - Deep Dive

> **TL;DR**: This repository is a goldmine of 300+ real-world case studies from 80+ companies (Netflix, Airbnb, Uber, Stripe, etc.) showing exactly how they built their ML systems. It's not theory - it's battle-tested production architectures for recommendations, fraud detection, search ranking, and more.

---

## What Is This Repository?

This isn't a textbook or tutorial collection. It's a **curated library of actual engineering blog posts** from companies that have deployed ML systems at massive scale.

Each case study answers questions like:
- What problem were they solving?
- What data did they use?
- What model architecture did they choose?
- How did they evaluate success?
- How did they deploy and monitor it?
- What did they learn?

**The key insight**: These are real systems serving millions of users, not toy examples.

---

## Why This Matters

### The Gap Between ML Theory and Production

Most ML education focuses on:
- Training a model on a clean dataset
- Optimizing accuracy metrics
- Maybe deploying to a simple API

**Real production ML systems require**:
- Data pipelines handling petabytes
- Feature stores with thousands of signals
- A/B testing infrastructure
- Real-time serving at millisecond latency
- Model monitoring and drift detection
- Fallback mechanisms when models fail
- Business metric alignment (not just accuracy)

This repository bridges that gap with real examples.

### A Key Finding

> "A key finding from Booking.com is that increasing model performance does not necessarily translate to a gain in business value."

Offline metrics (AUC, F1-score) are proxies. What matters is actual business impact. These case studies show how companies think about that translation.

---

## How It's Organized

The repository organizes case studies in two ways:

### 1. By Industry

| Industry | Example Companies |
|----------|-------------------|
| **Tech Giants** | Google, Meta, Microsoft |
| **Streaming/Media** | Netflix, Spotify, YouTube |
| **E-commerce** | Amazon, Etsy, eBay, Alibaba |
| **Travel/Hospitality** | Airbnb, Booking.com, Expedia |
| **Delivery/Mobility** | Uber, Lyft, DoorDash, Instacart |
| **Fintech** | Stripe, PayPal, Square |
| **Social/Networking** | LinkedIn, Twitter, Pinterest |
| **Healthcare** | Various healthcare AI applications |

### 2. By ML Use Case

| Category | What It Covers |
|----------|----------------|
| **Recommender Systems** | Personalized content, products, connections |
| **Search & Ranking** | Query understanding, result ordering |
| **Fraud Detection** | Transaction screening, risk scoring |
| **Computer Vision** | Image classification, object detection |
| **NLP** | Text classification, entity extraction, translation |
| **Forecasting** | Demand prediction, ETA estimation |
| **Personalization** | User segmentation, dynamic experiences |

---

## The ML System Design Framework

These case studies follow a common pattern. Here's the framework:

```
┌─────────────────────────────────────────────────────────────────────┐
│                   ML SYSTEM DESIGN FRAMEWORK                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  1. PROBLEM FRAMING                                                 │
│     └── What are we trying to predict/optimize?                     │
│     └── Classification? Regression? Ranking? Generation?            │
│     └── What's the business metric we're improving?                 │
│                                                                     │
│  2. DATA PIPELINE                                                   │
│     └── What data sources exist?                                    │
│     └── How is it collected, stored, processed?                     │
│     └── Feature engineering & feature stores                        │
│                                                                     │
│  3. MODEL DEVELOPMENT                                               │
│     └── Model architecture selection                                │
│     └── Training infrastructure                                     │
│     └── Offline evaluation metrics                                  │
│                                                                     │
│  4. SERVING & DEPLOYMENT                                            │
│     └── Real-time vs. batch inference                               │
│     └── Latency requirements                                        │
│     └── A/B testing strategy                                        │
│                                                                     │
│  5. MONITORING & ITERATION                                          │
│     └── Model performance tracking                                  │
│     └── Data drift detection                                        │
│     └── Retraining triggers                                         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Featured Case Studies (Highlights)

### Netflix: Recommendation System

**The Scale**: Recommendations generate over $1B yearly revenue for Netflix.

**Key Insight**:
> "Different model architectures excel at different tasks. Even though many deep-learning models can be understood as extensions of existing (simple) recommendation algorithms, initially significant improvements over well-tuned non-deep-learning approaches were not observed."

**What They Learned**:
- Simple models often work surprisingly well
- Deep learning shines when you add heterogeneous features
- Multi-layered architecture: ranking, row selection, ratings prediction
- Each layer has its own ML service

**Architecture Pattern**:
```
User Request
    ↓
Row Selection Model → "Which categories to show?"
    ↓
Ranking Model → "Order items within each row"
    ↓
Personalized Homepage
```

---

### Airbnb: Search Ranking

**The Problem**: Rank listings for search results in a way that maximizes bookings.

**The Approach** (from their 2019 blog post):
1. **System Design**: Overall architecture
2. **Feature Engineering**: Which signals matter
3. **Model Selection**: Gradient Boosted Decision Trees
4. **Personalization**: Different rankings for different users
5. **Online Scoring**: Real-time inference
6. **A/B Testing**: Validating improvements
7. **Business Heuristics**: Domain knowledge integration

**Key Insight**: They explicitly cover how ML + distributed systems work together to deliver seamless user experiences.

---

### Uber: Real-Time Systems

**The Challenge**: Match riders and drivers in real-time, across the globe, with minimal latency.

**Scale**:
- Millions of GPS updates per second
- 100+ petabytes of data
- Global availability required

**Architecture Components**:
- **Kafka**: Event streaming
- **Redis**: Geospatial lookups
- **H3**: Hexagonal hierarchical geospatial indexing
- **Microservices**: Trip management, payments, dispatch, ETA

**Key Insight**: Evolution matters - they document moving from Vertica → Hadoop → custom Spark library (Hudi).

---

### Stripe: Fraud Detection (Radar)

**The System**: Radar - ML-powered fraud prevention

**What They Predict**:
- Fraudulent transactions
- Unusual spending behavior
- High-value merchant identification

**Business Impact**: Reduced failed transactions, better merchant experience.

**Key Pattern**: Real-time scoring with fallback mechanisms.

---

### DoorDash: Delivery Time Prediction

**The Problem**: Accurately predict when food will arrive.

**Input Signals**:
- Historical delivery data
- Restaurant cooking times
- Real-time traffic conditions
- Order details

**Key Practice**:
> "Companies like DoorDash log a variety of data at online prediction time, like model input features, model outputs, and data points from relevant production systems to prevent training-serving skew."

**Monitoring Innovation**: Track distributions of model inputs/outputs over time to identify when predictions go wrong.

---

### Spotify: Music Recommendations

**The System**: Personalized playlists and song suggestions

**Input Signals**:
- Listening history
- Genre preferences
- Mood indicators
- Time of day
- Context (workout, commute, etc.)

**Output**: Curated playlists matching user's current context.

---

## Common Patterns Across Case Studies

### 1. The Funnel/Layered Approach

Almost every large-scale system uses multiple models in sequence:

```
Candidate Generation (fast, rough filtering)
         ↓
    Ranking (more accurate scoring)
         ↓
    Re-ranking (business rules, diversity)
         ↓
    Final Results
```

**Why**: You can't run expensive models on millions of items. Filter first, then apply sophisticated scoring.

### 2. Feature Stores

Companies like DoorDash, Uber, and Airbnb invest heavily in feature stores:
- Consistent feature computation between training and serving
- Reusable features across multiple models
- Point-in-time correctness for training data

### 3. A/B Testing Infrastructure

Every company has sophisticated experimentation:
- Traffic splitting (percentage-based, user-based, cohort-based)
- Guardrail metrics to prevent regressions
- Statistical significance tracking
- Gradual rollouts (canary deployments)

### 4. Monitoring & Observability

Production ML requires:
- **Performance monitoring**: Latency, throughput, error rates
- **Data drift detection**: Input distribution changes
- **Model performance tracking**: Prediction accuracy over time
- **Business metric correlation**: Does model improvement help the business?

---

## The 7-Step ML System Design Formula

Based on patterns from these case studies:

| Step | Question to Answer |
|------|-------------------|
| **1. Requirements** | What's the business goal? Latency needs? Scale? |
| **2. Data** | What signals exist? How to collect/process them? |
| **3. Features** | Which features to engineer? How to store/serve them? |
| **4. Model** | Which algorithm? Training infrastructure? |
| **5. Evaluation** | Offline metrics? Online A/B test design? |
| **6. Serving** | Real-time or batch? How to handle failures? |
| **7. Monitoring** | What to track? When to retrain? |

---

## Categories of ML Problems

### Recommender Systems
- **Pointwise**: Score each item independently
- **Pairwise**: Compare pairs of items
- **Listwise**: Optimize entire ranking

### Search & Ranking
- Query understanding
- Candidate retrieval
- Multi-stage ranking
- Personalization

### Fraud Detection
- Real-time scoring
- Anomaly detection
- Rule + ML hybrid systems
- False positive management

### Computer Vision
- Image classification
- Object detection
- Content moderation
- Visual search

### NLP
- Named entity recognition
- Sentiment analysis
- Text classification
- Language identification
- Machine translation

### Forecasting
- Demand prediction
- ETA estimation
- Capacity planning
- Inventory optimization

---

## Key Tradeoffs to Understand

| Tradeoff | Considerations |
|----------|---------------|
| **Accuracy vs. Latency** | More complex models are slower to serve |
| **Freshness vs. Cost** | More frequent retraining costs more |
| **Personalization vs. Cold Start** | New users have no history |
| **Exploration vs. Exploitation** | Try new items vs. show proven winners |
| **Model Complexity vs. Interpretability** | Black boxes are hard to debug |
| **Online vs. Batch** | Real-time costs more but enables reactivity |

---

## How This Repository Can Help Us Achieve Our Goals

Looking at our "Software Development in the Age of AI" repository, this case study collection provides immense value:

### 1. **Reference Architecture for AI Projects**

Our repo focuses on AI-assisted development. These case studies provide:
- Proven architectures we can reference when building ML features
- Patterns for scaling AI systems
- Real examples of production ML infrastructure

**Practical use**: When planning an ML feature, find a similar case study and learn from their approach.

### 2. **Prompting & Context for AI Assistants**

We document prompting techniques. These case studies can:
- Provide context for AI assistants working on ML problems
- Serve as examples when asking AI to design systems
- Help validate AI-generated architectures against real-world patterns

**Example prompt enhancement**:
> "Design a recommendation system like Netflix's approach described in [case study], adapted for our use case..."

### 3. **Knowledge Base Expansion**

Our `.claude/knowledge/` system could incorporate:
- ML system design patterns as reference material
- Industry-specific best practices
- Common pitfalls and solutions

### 4. **Playbook Development**

We have playbooks for development workflows. We could add:
- "Designing an ML System" playbook based on these patterns
- "Evaluating ML Features" using A/B testing practices from case studies
- "Monitoring ML in Production" from real examples

### 5. **Interview Preparation Resource**

For our community:
- ML system design is a critical interview skill
- These case studies are frequently referenced in interviews
- Understanding real architectures beats theoretical knowledge

### 6. **Bridging Coding and Systems Thinking**

Our repo helps with AI-assisted *coding*. This extends to AI-assisted *architecture*:

| Our Current Focus | What Case Studies Add |
|-------------------|----------------------|
| Writing code | Designing systems |
| Local development | Production deployment |
| Individual features | End-to-end pipelines |
| Code quality | System reliability |

### 7. **Cross-Pollination with Other Deep Dives**

| Our Deep Dive | Connection to ML System Design |
|---------------|-------------------------------|
| **CCPM** | PRDs can reference ML case studies for requirements |
| **Vibe Kanban** | Orchestrate ML development tasks across agents |
| **olmOCR** | Part of data pipeline for ML training |
| **Parlant** | ML-powered chatbot behavior control |

### 8. **Learning from Scale**

These companies operate at scales most of us will never reach, but their patterns teach:
- How to think about reliability
- When to add complexity vs. keep it simple
- How to measure what matters
- How to iterate based on real feedback

Even if our systems are smaller, these principles apply.

---

## How to Use This Repository

### For Learning
1. Pick a domain you're interested in (recommendations, fraud, etc.)
2. Read 3-5 case studies from different companies
3. Note the patterns that repeat
4. Understand where they diverge and why

### For Interview Prep
1. Study the 7-step framework
2. Practice applying it to common problems
3. Use case studies as reference implementations
4. Understand tradeoffs, not just solutions

### For Building Systems
1. Find a case study similar to your problem
2. Adapt their architecture to your scale
3. Learn from their evaluation approach
4. Avoid their documented pitfalls

---

## Key Links

| Resource | URL |
|----------|-----|
| Main Repository | https://github.com/Engineer1999/A-Curated-List-of-ML-System-Design-Case-Studies |
| Evidently AI Database (800 studies) | https://www.evidentlyai.com/ml-system-design |
| Chip Huyen's ML Design | https://huyenchip.com/machine-learning-systems-design/case-studies.html |
| ML Interviews GitHub | https://github.com/alirezadir/Machine-Learning-Interviews |
| ML System Design Book | https://www.amazon.com/Machine-Learning-System-Design-Interview/dp/1736049127 |

---

## Quick Stats

| Metric | Value |
|--------|-------|
| Case Studies | 300+ |
| Companies Covered | 80+ |
| Primary Categories | Recommendations, Search, Fraud, CV, NLP, Forecasting |
| Top Companies Featured | Netflix, Airbnb, Uber, Spotify, LinkedIn, Stripe, DoorDash, Meta |
| License | Open access (links to public blog posts) |

---

## Bottom Line

This repository is essentially a **"steal like an artist" guide for ML system design**.

Instead of inventing from scratch, you can:
1. Find a similar problem from a company that solved it
2. Understand their approach and reasoning
3. Adapt their patterns to your context
4. Avoid their documented mistakes

For anyone building production ML systems - or just trying to understand how the tech giants do it - this is an invaluable resource.

The real value isn't just the technical patterns. It's understanding **how to think** about ML systems: starting with business goals, measuring what matters, iterating based on real feedback, and building for reliability at scale.

---

*Document created: December 2025*
*Sources: [GitHub - ML System Design Case Studies](https://github.com/Engineer1999/A-Curated-List-of-ML-System-Design-Case-Studies), [Evidently AI](https://www.evidentlyai.com/ml-system-design), [Chip Huyen ML Systems Design](https://huyenchip.com/machine-learning-systems-design/case-studies.html)*
