# Screenshot-to-Code Deep Dive

## What Is It?

**Screenshot-to-Code** is an open-source AI tool that converts screenshots, mockups, and video recordings into clean, functional code. With **71.3k+ stars**, it supports multiple frameworks (React, Vue, HTML+Tailwind) and AI models (Claude, GPT-4o, Gemini).

**Repository:** [abi/screenshot-to-code](https://github.com/abi/screenshot-to-code)

---

## The Problem It Solves

Converting designs to code traditionally requires:
- Manual HTML/CSS writing from mockups
- Time-consuming pixel matching
- Framework-specific knowledge

Screenshot-to-Code provides:
- **Instant code generation** from any screenshot
- **Multiple framework output** (React, Vue, HTML)
- **Video-to-code** for capturing interactions
- **Self-hostable** for privacy

---

## Key Statistics

| Metric | Value |
|--------|-------|
| **Stars** | 71,300 |
| **Forks** | 8,811 |
| **Contributors** | 25 |
| **License** | MIT |
| **Languages** | Python (52%), TypeScript (46%) |
| **Created** | November 2023 |

---

## Supported Output Frameworks

| Framework | Status |
|-----------|--------|
| **HTML + Tailwind CSS** | Primary output |
| **React + Tailwind CSS** | Supported |
| **Vue + Tailwind CSS** | Supported |
| **Bootstrap** | Supported |
| **Ionic + Tailwind CSS** | Supported |
| **HTML + CSS** | Supported |
| **SVG** | Supported |

---

## AI Models Supported

| Model | Provider | Rating |
|-------|----------|--------|
| **Claude 3.7 Sonnet** | Anthropic | Best (70.31%) |
| **GPT-4o** | OpenAI | Recommended (65.10%) |
| **Google Gemini** | Google | Supported |
| **Claude Opus 3** | Anthropic | Legacy (61.46%) |

**Image generation:**
- DALL-E 3 (OpenAI)
- Flux Schnell (Replicate)

---

## How It Works

### Architecture

```
Frontend (React/Vite) ← WebSocket → Backend (FastAPI)
                                         ↓
                              AI Provider APIs
                         (OpenAI, Anthropic, Google)
```

### Pipeline

1. **Input:** Upload screenshot or record video
2. **Frame Processing:** For video, extracts 20 frames
3. **Prompt Assembly:** Creates optimized prompt
4. **Parallel Generation:** Generates 4 variants simultaneously
5. **Output:** Clean, functional code

---

## Accuracy & Quality

### Benchmark Results (16 screenshots tested)

| Model | Score | Notes |
|-------|-------|-------|
| **Claude 3 Sonnet** | 70.31% | Best overall |
| **GPT-4 Vision** | 65.10% | Baseline |
| **Claude 3 Opus** | 61.46% | Underperformed |

### Strengths
- Excellent OCR and text extraction
- Good chart interpretation
- Production-ready HTML/Tailwind output
- Less "lazy" than GPT-4 (completes more elements)

### Known Weaknesses
- **Flex layouts:** Struggles with side-by-side elements
- **Colors:** Frequently misidentifies backgrounds
- **Not pixel-perfect:** Expect ~70% accuracy

---

## Video to Code (Experimental)

### How to Use

1. Use built-in screen recorder OR upload video
2. System extracts 20 frames
3. AI analyzes frames to generate code
4. Works best with videos **under 30 seconds**

### Tips
- Linger on each feature for ~1 second
- Keep transitions clear
- Best for multi-state UI capture

---

## Installation & Self-Hosting

### Docker Compose (Recommended)

```bash
# Clone repository
git clone https://github.com/abi/screenshot-to-code.git
cd screenshot-to-code

# Create .env file with API keys
echo "OPENAI_API_KEY=sk-..." > .env
echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env

# Start services
docker-compose up
```

**Access:** http://localhost:5173

### Local Development

**Backend:**
```bash
cd backend
pip install poetry
poetry install
poetry run uvicorn main:app --port 7001
```

**Frontend:**
```bash
cd frontend
npm install
npm run dev
```

### Configuration

```env
# Required (at least one)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

# Optional
GEMINI_API_KEY=...
REPLICATE_API_KEY=...  # For image generation

# For regions with API restrictions
OPENAI_BASE_URL=https://your-proxy/v1
```

---

## Pricing

**The tool itself is FREE** (MIT license)

### API Costs (Pay-as-you-go)

| Provider | Input | Output |
|----------|-------|--------|
| **Claude Sonnet** | $3/MTok | $15/MTok |
| **GPT-4o** | ~$10/MTok | ~$30/MTok |
| **Gemini** | Competitive | Competitive |

### Hosted Service

**screenshottocode.com** offers a managed version (pricing not public)

---

## Comparison to Alternatives

| Tool | Best For | Approach |
|------|----------|----------|
| **v0 (Vercel)** | Highest-quality React | Frontend only |
| **Bolt.new** | Full-stack in browser | WebContainer |
| **Lovable** | Non-developers, MVPs | Visual editing |
| **Screenshot-to-Code** | Self-hosting, flexibility | Open-source |

### Unique Advantages

1. **Open-source** - Full control, self-hosted
2. **Model flexibility** - Use any supported LLM
3. **Multi-framework** - React, Vue, HTML, Ionic
4. **Video support** - Capture interactions
5. **Cost control** - Use your own API keys

---

## Best Practices

### Input Quality
- Use high-resolution screenshots (1920px+ width)
- Ensure text is clearly readable
- Complete UI views, not partial

### For Best Results
1. Use Claude Sonnet 3.7 (best accuracy)
2. Generate multiple variants (default: 4)
3. Plan for 10-20% manual refinement
4. Iterate 2-3 times for complex UIs

### Post-Processing
- Focus first on flex layout fixes
- Adjust colors second
- Test responsiveness
- Validate functionality

---

## Limitations

| Limitation | Workaround |
|------------|------------|
| Not pixel-perfect (~70%) | Manual refinement |
| Flex layout issues | Adjust CSS manually |
| Color detection errors | Override in code |
| No mobile app generation | Frontend-only tool |
| Requires internet | Cloud AI dependency |

---

## Use Cases

### Ideal For
- **Rapid prototyping** - Design to code in minutes
- **Landing pages** - Marketing sites from mockups
- **Component libraries** - Generate variants quickly
- **Learning** - Understand HTML/CSS patterns

### Not Ideal For
- Pixel-perfect implementations
- Complex interactive applications
- Backend/logic generation
- Offline scenarios

---

## Technology Stack

| Component | Technology |
|-----------|------------|
| Backend | FastAPI (Python) |
| Frontend | React + Vite |
| Package Manager | Poetry |
| Deployment | Docker Compose |
| AI Providers | OpenAI, Anthropic, Google |

---

## Example Workflow

1. **Screenshot your design**
   ```
   Take a clean screenshot of your mockup
   ```

2. **Upload to screenshot-to-code**
   ```
   Drag and drop or paste
   ```

3. **Select framework**
   ```
   React + Tailwind CSS
   ```

4. **Choose model**
   ```
   Claude 3.7 Sonnet (recommended)
   ```

5. **Generate and refine**
   ```
   Review 4 variants, select best, iterate
   ```

---

## Resources

- **Repository:** https://github.com/abi/screenshot-to-code
- **Hosted Version:** https://screenshottocode.com
- **Blog - Claude Evaluation:** github.com/abi/screenshot-to-code/blob/main/blog/evaluating-claude.md

---

*Screenshot-to-Code is the leading open-source solution for design-to-code conversion. While not pixel-perfect, it provides an excellent starting point that can reduce development time by 50-70%.*
