# Web Scraper Fix Summary

## Problem
The web scraper was failing to fetch pages from sites like Medium that use anti-bot protection (Cloudflare, etc.). Basic HTTP requests with standard headers were getting blocked with 403 Forbidden errors.

## Solution
Based on the working example in `web_3.py`, implemented the following fixes:

### 1. Use `curl-cffi` Library
- Added `curl-cffi>=0.7.0` to project dependencies
- This library can impersonate a real Chrome browser including TLS fingerprinting
- Falls back to regular `requests` library if `curl-cffi` is not available

### 2. Complete Browser Headers
Added all the headers that a real Chrome browser sends:
- `Accept`: Full MIME type list with quality values
- `Accept-Encoding`: gzip, deflate, br
- `Accept-Language`: en-US,en;q=0.9
- `Cache-Control`: max-age=0
- `Sec-Ch-Ua`: Chrome version information
- `Sec-Ch-Ua-Mobile`: ?0
- `Sec-Ch-Ua-Platform`: "macOS"
- `Sec-Fetch-Dest`: document
- `Sec-Fetch-Mode`: navigate
- `Sec-Fetch-Site`: none
- `Sec-Fetch-User`: ?1
- `Upgrade-Insecure-Requests`: 1

### 3. TLS Fingerprinting
- When `curl-cffi` is available, use `impersonate="chrome116"` parameter
- This mimics Chrome's TLS fingerprint, making requests indistinguishable from a real browser
- Bypasses sophisticated bot detection that checks TLS handshake patterns

## Changes Made

### `src/trendsleuth/web_scraper.py`
```python
# Import curl_cffi with fallback
try:
    from curl_cffi import requests
except ImportError:
    import requests  # type: ignore

# In fetch_page_text():
try:
    response = requests.get(
        url,
        headers=headers,
        timeout=timeout,
        allow_redirects=True,
        impersonate="chrome116",  # Mimic Chrome's TLS fingerprint
    )
except TypeError:
    # Fallback if using regular requests library
    response = requests.get(
        url,
        headers=headers,
        timeout=timeout,
        allow_redirects=True,
    )
```

### `pyproject.toml`
```toml
dependencies = [
    ...
    "curl-cffi>=0.7.0",
    "requests>=2.32.0",
]
```

## Testing
- All 111 tests continue to pass
- Existing tests use mocked requests, so they work with both libraries
- Real-world testing with Medium URLs should now work

## Why This Works
Modern anti-bot systems check multiple signals:
1. **HTTP headers**: Must match a real browser exactly
2. **TLS fingerprint**: The order and configuration of TLS extensions uniquely identifies clients
3. **JavaScript execution**: Some sites check for browser JS APIs (not relevant for our text scraping)

By using `curl-cffi` with Chrome impersonation:
- We match Chrome's exact TLS fingerprint
- We send all the headers Chrome sends
- Combined, this makes requests indistinguishable from a real Chrome browser

## Fallback Behavior
If `curl-cffi` is not installed:
- Falls back to regular `requests` library
- Still sends complete browser headers
- Will work for most sites, but may fail on heavily protected sites like Medium
- No errors or crashes, just graceful degradation
