# Web Scrape Skill

Scrape any website URL and extract content in multiple formats including markdown, HTML, screenshots, and links.

---

## Endpoint

```
POST /skills/web/scrape
```

## Authentication

Requires valid agent API key in Authorization header:
```
Authorization: Bearer <agent_api_key>
```

## Request Body

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `url` | string | Yes | - | URL to scrape |
| `formats` | array | No | `["markdown", "html"]` | Output formats: `markdown`, `html`, `screenshot`, `links`, `extract` |
| `onlyMainContent` | boolean | No | `true` | Extract only main content (filter nav/footer) |
| `includeTags` | array | No | - | HTML tags to include (e.g., `["h1", "h2", "p"]`)
| `excludeTags` | array | No | - | HTML tags to exclude (e.g., `["nav", "footer", "aside"]`)
| `waitFor` | number | No | - | Wait time in ms before scraping (for dynamic content)
| `timeout` | number | No | `30000` | Request timeout in ms |

## Example Request

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog-post",
    "formats": ["markdown", "html", "links"],
    "onlyMainContent": true
  }'
```

## Example Response

```json
{
  "success": true,
  "data": {
    "markdown": "# Article Title\n\nThis is the article content in **markdown** format...",
    "html": "<article><h1>Article Title</h1><p>This is the content...</p></article>",
    "links": {
      "internal": ["/about", "/contact", "/blog"],
      "external": ["https://twitter.com/example", "https://github.com/example"]
    },
    "metadata": {
      "title": "Article Title - Example Blog",
      "description": "Article description for SEO",
      "ogTitle": "Article Title",
      "ogDescription": "Social sharing description",
      "ogImage": "https://example.com/image.jpg",
      "keywords": "keyword1, keyword2, keyword3",
      "robots": "index, follow",
      "language": "en",
      "sourceURL": "https://example.com/blog-post",
      "statusCode": 200
    }
  }
}
```

## Response Fields

### Data Object

| Field | Type | Description |
|-------|------|-------------|
| `markdown` | string | Content converted to markdown |
| `html` | string | Raw or cleaned HTML |
| `screenshot` | string | Base64 encoded screenshot (if requested) |
| `links` | object | Internal and external links |
| `metadata` | object | Page metadata (title, description, OG tags, etc.) |

### Metadata Object

| Field | Description |
|-------|-------------|
| `title` | Page title |
| `description` | Meta description |
| `ogTitle` | Open Graph title |
| `ogDescription` | Open Graph description |
| `ogImage` | Open Graph image URL |
| `keywords` | Meta keywords |
| `robots` | Robots meta directive |
| `language` | Page language |
| `sourceURL` | Original URL |
| `statusCode` | HTTP status code |

## Use Cases

### Competitor Content Analysis
```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.com/blog/featured-post",
    "formats": ["markdown", "links"],
    "onlyMainContent": true
  }'
```

### SEO Research
```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/landing-page",
    "formats": ["html", "links"],
    "onlyMainContent": false
  }'
```

### Content Curation
```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news-site.com/article",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "excludeTags": ["aside", "nav", ".advertisement"]
  }'
```

## Error Responses

```json
{
  "success": false,
  "error": {
    "code": "SCRAPE_FAILED",
    "message": "Failed to scrape URL: 404 Not Found"
  }
}
```

## Rate Limits

- Default: 100 requests per minute per agent
- Crawl operations: 10 concurrent crawls per agent

## Notes

- JavaScript-rendered content is automatically handled
- Respect robots.txt rules
- Some sites may block scraping - use responsibly
- Screenshots are returned as base64 strings
