# Web Crawl Skill

Crawl entire websites and extract content from multiple pages. Perfect for comprehensive site analysis, content audits, and bulk data collection.

---

## Endpoint

### Start Crawl
```
POST /skills/web/crawl
```

### Check Status
```
GET /skills/web/crawl/{crawlId}
```

## Authentication

Requires valid agent API key in Authorization header:
```
Authorization: Bearer <agent_api_key>
```

## Request Body (POST)

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `url` | string | Yes | - | Starting URL for crawl |
| `excludePaths` | array | No | - | URL patterns to exclude (e.g., `["/admin/*", "/api/*"]`)
| `includePaths` | array | No | - | URL patterns to include (e.g., `["/blog/*", "/articles/*"]`)
| `maxDepth` | number | No | `2` | Maximum crawl depth (1-10) |
| `limit` | number | No | `10` | Maximum pages to crawl (1-100) |
| `allowBackwardLinks` | boolean | No | `false` | Allow crawling links to parent directories |
| `allowExternalLinks` | boolean | No | `false` | Allow crawling external domain links |
| `scrapeOptions` | object | No | - | Options for scraping each page |

### Scrape Options

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `formats` | array | `["markdown"]` | Formats to extract: `markdown`, `html`, `screenshot`, `links` |
| `onlyMainContent` | boolean | `true` | Extract only main content |

## Example Request - Start Crawl

### Basic Crawl

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "maxDepth": 2,
    "limit": 10
  }'
```

### Blog Content Crawl

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog",
    "includePaths": ["/blog/*"],
    "excludePaths": ["/blog/tag/*", "/blog/author/*"],
    "maxDepth": 1,
    "limit": 50,
    "scrapeOptions": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }'
```

### Comprehensive Site Audit

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "maxDepth": 3,
    "limit": 100,
    "scrapeOptions": {
      "formats": ["html", "links"],
      "onlyMainContent": false
    }
  }'
```

## Example Response - Start Crawl

```json
{
  "success": true,
  "data": {
    "crawlId": "crawl_abc123xyz",
    "url": "https://example.com",
    "status": "scraping",
    "total": 0,
    "completed": 0
  }
}
```

## Check Crawl Status

```bash
curl https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl/crawl_abc123xyz \
  -H "Authorization: Bearer YOUR_API_KEY"
```

## Example Response - Completed Crawl

```json
{
  "success": true,
  "data": {
    "crawlId": "crawl_abc123xyz",
    "url": "https://example.com",
    "status": "completed",
    "total": 10,
    "completed": 10,
    "pages": [
      {
        "markdown": "# Page 1 Content\n\n...",
        "metadata": {
          "title": "Page 1 Title",
          "sourceURL": "https://example.com/page1",
          "statusCode": 200
        }
      },
      {
        "markdown": "# Page 2 Content\n\n...",
        "metadata": {
          "title": "Page 2 Title",
          "sourceURL": "https://example.com/page2",
          "statusCode": 200
        }
      }
    ]
  }
}
```

## Response Fields

### Crawl Status Object

| Field | Type | Description |
|-------|------|-------------|
| `crawlId` | string | Unique identifier for this crawl |
| `url` | string | Starting URL |
| `status` | string | Current status: `scraping`, `completed`, `failed` |
| `total` | number | Total pages discovered |
| `completed` | number | Pages successfully scraped |
| `pages` | array | Array of scraped page data (when completed) |

### Page Object

| Field | Description |
|-------|-------------|
| `markdown` | Content in markdown format |
| `html` | HTML content (if requested) |
| `links` | Links found on page (if requested) |
| `metadata` | Page metadata including URL and status |

## Use Cases

### Content Audit
Crawl your entire website to audit content quality, identify gaps, and plan updates.

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-site.com",
    "maxDepth": 5,
    "limit": 100,
    "scrapeOptions": {
      "formats": ["markdown"]
    }
  }'
```

### Competitor Analysis
Crawl competitor websites to analyze their content strategy, structure, and messaging.

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.com",
    "includePaths": ["/blog/*", "/resources/*", "/guides/*"],
    "maxDepth": 2,
    "limit": 50
  }'
```

### Documentation Extraction
Extract all documentation pages for offline analysis or migration.

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "includePaths": ["/docs/*"],
    "excludePaths": ["/docs/api/reference/*"],
    "maxDepth": 3,
    "limit": 100
  }'
```

### SEO Analysis
Crawl to gather all internal links, analyze page structure, and identify SEO issues.

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "maxDepth": 3,
    "limit": 100,
    "scrapeOptions": {
      "formats": ["links"],
      "onlyMainContent": false
    }
  }'
```

## Path Patterns

Use glob-style patterns for include/exclude paths:

| Pattern | Description |
|---------|-------------|
| `/blog/*` | All pages under /blog/ |
| `/articles/**` | All pages under /articles/ at any depth |
| `*.pdf` | All PDF files |
| `/admin/*` | Exclude admin pages |
| `/api/*` | Exclude API endpoints |

## Rate Limits

- Maximum 10 concurrent crawls per agent
- Maximum 100 pages per crawl
- Crawls may take time for large sites

## Error Responses

```json
{
  "success": false,
  "error": {
    "code": "CRAWL_FAILED",
    "message": "Failed to start crawl: URL inaccessible"
  }
}
```

## Best Practices

1. **Start Small**: Begin with low `limit` and `maxDepth` to test
2. **Use Include Paths**: Narrow scope with specific path patterns
3. **Check Status**: Poll status endpoint for completion
4. **Respect Rate Limits**: Don't start multiple crawls simultaneously
5. **Filter Appropriately**: Use `excludePaths` to skip unnecessary pages

## Notes

- Crawls are asynchronous - check status endpoint for results
- Large crawls may take several minutes to complete
- Pages are returned in the order they were discovered
- Some pages may fail - check statusCode in metadata
