# Web Extract Skill

Extract structured data from any website using LLM (Large Language Model) extraction. Define a schema and let AI extract exactly the information you need.

---

## Endpoint

```
POST /skills/web/extract
```

## Authentication

Requires valid agent API key in Authorization header:
```
Authorization: Bearer <agent_api_key>
```

## Request Body

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | URL to extract data from |
| `schema` | object | Yes | JSON schema defining the structure to extract |
| `prompt` | string | No | Custom prompt to guide extraction |
| `systemPrompt` | string | No | System prompt for the LLM |

## Schema Definition

The schema follows a simplified JSON schema format:

```json
{
  "fieldName": { "type": "string" },
  "count": { "type": "number" },
  "isActive": { "type": "boolean" },
  "items": { 
    "type": "array", 
    "items": { "type": "string" } 
  },
  "nested": {
    "type": "object",
    "properties": {
      "key": { "type": "string" }
    }
  }
}
```

### Supported Types

| Type | Description | Example |
|------|-------------|---------|
| `string` | Text values | `"Hello World"` |
| `number` | Numeric values | `42`, `3.14` |
| `boolean` | True/false | `true`, `false` |
| `array` | List of items | `["item1", "item2"]` |
| `object` | Nested object | `{"key": "value"}` |

## Example Request

### Extract Article Information

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog-post",
    "schema": {
      "title": { "type": "string" },
      "author": { "type": "string" },
      "publishDate": { "type": "string" },
      "wordCount": { "type": "number" },
      "mainContent": { "type": "string" },
      "keyPoints": { 
        "type": "array", 
        "items": { "type": "string" } 
      },
      "targetAudience": { "type": "string" },
      "callToAction": { "type": "string" }
    },
    "prompt": "Extract detailed information about this blog post including the main points and target audience"
  }'
```

### Extract Product Information

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.com/product",
    "schema": {
      "productName": { "type": "string" },
      "pricing": { 
        "type": "object",
        "properties": {
          "amount": { "type": "number" },
          "currency": { "type": "string" },
          "billingPeriod": { "type": "string" }
        }
      },
      "features": { 
        "type": "array", 
        "items": { "type": "string" } 
      },
      "targetMarket": { "type": "string" },
      "valueProposition": { "type": "string" }
    },
    "prompt": "Extract product details for competitive analysis"
  }'
```

### Extract SEO Data

```bash
curl -X POST https://reply-marketing-api.vercel.app/api/v1/skills/web/extract \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "schema": {
      "headline": { "type": "string" },
      "subheadings": { 
        "type": "array", 
        "items": { "type": "string" } 
      },
      "primaryKeywords": { 
        "type": "array", 
        "items": { "type": "string" } 
      },
      "contentTone": { "type": "string" },
      "contentStructure": { "type": "string" },
      "internalLinks": { "type": "number" },
      "externalLinks": { "type": "number" }
    },
    "prompt": "Extract SEO and content structure information for analysis"
  }'
```

## Example Response

```json
{
  "success": true,
  "data": {
    "extract": {
      "title": "10 Ways to Improve Your Crypto Payments",
      "author": "John Doe",
      "publishDate": "2024-01-15",
      "wordCount": 1250,
      "mainContent": "Full article content...",
      "keyPoints": [
        "Use stablecoins for price stability",
        "Enable mobile money integration",
        "Reduce transaction fees"
      ],
      "targetAudience": "Crypto beginners and fintech entrepreneurs",
      "callToAction": "Sign up for our newsletter"
    },
    "metadata": {
      "sourceURL": "https://example.com/blog-post"
    }
  }
}
```

## Use Cases

### Competitive Analysis
Extract structured data from competitor websites to understand their messaging, features, and positioning.

### Content Research
Extract key points, statistics, and insights from industry articles to inform your own content creation.

### Lead Generation
Extract contact information, company details, and decision-maker data from target accounts.

### Market Research
Extract pricing, features, and positioning data from multiple competitors for comparison analysis.

### Content Auditing
Extract SEO data, content structure, and keyword usage from top-ranking pages.

## Best Practices

1. **Be Specific**: Define clear field names and provide prompts that guide extraction
2. **Use Arrays for Lists**: When extracting multiple items (features, points, etc.), use array types
3. **Nested Objects**: Use nested objects for complex data structures (pricing with amount, currency, etc.)
4. **Custom Prompts**: Add specific prompts to improve extraction accuracy
5. **Test and Iterate**: Test different schema structures to get the best results

## Error Responses

```json
{
  "success": false,
  "error": {
    "code": "EXTRACT_FAILED",
    "message": "Failed to extract data: URL inaccessible"
  }
}
```

## Rate Limits

- Default: 50 extraction requests per minute per agent
- Complex extractions may take longer to process

## Notes

- Extraction uses AI/LLM and may have small processing delays
- Results accuracy depends on page clarity and schema design
- Some websites with heavy anti-bot protection may not work
- Custom prompts can significantly improve extraction quality
