# CSQL (CutySoup Query Language) Guide

**Sprint 7 Phase 1: Core Engine - Complete Implementation**

CSQL is a powerful JSON-based query language for structured data extraction from HTML documents. It provides a declarative approach to web scraping that eliminates the need for imperative DOM traversal.

## Table of Contents

- [Quick Start](#quick-start)
- [Basic Concepts](#basic-concepts)
- [Query Structure](#query-structure)
- [Field Types](#field-types)
- [Data Transformations](#data-transformations)
- [Type Casting](#type-casting)
- [Convenience Methods](#convenience-methods)
- [Real-World Examples](#real-world-examples)
- [Error Handling](#error-handling)
- [API Reference](#api-reference)

## Quick Start

```python
from cutysoup import CutySoup

# Parse HTML
html = '''
<div class="products">
    <div class="product">
        <h3 class="name">Laptop</h3>
        <span class="price">$999.99</span>
    </div>
    <div class="product">
        <h3 class="name">Mouse</h3>
        <span class="price">$29.99</span>
    </div>
</div>
'''

soup = CutySoup(html)

# CSQL Query
query = {
    "products": {
        "selector": ".product",
        "type": "list",
        "fields": {
            "name": ".name",
            "price": {
                "selector": ".price",
                "transform": [{"regex": r"[\d.]+"}],
                "cast": "float"
            }
        }
    }
}

result = soup.query(query)
# Result: {"products": [{"name": "Laptop", "price": 999.99}, {"name": "Mouse", "price": 29.99}]}
```

## Basic Concepts

### Selectors
CSQL uses CSS selectors to target HTML elements:
```python
"selector": "h1"              # Single element
"selector": ".class-name"     # By class
"selector": "#element-id"     # By ID
"selector": "div.product a"   # Complex selector
```

### Extraction Types
- **text** (default): Extract text content
- **attribute**: Extract attribute value
- **html**: Extract HTML content
- **element**: Return the element itself

### Field Types
- **Single**: Extract from first matching element
- **List**: Extract from all matching elements

## Query Structure

### Basic Structure
```python
{
    "field_name": "selector",           # Simple text extraction
    "field_name": {                     # Complex field
        "selector": "css-selector",
        "extract": "text|attribute|html",
        "attribute": "attr-name",       # Required for attribute extraction
        "type": "list",                 # Optional: extract from all matches
        "default": "fallback-value",    # Optional: default if not found
        "transform": [...],             # Optional: data transformations
        "cast": "string|integer|float|boolean"  # Optional: type casting
    }
}
```

### Nested Fields
```python
{
    "articles": {
        "selector": "article",
        "type": "list",
        "fields": {                     # Nested extraction
            "title": ".title",
            "author": ".author",
            "metadata": {
                "selector": ".meta",
                "fields": {
                    "date": ".date",
                    "category": ".category"
                }
            }
        }
    }
}
```

## Field Types

### Simple Text Extraction
```python
query = {
    "title": "h1",                      # Extract text from h1
    "description": ".description"       # Extract text from .description
}
```

### Attribute Extraction
```python
query = {
    "link": {
        "selector": "a",
        "extract": "attribute",
        "attribute": "href"
    },
    "image_src": {
        "selector": "img",
        "extract": "attribute", 
        "attribute": "src"
    }
}
```

### List Extraction
```python
query = {
    "all_links": {
        "selector": "a",
        "type": "list",
        "extract": "attribute",
        "attribute": "href"
    },
    "all_headings": {
        "selector": "h1, h2, h3",
        "type": "list"                  # Default: extract text
    }
}
```

### Default Values
```python
query = {
    "optional_field": {
        "selector": ".might-not-exist",
        "default": "Not Available"
    }
}
```

## Data Transformations

CSQL provides powerful data transformation capabilities:

### Regex Extraction
```python
{
    "price": {
        "selector": ".price-text",      # "Price: $1,299.99"
        "transform": [
            {"regex": r"[\d,.]+"} 	    # Extract "1,299.99"
        ]
    }
}
```

### Text Replacement
```python
{
    "clean_number": {
        "selector": ".number",          # "1,234"
        "transform": [
            {"replace": {",": ""}}      # Result: "1234"
        ]
    }
}
```

### Text Manipulation
```python
{
    "clean_text": {
        "selector": ".messy-text",      # "  HELLO WORLD  "
        "transform": [
            {"strip": True},            # Remove whitespace
            {"lower": True}             # Convert to lowercase
        ]
    }
}
```

### Complex Transformation Chains
```python
{
    "review_count": {
        "selector": ".reviews",         # "Based on 1,234 reviews"
        "transform": [
            {"regex": r"([0-9,]+)"},    # Extract "1,234"
            {"replace": {",": ""}},     # Remove comma -> "1234"
        ],
        "cast": "integer"               # Convert to 1234 (int)
    }
}
```

## Type Casting

Convert extracted strings to appropriate data types:

### Numeric Types
```python
{
    "price": {
        "selector": ".price",
        "transform": [{"regex": r"[\d.]+"}],
        "cast": "float"                 # Convert to float
    },
    "quantity": {
        "selector": ".quantity",
        "cast": "integer"               # Convert to integer
    }
}
```

### Boolean Type
```python
{
    "in_stock": {
        "selector": ".availability",    # "In Stock" or "Out of Stock"
        "transform": [
            {"replace": {"In Stock": "true", "Out of Stock": "false"}}
        ],
        "cast": "boolean"               # Convert to true/false
    }
}
```

## Convenience Methods

The `CSQLQuery` class provides convenient methods for common patterns:

### Extract List of Structured Data
```python
from cutysoup import CSQLQuery

query = CSQLQuery.from_html(html)
products = query.extract_list(".product", {
    "name": ".name",
    "price": ".price",
    "id": {"extract": "attribute", "attribute": "data-id"}
})
```

### Extract Single Structured Object
```python
product = query.extract_single(".product", {
    "name": ".name",
    "description": ".description"
})
```

### Extract Simple Text Lists
```python
names = query.extract_text_list(".product-name")
# Result: ["Product 1", "Product 2", ...]
```

### Extract Attribute Lists
```python
urls = query.extract_attributes("a", "href")
# Result: ["/link1", "/link2", ...]
```

## Real-World Examples

### E-commerce Product Scraping
```python
query = {
    "products": {
        "selector": ".product-card",
        "type": "list",
        "fields": {
            "id": {"extract": "attribute", "attribute": "data-id"},
            "name": ".product-title",
            "price": {
                "selector": ".current-price",
                "transform": [{"regex": r"[\d.]+"}],
                "cast": "float"
            },
            "original_price": {
                "selector": ".original-price", 
                "transform": [{"regex": r"[\d.]+"}],
                "cast": "float",
                "default": None
            },
            "rating": {
                "selector": ".rating",
                "extract": "attribute",
                "attribute": "data-rating",
                "cast": "float"
            },
            "reviews": {
                "selector": ".review-count",
                "transform": [{"regex": r"(\d+)"}],
                "cast": "integer"
            },
            "in_stock": {
                "selector": ".availability",
                "transform": [
                    {"replace": {"In Stock": "true", "Out of Stock": "false"}}
                ],
                "cast": "boolean"
            }
        }
    }
}
```

### News Article Extraction
```python
query = {
    "articles": {
        "selector": "article",
        "type": "list", 
        "fields": {
            "headline": "h1, h2",
            "author": ".author",
            "publish_date": {
                "selector": "time",
                "extract": "attribute",
                "attribute": "datetime"
            },
            "category": ".category",
            "summary": ".excerpt, .summary",
            "tags": {
                "selector": ".tag",
                "type": "list"
            },
            "is_featured": {
                "selector": "",                 # Current context
                "extract": "attribute",
                "attribute": "class",
                "transform": [
                    {"replace": {"featured": "true"}},
                    {"regex": r"true"}
                ],
                "cast": "boolean",
                "default": False
            }
        }
    }
}
```

### Social Media Post Scraping
```python
query = {
    "posts": {
        "selector": ".post",
        "type": "list",
        "fields": {
            "author": ".author-name",
            "handle": {
                "selector": ".author-handle",
                "transform": [{"regex": r"@(.+)"}]    # Remove @ symbol
            },
            "content": ".post-content",
            "timestamp": {
                "selector": ".timestamp",
                "extract": "attribute", 
                "attribute": "data-time"
            },
            "likes": {
                "selector": ".like-count",
                "transform": [
                    {"regex": r"([\d,]+)"},         # Extract number
                    {"replace": {",": ""}}          # Remove commas
                ],
                "cast": "integer"
            },
            "retweets": {
                "selector": ".retweet-count",
                "transform": [{"regex": r"(\d+)"}],
                "cast": "integer", 
                "default": 0
            },
            "media": {
                "selector": ".media img",
                "type": "list",
                "extract": "attribute",
                "attribute": "src"
            }
        }
    }
}
```

## Error Handling

CSQL provides comprehensive error handling:

### Common Errors
- **Invalid JSON**: Malformed query plans
- **Missing Selector**: Field without required selector
- **Missing Attribute**: Attribute extraction without attribute name
- **Invalid Cast Type**: Unknown type casting
- **Regex Errors**: Invalid regular expressions

### Error Examples
```python
from cutysoup import CSQLError

try:
    result = soup.query({
        "field": {
            "extract": "attribute"  # Missing selector and attribute
        }
    })
except CSQLError as e:
    print(f"Query error: {e}")
```

## API Reference

### CutySoup.query(query_plan)
Execute a CSQL query on the parsed HTML.

**Parameters:**
- `query_plan` (str|dict): JSON string or dictionary containing the query

**Returns:**
- `dict`: Extracted data according to the query plan

### CSQLEngine(soup)
Core CSQL execution engine.

**Methods:**
- `execute_query(query_plan)`: Execute query and return results

### CSQLQuery(soup)
High-level query builder with convenience methods.

**Methods:**
- `execute(query_plan)`: Execute query plan
- `extract_list(selector, fields)`: Extract list of structured objects
- `extract_single(selector, fields)`: Extract single structured object  
- `extract_text_list(selector)`: Extract list of text values
- `extract_attributes(selector, attribute)`: Extract list of attribute values

**Class Methods:**
- `from_html(html)`: Create CSQLQuery from HTML string

## Next Steps: CSQL Phase 2

The current implementation covers **Phase 1** of the CSQL roadmap. Future phases will add:

- **Phase 2**: Advanced DOM traversal (parent, siblings, closest)
- **Phase 3**: Conditional logic and fallback selectors
- **Phase 4**: Variables and reusable query components
- **Phase 5**: Advanced text processing and custom transformations
- **Phase 6**: Performance optimizations and C++ acceleration

---

**CSQL Phase 1 is complete and ready for production use!**

For more examples, see the `demos/csql_demo.py` file.