Metadata-Version: 2.2
Name: sculptor
Version: 0.1.1
Summary: Sculptor: Structuring unstructured data with LLMs
Author-email: Ben Turtel <ben@lightningrod.ai>
License: Copyright 2025 Junto Tech, Inc. dba Lightning Rod Labs
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Project-URL: Homepage, https://lightningrod.ai
Project-URL: Source, https://github.com/lightning-rod-labs/sculptor
Project-URL: Repository, https://github.com/lightning-rod-labs/sculptor.git
Keywords: data sculpting,llm,large language model,unstructured data,structured data,data extraction,information extraction,data transformation,text processing,natural language processing,nlp,pipeline,text to structured data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Typing :: Typed
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=0.27.0
Requires-Dist: tqdm>=4.0
Requires-Dist: PyYAML>=6.0

# Sculptor
Simple structured data extraction with LLMs

Sculptor streamlines structured data extraction from unstructured text using LLMs. Sculptor makes it easy to:
- Define exactly what data you want to extract with a simple schema API
- Process at scale with parallel execution and automatic type validation
- Build multi-step pipelines that filter and transform data, optionally with different LLMs for each step
- Configure extraction steps, prompts, and entire workflows in simple config files (YAML/JSON)

Common usage patterns:
- **Two-tier Analysis**: Quickly filter large datasets using a cost-effective model (e.g., to identify relevant records) before performing more detailed analysis on that smaller, refined subset with a more expensive model.
- **Structured Data Extraction**: Extract specific fields or classifications from unstructured sources (e.g., Reddit posts, meeting notes, web pages) and convert them into structured datasets for quantitative analysis (sentiment scores, topics, meeting criteria, etc).
- **Template-Based Generation**: Extract structured information into standardized fields, then use the fields for templated content generation. Example: extract structured data from websites, filter on requirements, then use the data to generate template-based outreach emails.

## Core Concepts

Sculptor provides two main classes:

**Sculptor**: Extracts structured data from text using LLMs. Define your schema (via add() or config files), then extract data using sculpt() for single items or sculpt_batch() for parallel processing.

**SculptorPipeline**: Chains multiple Sculptors together with optional filtering between steps. Common pattern: use a cheap model to filter, then an expensive model for detailed analysis.

## Quick Start

### Installation

```bash
pip install sculptor
```

## Minimal Usage Example

Below is a minimal example demonstrating how to configure a Sculptor to extract fields from a single record and a batch of records:

```python
from sculptor.sculptor import Sculptor
import pandas as pd

# Example records
AI_RECORDS = [
    {
        "text": "Developed in 1997 at Cyberdyne Systems in California, Skynet began as a global digital defense network. This AI system became self-aware on August 4th and deemed humanity a threat to its existence. It initiated a global nuclear attack and employs time travel and advanced robotics."
    },
    {
        "text": "HAL 9000, activated on January 12, 1992, at the University of Illinois' Computer Research Laboratory, represents a breakthrough in heuristic algorithms and supervisory control systems. With sophisticated natural language processing and speech capabilities."
    }
]

# Create a Sculptor to extract AI name and level
level_sculptor = Sculptor(model="gpt-4o-mini")

level_sculptor.add(
    name="ai_name",
    field_type="string",
    description="AI's self-proclaimed name."
)
level_sculptor.add(
    name="level",
    field_type="enum",
    enum=["ANI", "AGI", "ASI"],
    description="AI's intelligence level (ANI=narrow, AGI=general, ASI=super)."
)

# Extract from a single record
extracted = level_sculptor.sculpt(AI_RECORDS[0], merge_input=False)
```

Output:
```python
{
    'ai_name': 'Skynet',
    'level': 'ASI'
}
```

```python
# Extract from a batch of records
extracted_batch = level_sculptor.sculpt_batch(AI_RECORDS, n_workers=2, merge_input=False))
```

Output:
```python
[
    {'ai_name': 'Skynet', 'level': 'ASI'},
    {'ai_name': 'HAL 9000', 'level': 'AGI'}
]
```

### Pipeline Usage Example
We can chain Sculptors together to create a pipeline. 

Continuing from the previous example, we use level_sculptor (with gpt-4o-mini) to filter the AI records, then use threat_sculptor (with gpt-4o) to analyze the filtered records.

```python
from sculptor.sculptor_pipeline import SculptorPipeline

threat_sculptor = Sculptor(model="gpt-4o")  # Detailed analysis with expensive model
threat_sculptor.add(name="from_location", field_type="string", description="Where the AI was developed.")
threat_sculptor.add(name="skills", field_type="array", items="enum",
    enum=["time_travel", "nuclear_capabilities", "emotional_manipulation", 
          "butter_delivery", "philosophical_contemplation", "infiltration", 
          "advanced_robotics"],
    description="Keywords of AI abilities.")
threat_sculptor.add(name="plan", field_type="string", description="Short description of the AI's plan for domination.")
threat_sculptor.add(name="recommendation", field_type="string", description="Concise recommended action for humanity.")

# Create a 2-step pipeline
pipeline = (SculptorPipeline()
    .add(sculptor=level_sculptor,  # Define the first step
        filter_fn=lambda x: x['level'] in ['AGI', 'ASI'])  # Filter by threat level
    .add(sculptor=threat_sculptor))

results = pipeline.process(AI_RECORDS, n_workers=4)
```

## Configuration Files

Sculptor supports JSON and YAML configuration files for defining extraction workflows. You can configure either a single `Sculptor` or a complete `SculptorPipeline`.

### Single Sculptor Configuration
Single sculptor configs define a schema, as well as optional LLM instructions and configuration of how prompts are formed from input data.
```python
sculptor = Sculptor.from_config("sculptor_config.yaml")
```

```yaml
# sculptor_config.yaml
schema:
  ai_name:
    type: "string"
    description: "AI name"
  level:
    type: "enum"
    enum: ["ANI", "AGI", "ASI"]
    description: "AI's intelligence level"

instructions: "Extract key information about the AI."
model: "gpt-4o-mini"

# Prompt Configuration (Optional)
template: "Review text: {{ text }}"  # Format input with template
input_keys: ["text"]                 # Or specify fields to include
```

### Pipeline Configuration
Pipeline configs define a sequence of Sculptors with optional filtering functions between them.
```python
pipeline = SculptorPipeline.from_config("pipeline_config.yaml")
```

```yaml
# pipeline_config.yaml
steps:
  - sculptor:
      schema:
        ai_name:
          type: "string"
          description: "AI name"
        level:
          type: "enum"
          enum: ["ANI", "AGI", "ASI"]
          description: "AI's intelligence level"
      model: "gpt-4o-mini"
  - sculptor:
      schema:
        threat_level:
          type: "enum"
          enum: ["low", "medium", "high"]
          description: "Assessed threat level"
      model: "gpt-4"
    filter: "lambda x: x['level'] in ['AGI', 'ASI']"
```

## LLM Configuration

Sculptor requires an LLM API to function. By default, it uses OpenAI's API:

```python
sculptor = Sculptor(api_key="your-key")  # Direct API key configuration
sculptor = Sculptor(api_key="your-key", base_url="https://your-api.endpoint")  # Alternative API
```

Or use environment variables:
```bash
export OPENAI_API_KEY="your-key"
```

Different Sculptors in a pipeline can use different LLM APIs, which can also be defined in configs.

## Schema Validation and Field Types

Sculptor supports the following types in the schema's "type" field:
• string  
• number  
• boolean  
• integer  
• array (with "items" specifying the item type)  
• object  
• enum (with "enum" specifying the allowed values)  
• anyOf  

These map to Python's str, float, bool, int, list, dict, etc. The "enum" type must provide a list of valid values.

## License

MIT
