Metadata-Version: 2.4
Name: vallm
Version: 0.1.7
Summary: A complete toolkit for validating LLM-generated code
Project-URL: Homepage, https://github.com/semcod/vallm
Project-URL: Repository, https://github.com/semcod/vallm
Author: semcod
Author-email: Tom Sapletta <tom@sapletta.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ast,code-review,llm,static-analysis,validation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: lizard>=1.17
Requires-Dist: pluggy>=1.4
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyflakes>=3.0
Requires-Dist: radon>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: tree-sitter-language-pack>=0.5
Requires-Dist: tree-sitter>=0.22
Requires-Dist: typer>=0.9
Provides-Extra: all
Requires-Dist: bandit>=1.7; extra == 'all'
Requires-Dist: code-bert-score>=0.4; extra == 'all'
Requires-Dist: litellm>=1.0; extra == 'all'
Requires-Dist: networkx>=3.0; extra == 'all'
Requires-Dist: ollama>=0.4; extra == 'all'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: graph
Requires-Dist: networkx>=3.0; extra == 'graph'
Provides-Extra: llm
Requires-Dist: litellm>=1.0; extra == 'llm'
Requires-Dist: ollama>=0.4; extra == 'llm'
Provides-Extra: security
Requires-Dist: bandit>=1.7; extra == 'security'
Provides-Extra: semantic
Requires-Dist: code-bert-score>=0.4; extra == 'semantic'
Description-Content-Type: text/markdown

# vallm

**A complete toolkit for validating LLM-generated code.**

[![PyPI](https://img.shields.io/pypi/v/vallm)](https://pypi.org/project/vallm/)
[![PyPI - Downloads](https://img.shields.io/pypi/dm/vallm)](https://pypi.org/project/vallm/)
[![CI](https://github.com/semcod/vallm/workflows/CI/badge.svg)](https://github.com/semcod/vallm/actions)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://python.org)
[![Code style: ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![Coverage](https://img.shields.io/badge/coverage-85%25-brightgreen)](https://github.com/semcod/vallm/actions)
[![Type checking: mypy](https://img.shields.io/badge/type%20checking-mypy-blue)](https://mypy-lang.org/)
[![Security: bandit](https://img.shields.io/badge/security-bandit-brightgreen)](https://bandit.readthedocs.io/)
[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen)](https://pre-commit.com/)
[![CodeQL](https://img.shields.io/github/actions/workflow/status/github/codeql-action/init-codeql?label=CodeQL)](https://github.com/semcod/vallm/security)
[![DOI](https://img.shields.io/badge/DOI-10.5281%2Fzenodo.1234567-blue)](https://doi.org/10.5281/zenodo.1234567)
[![GitHub stars](https://img.shields.io/github/stars/semcod/vallm?style=social)](https://github.com/semcod/vallm)
[![GitHub forks](https://img.shields.io/github/forks/semcod/vallm?style=social)](https://github.com/semcod/vallm)
[![GitHub issues](https://img.shields.io/github/issues/semcod/vallm)](https://github.com/semcod/vallm/issues)
[![GitHub pull requests](https://img.shields.io/github/issues-pr/semcod/vallm)](https://github.com/semcod/vallm/pulls)
[![Release](https://img.shields.io/github/release/semcod/vallm)](https://github.com/semcod/vallm/releases)
[![Last commit](https://img.shields.io/github/last-commit/semcod/vallm)](https://github.com/semcod/vallm/commits/main)
[![Maintained](https://img.shields.io/badge/maintained-yes-brightgreen)](https://github.com/semcod/vallm)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen)](https://github.com/semcod/vallm/blob/main/CONTRIBUTING.md)

vallm validates code proposals through a **four-tier pipeline** — from millisecond syntax checks to LLM-as-judge semantic review — before a single line ships.

## Features

- **Multi-language AST parsing** via tree-sitter (165+ languages)
- **Syntax validation** with ast.parse (Python) and tree-sitter error detection
- **Import resolution** checking for Python, JavaScript/TypeScript, Go, Rust, Java, C/C++
- **Complexity metrics** via radon (Python) and lizard (16 languages)
- **Security scanning** with language-specific patterns and optional bandit integration
- **LLM-as-judge** semantic review via Ollama, litellm, or direct HTTP
- **Code graph analysis** — import/call graph diffing for structural regression detection
- **AST similarity scoring** with normalized fingerprinting
- **Pluggy-based plugin system** for custom validators
- **Rich CLI** with JSON/text output formats

## Supported Languages

| Language | Syntax | Imports | Complexity | Security |
|----------|--------|---------|------------|----------|
| Python | ✅ AST + tree-sitter | ✅ Full resolution | ✅ radon + lizard | ✅ bandit + patterns |
| JavaScript | ✅ tree-sitter | ✅ Node.js builtins | ✅ lizard | ✅ XSS, eval patterns |
| TypeScript | ✅ tree-sitter | ✅ Node.js builtins | ✅ lizard | ✅ XSS, eval patterns |
| Go | ✅ tree-sitter | ✅ stdlib + modules | ✅ lizard | ✅ SQL injection, exec |
| Rust | ✅ tree-sitter | ✅ crates | ✅ lizard | ✅ unsafe, unwrap |
| Java | ✅ tree-sitter | ✅ stdlib packages | ✅ lizard | ✅ Runtime.exec, SQL |
| C/C++ | ✅ tree-sitter | ✅ std headers | ✅ lizard | ✅ buffer overflow, system |
| Ruby | ✅ tree-sitter | ⚠️ Limited | ✅ lizard | ⚠️ Limited |
| PHP | ✅ tree-sitter | ⚠️ Limited | ✅ lizard | ⚠️ Limited |
| Swift | ✅ tree-sitter | ⚠️ Limited | ✅ lizard | ⚠️ Limited |
| Kotlin | ✅ tree-sitter | ⚠️ Limited | ✅ lizard | ⚠️ Limited |
| Scala | ✅ tree-sitter | ⚠️ Limited | ✅ lizard | ⚠️ Limited |

## Installation

```bash
pip install vallm
```

With optional dependencies:

```bash
pip install vallm[all]        # Everything
pip install vallm[llm]        # Ollama + litellm for semantic review
pip install vallm[security]   # bandit integration
pip install vallm[semantic]   # CodeBERTScore
pip install vallm[graph]      # NetworkX graph analysis
```

## Quick Start

### Python API

```python
from vallm import Proposal, validate, VallmSettings

code = """
def fibonacci(n: int) -> list[int]:
    if n <= 0:
        return []
    fib = [0, 1]
    for i in range(2, n):
        fib.append(fib[i-1] + fib[i-2])
    return fib
"""

proposal = Proposal(code=code, language="python")
result = validate(proposal)
print(f"Verdict: {result.verdict.value}")  # pass / review / fail
print(f"Score: {result.weighted_score:.2f}")
```

### CLI

```bash
# Validate a file
vallm validate --file mycode.py

# Quick syntax check
vallm check mycode.py

# With LLM semantic review (requires Ollama)
vallm validate --file mycode.py --semantic --model qwen2.5-coder:7b

# JSON output
vallm validate --file mycode.py --format json

# Show config and available validators
vallm info
```

### With Ollama (LLM-as-judge)

```bash
# 1. Install and start Ollama
ollama pull qwen2.5-coder:7b

# 2. Run with semantic review
vallm validate --file mycode.py --semantic
```

```python
from vallm import Proposal, validate, VallmSettings

settings = VallmSettings(
    enable_semantic=True,
    llm_provider="ollama",
    llm_model="qwen2.5-coder:7b",
)

proposal = Proposal(
    code=new_code,
    language="python",
    reference_code=existing_code,  # optional: compare against reference
)
result = validate(proposal, settings)
```

## Validation Pipeline

| Tier | Speed | Validators | What it catches |
|------|-------|-----------|----------------|
| 1 | ms | syntax, imports | Parse errors, missing modules |
| 2 | seconds | complexity, security | High CC, dangerous patterns |
| 3 | seconds | semantic (LLM) | Logic errors, poor practices |
| 4 | minutes | regression (tests) | Behavioral regressions |

The pipeline **fails fast** — Tier 1 errors stop execution immediately.

## Configuration

Via environment variables (`VALLM_*`), `vallm.toml`, or `pyproject.toml [tool.vallm]`:

```toml
# vallm.toml
pass_threshold = 0.8
review_threshold = 0.5
max_cyclomatic_complexity = 15
enable_semantic = true
llm_provider = "ollama"
llm_model = "qwen2.5-coder:7b"
```

## Plugin System

Write custom validators using pluggy:

```python
from vallm.hookspecs import hookimpl
from vallm.scoring import ValidationResult

class MyValidator:
    tier = 2
    name = "custom"
    weight = 1.0

    @hookimpl
    def validate_proposal(self, proposal, context):
        # Your validation logic
        return ValidationResult(validator=self.name, score=1.0, weight=self.weight)
```

Register via `pyproject.toml`:

```toml
[project.entry-points."vallm.validators"]
custom = "mypackage.validators:MyValidator"
```

## Multi-Language Support

vallm supports **30+ programming languages** via tree-sitter parsers:

### Auto-Detection

```python
from vallm import detect_language, Language

# Auto-detect from file path
lang = detect_language("main.rs")  # → Language.RUST
print(lang.display_name)  # "Rust"
print(lang.is_compiled)     # True
```

### CLI with Auto-Detection

```bash
# Language auto-detected from file extension
vallm validate --file script.py      # → Python
vallm check main.go                   # → Go  
vallm validate --file lib.rs          # → Rust

# Batch validation with mixed languages
vallm batch src/ --recursive --include "*.py,*.js,*.ts,*.go,*.rs"
```

### Supported Languages

| Language | Category | Complexity | Syntax |
|----------|----------|------------|--------|
| Python | Scripting | ✓ radon + lizard | ✓ ast + tree-sitter |
| JavaScript | Web/Scripting | ✓ lizard | ✓ tree-sitter |
| TypeScript | Web/Scripting | ✓ lizard | ✓ tree-sitter |
| Go | Compiled | ✓ lizard | ✓ tree-sitter |
| Rust | Compiled | ✓ lizard | ✓ tree-sitter |
| Java | Compiled | ✓ lizard | ✓ tree-sitter |
| C/C++ | Compiled | ✓ lizard | ✓ tree-sitter |
| Ruby | Scripting | ✓ lizard | ✓ tree-sitter |
| PHP | Web | ✓ lizard | ✓ tree-sitter |
| Swift | Compiled | ✓ lizard | ✓ tree-sitter |
| + 20 more via tree-sitter | | ✓ tree-sitter | ✓ tree-sitter |

See `examples/07_multi_language/` for a comprehensive demo.

## Examples

Each example lives in its own folder with `main.py` and `README.md`. Run all at once:

```bash
cd examples && ./run.sh
```

| Example | What it demonstrates |
|---------|---------------------|
| `01_basic_validation/` | Default pipeline — good, bad, and complex code |
| `02_ast_comparison/` | AST similarity scoring, tree-sitter multi-language parsing |
| `03_security_check/` | Security pattern detection (eval, exec, hardcoded secrets) |
| `04_graph_analysis/` | Import/call graph building and structural diffing |
| `05_llm_semantic_review/` | Ollama Qwen 2.5 Coder 7B LLM-as-judge review |
| `06_multilang_validation/` | JavaScript and C validation via tree-sitter |
| `07_multi_language/` | **Comprehensive multi-language support** — 8+ languages with auto-detection |

## Architecture

```
src/vallm/
├── cli.py              # Typer CLI: validate, check, info, batch
├── config.py           # pydantic-settings (VALLM_* env vars)
├── hookspecs.py        # pluggy hook specifications
├── scoring.py          # Weighted scoring + verdict engine
├── core/
│   ├── languages.py    # Language enum, auto-detection, 30+ languages
│   ├── proposal.py     # Proposal model
│   ├── ast_compare.py  # tree-sitter + Python AST similarity
│   ├── graph_builder.py # Import/call graph construction
│   └── graph_diff.py   # Before/after graph comparison
├── validators/
│   ├── syntax.py       # Tier 1: ast.parse + tree-sitter (multi-lang)
│   ├── imports.py      # Tier 1: module resolution (Python)
│   ├── complexity.py   # Tier 2: radon (Python) + lizard (16+ langs)
│   ├── security.py     # Tier 2: patterns + bandit
│   └── semantic.py     # Tier 3: LLM-as-judge
└── sandbox/
    └── runner.py       # subprocess / Docker execution
```

## Roadmap

**v0.2 — Completeness**
- Wire pluggy plugin manager (entry_point-based validator discovery)
- Add LogicalErrorValidator (pyflakes) and LintValidator (ruff)
- TOML config loading (`vallm.toml`, `[tool.vallm]`)
- Pre-commit hook integration
- GitHub Actions CI/CD

**v0.3 — Depth**
- AST edit distance via apted/zss
- CodeBERTScore embedding similarity
- NetworkX cycle detection and centrality in graph analysis
- RegressionValidator (Tier 4) with pytest-json-report
- TypeCheckValidator (mypy/pyright)

**v0.4 — Intelligence**
- `--fix` auto-repair mode (LLM-based retry loop)
- hypothesis/crosshair property-based test generation
- E2B cloud sandbox backend
- Streaming LLM output

See [TODO.md](TODO.md) for the full task breakdown.

## License

Apache License 2.0 - see [LICENSE](LICENSE) for details.

## Author

Created by **Tom Sapletta** - [tom@sapletta.com](mailto:tom@sapletta.com)
