# Files-to-Prompt Deep Dive

## What Is It?

**files-to-prompt** is a lightweight CLI tool that concatenates files from a directory into a single, LLM-optimized prompt. With **2.6k+ stars**, it's the go-to solution for feeding entire codebases to AI assistants.

**Repository:** [simonw/files-to-prompt](https://github.com/simonw/files-to-prompt)

---

## The Problem It Solves

When working with LLMs, you often need to:
- Share entire directories of code
- Respect .gitignore patterns
- Format content for optimal AI comprehension
- Count tokens before submitting

files-to-prompt provides:
- **Smart file concatenation** with path headers
- **Automatic .gitignore** respect
- **Multiple output formats** (plain, XML, Markdown)
- **Pipe-friendly** for Unix workflows

---

## Key Statistics

| Metric | Value |
|--------|-------|
| **Stars** | 2,600 |
| **Forks** | 153 |
| **License** | Apache 2.0 |
| **Latest Version** | 0.6 |
| **Language** | Python |

---

## Basic Usage

```bash
# Concatenate all files in directory
files-to-prompt .

# Specific directory
files-to-prompt src/

# Filter by extension
files-to-prompt . -e py -e js

# Output to file
files-to-prompt . -o context.txt
```

---

## Output Formats

### Default (Plain Text)

```bash
files-to-prompt src/
```

Output:
```
src/main.py
---
def main():
    print("Hello")

src/utils.py
---
def helper():
    pass
```

### Claude XML Format (`--cxml` or `-c`)

Optimized for Claude's extended context window:

```bash
files-to-prompt src/ --cxml
```

Output:
```xml
<documents>
  <document>
    <source>src/main.py</source>
    <document_content>
def main():
    print("Hello")
    </document_content>
  </document>
</documents>
```

### Markdown Format (`--markdown` or `-m`)

```bash
files-to-prompt src/ --markdown
```

Output:
```markdown
## src/main.py

```python
def main():
    print("Hello")
```
```

---

## Filtering Options

### By Extension

```bash
# Only Python and JavaScript
files-to-prompt . -e py -e js

# Only Markdown
files-to-prompt . -e md
```

### Ignore Patterns

```bash
# Ignore test directories
files-to-prompt . --ignore "test*"

# Ignore multiple patterns
files-to-prompt . --ignore "test*" --ignore "build"

# Ignore files only (traverse matching directories)
files-to-prompt . --ignore "__pycache__" --ignore-files-only
```

### .gitignore Behavior

```bash
# Default: respects .gitignore
files-to-prompt .

# Override: include gitignored files
files-to-prompt . --ignore-gitignore

# Include hidden files
files-to-prompt . --include-hidden
```

---

## Input Sources

### Directory

```bash
files-to-prompt src/
```

### Multiple Paths

```bash
files-to-prompt src/ tests/ README.md
```

### From stdin (find, git, etc.)

```bash
# Recent files only
find . -mtime -1 -type f | files-to-prompt

# With null separator
find . -name "*.py" -print0 | files-to-prompt --null

# Git tracked files
git ls-files | files-to-prompt
```

---

## Additional Options

### Line Numbers

```bash
files-to-prompt . -n
# or
files-to-prompt . --line-numbers
```

Output includes line numbers for each file.

### Output to File

```bash
files-to-prompt . -o output.txt
```

---

## Integration with LLM CLI

The primary use case is piping to Simon Willison's `llm` tool:

```bash
# Explain codebase
files-to-prompt src/ | llm "Explain this codebase architecture"

# Generate documentation from tests
files-to-prompt tests/ -e py --cxml | \
  llm -m claude-3.5-sonnet -s "Write usage documentation with examples"

# Code review
files-to-prompt . -e py | llm "Review for security issues"
```

---

## Claude Integration

### Why XML Format?

The `--cxml` flag implements Anthropic's recommended format for extended context:
- Semantic document structure
- Clear source attribution
- Optimized for Claude's processing

### Best Practice

```bash
# For Claude specifically
files-to-prompt . --cxml | llm -m claude-3.5-sonnet "Analyze"

# For other models
files-to-prompt . | llm -m gpt-4o "Analyze"
```

---

## Token Considerations

### files-to-prompt Does NOT Count Tokens

Use external tools:

```bash
# With ttok (Simon Willison's token counter)
files-to-prompt . | ttok
# Output: 63,014 tokens

# Estimate before sending
files-to-prompt . -e py | ttok
```

### Context Window Limits

Even with large context models:
- Very large outputs may only process the end
- Some models have effective limits below advertised
- Always test with representative content

---

## Comparison to Similar Tools

| Tool | Interface | Source | Strengths |
|------|-----------|--------|-----------|
| **files-to-prompt** | CLI | Local | Unix pipes, formats |
| **repo2txt** | Web | GitHub URLs | Visual selection |
| **gpt-repository-loader** | CLI | Local | GPT-4 built |
| **gitingest** | Web | GitHub URLs | Search features |

### files-to-prompt Advantages

1. **CLI-first** - Integrates with Unix workflows
2. **Pipe support** - Works with find, git, grep
3. **Multiple formats** - Plain, XML, Markdown
4. **Lightweight** - Single purpose, fast
5. **Privacy** - Runs locally

---

## Best Practices

### Pre-filter by Extension

```bash
# Reduces token count significantly
files-to-prompt . -e py -e md
```

### Use Ignore Patterns

```bash
# Exclude generated code
files-to-prompt . --ignore "node_modules" --ignore "*.min.js"
```

### Start Specific

```bash
# Better than entire repo
files-to-prompt src/auth/ -e py | llm "Explain auth flow"
```

### Test Token Count First

```bash
# Before sending to expensive models
files-to-prompt . -e py | ttok
```

### Use Appropriate Format

```bash
# Claude → XML
files-to-prompt . --cxml

# GPT/Other → Plain or Markdown
files-to-prompt . --markdown
```

---

## Known Limitations

| Issue | Workaround |
|-------|------------|
| No token counting | Use `ttok` separately |
| Windows subdirectory bug | Open issue #62 |
| Extension suffix matching | `-e py` matches `.spython` |
| Binary files skipped | By design (with warning) |

---

## Development History

| Version | Date | Features |
|---------|------|----------|
| 0.1 | Mar 2024 | Initial release |
| 0.2 | Apr 2024 | --ignore patterns |
| 0.3 | Sep 2024 | --cxml (Claude XML) |
| 0.4 | Oct 2024 | -e extension filter |
| 0.5 | Feb 2025 | Line numbers, Windows fixes |
| 0.6 | Feb 2025 | --markdown, stdin support |

---

## Example Workflows

### Document from Tests

```bash
files-to-prompt tests/ -e py --cxml | \
  llm -m claude-3.5-sonnet \
  -s "Write comprehensive usage documentation with realistic examples"
```

### Quick Code Review

```bash
git diff --name-only | files-to-prompt | \
  llm "Review these changed files for issues"
```

### Architecture Overview

```bash
files-to-prompt src/ -e py --ignore "test*" | \
  llm "Describe the architecture and main components"
```

### Copy to Clipboard

```bash
# macOS
files-to-prompt . --cxml | pbcopy

# Linux
files-to-prompt . --cxml | xclip -i
```

---

## Installation

```bash
# pip
pip install files-to-prompt

# pipx
pipx install files-to-prompt

# uv
uv tool install files-to-prompt
```

---

## Resources

- **Repository:** https://github.com/simonw/files-to-prompt
- **PyPI:** https://pypi.org/project/files-to-prompt/
- **Author:** Simon Willison
- **Built with:** Claude 3 Opus (documented in blog)

---

*files-to-prompt is the simplest way to turn local directories into LLM context. If you're using AI for code analysis, this tool should be in your workflow.*
