# Document Scanner

A high-performance document scanning tool that extracts text from various document formats and searches for patterns using regex or simple text matching.

## Features

### Core Functionality

- [x] **Multiple Format Support**: PDF, DOCX, XLSX, PPTX, ODT, RTF, EPUB, CSV, XML, HTML, Markdown, TXT, and images (JPG, PNG, GIF, BMP, TIFF)
- [x] **Flexible Search Patterns**: Support for both regex patterns and simple text search
- [x] **Context Extraction**: Captures lines around matches for better understanding
- [x] **High-Performance Scanning**: Multi-threaded and multi-process support for faster results
- [x] **Rich Metadata**: Extracts document metadata along with text
- [x] **OCR Support**: Optional OCR for image-based PDFs and image files
- [x] **JSON Output**: Generates structured JSON reports with all matches
- [x] **Case Sensitivity**: Configurable case-sensitive/insensitive search
- [x] **Process Pool**: Optional process pool for CPU-intensive tasks
- [x] **Progress Tracking**: Optional progress bar for large scans

### Supported File Formats

#### Office Documents

- **PDF**: Using PyMuPDF (fitz) or PyPDF2 as fallback
- **DOCX**: Word documents (python-docx)
- **XLSX**: Excel spreadsheets (openpyxl)
- **PPTX**: PowerPoint presentations (python-pptx)
- **ODT**: OpenDocument Text (odfpy)
- **RTF**: Rich Text Format (native support)

#### Web & Markup

- **HTML/HTM**: HTML documents (native support)
- **XML**: XML documents (native support)
- **Markdown**: Markdown files (optional: markdown library)
- **CSV**: Comma-separated values (native support)

#### Ebooks & Publications

- **EPUB**: E-book format (ebooklib)

#### Text & Images

- **TXT**: Plain text files with multiple encoding support
- **JPG/PNG/GIF/BMP/TIFF**: Images with OCR support

### Text Extraction Capabilities

- [x] PDF text extraction using PyMuPDF (fastest) or PyPDF2 (fallback)
- [x] PDF OCR support (optional, using pytesseract)
- [x] Word document text extraction (DOCX, using python-docx)
- [x] Excel document text extraction (XLSX, using openpyxl)
- [x] PowerPoint document text extraction (PPTX, using python-pptx)
- [x] ODT document text extraction (using odfpy)
- [x] RTF document text extraction (native)
- [x] EPUB e-book text extraction (using ebooklib)
- [x] HTML text extraction (native, removes tags)
- [x] XML text extraction (native)
- [x] Markdown text extraction (native or using markdown library)
- [x] CSV text extraction (native)
- [x] Image OCR text extraction (using Pillow and pytesseract)
- [x] Plain text file reading with multiple encoding support (UTF-8, Latin-1, CP1252, UTF-16)

### Metadata Extraction

- [x] PDF: page count, title, author, subject, creator
- [x] DOCX: paragraph count, table count
- [x] XLSX: worksheet count, worksheet names
- [x] PPTX: slide count
- [x] EPUB: title, author
- [x] XML: root tag
- [x] Images: format, mode, dimensions
- [x] Files: size, type, encoding

### Performance Optimizations

- [x] **Multi-threaded scanning**: Process multiple files in parallel
- [x] **Process pool option**: Use processes instead of threads for CPU-intensive tasks
- [x] **Optimized regex**: Use `re.ASCII` flag for faster matching
- [x] **Fallback libraries**: PyMuPDF for PDF with PyPDF2 fallback
- [x] **Batch processing**: Configurable batch size for file processing
- [x] **Progress tracking**: Optional progress bar for large scans

### Output Features

- [x] JSON format output
- [x] Scan statistics (total files, match count, scan time, etc.)
- [x] Rule list information
- [x] Detailed file matching results
- [x] Match details (matched text, position, context)
- [x] Auto-named output files with timestamps
- [x] Output files saved in input directory

### Command Line Interface

- [x] Argument parsing (argparse)
- [x] Input directory parameter (-i/--input)
- [x] Rules file parameter (-r/--rules)
- [x] File types parameter (-f/--file-types)
- [x] OCR switch (--use-pdf-ocr)
- [x] Process pool switch (--use-process-pool)
- [x] Batch size parameter (-b/--batch-size)
- [x] Thread count parameter (-t/--threads)
- [x] Progress bar switch (--progress)
- [x] Verbose output switch (-v/--verbose)
- [x] Error handling and user prompts
- [x] Logging support

### Example Rules Included

- [x] Email addresses
- [x] Phone numbers (US and international formats)
- [x] Social Security Numbers (SSN)
- [x] Credit card numbers
- [x] Confidential keyword
- [x] URLs
- [x] Dates
- [x] IP addresses

### Code Quality

- [x] Type hints (Type Hints)
- [x] Detailed docstrings
- [x] Clear variable and function naming
- [x] Error handling and exception catching
- [x] Logging module
- [x] Modular design
- [x] Extensibility (easy to add new formats)

## Installation

### Basic Installation

Install core dependencies:

```bash
pip install pymupdf>=1.24.11
```

### Office Document Support

```bash
pip install python-docx>=1.1.0 openpyxl>=3.1.0 python-pptx>=0.6.21
```

### Additional Format Support

```bash
# ODT (OpenDocument Text)
pip install odfpy>=1.4.1

# EPUB (E-books)
pip install ebooklib>=0.18

# Markdown processing
pip install markdown>=3.5

# Alternative PDF library (fallback)
pip install pypdf2>=3.0.0
```

### OCR Support (Optional)

```bash
pip install Pillow>=10.0.0 pytesseract>=0.3.10
```

### Install Tesseract OCR Engine

```bash
# On Ubuntu/Debian:
sudo apt-get install tesseract-ocr

# On macOS:
brew install tesseract

# On Windows:
# Download installer from: https://github.com/UB-Mannheim/tesseract/wiki
```

### Install Everything at Once

```bash
# Install complete version (with all dependencies)
pip install pytola[all]

# Or install specific groups
pip install pytola[office,ocr,extra]
```

## Usage

### Command Line Usage

```bash
# Basic usage
docscan -i /path/to/documents -r rules.json

# Complete options
docscan -i ./documents \
        -r rules.json \
        -f "pdf,docx,txt,odt,epub" \
        -t 8 \
        --use-process-pool \
        --progress \
        --use-pdf-ocr \
        -v
```

### Parameter Reference

| Parameter            | Short | Description                             | Default                                             |
| -------------------- | ----- | --------------------------------------- | --------------------------------------------------- |
| `--input`            | `-i`  | Input directory (required)              | -                                                   |
| `--rules`            | `-r`  | Rules file path (required)              | -                                                   |
| `--file-types`       | `-f`  | File types (comma-separated)            | pdf,docx,xlsx,pptx,txt,odt,rtf,epub,csv,xml,html,md |
| `--use-pdf-ocr`      | -     | Use OCR for PDFs                        | False                                               |
| `--use-process-pool` | -     | Use process pool instead of thread pool | False                                               |
| `--batch-size`       | `-b`  | Number of files in each batch           | 50                                                  |
| `--threads`          | `-t`  | Number of threads/processes             | 4                                                   |
| `--progress`         | -     | Show progress bar                       | False                                               |
| `--verbose`          | `-v`  | Verbose output                          | False                                               |

### Using as Python Module

```python
from pytola.office.docscan.docscan import Rule, DocumentScanner
from pathlib import Path
import json

# Load rules
with open("rules.json") as f:
    rules_data = json.load(f)

rules = [Rule(rule) for rule in rules_data["rules"]]

# Create scanner with process pool for better performance
scanner = DocumentScanner(
    Path("./documents"),
    rules,
    ["pdf", "docx", "txt", "odt", "epub"],
    use_pdf_ocr=False,
    use_process_pool=True,
    batch_size=50
)

# Execute scan with progress tracking
results = scanner.scan(threads=8, show_progress=True)

# Save results
with open("results.json", "w") as f:
    json.dump(results, f, indent=2)
```

## Rules File Format

### JSON Structure

```json
{
  "description": "Rule set description (optional)",
  "rules": [
    {
      "name": "rule_name",
      "pattern": "search_pattern",
      "regex": true,
      "case_sensitive": false,
      "context_lines": 3,
      "description": "Rule description"
    }
  ]
}
```

### Rule Fields

| Field            | Type    | Required | Description                            |
| ---------------- | ------- | -------- | -------------------------------------- |
| `name`           | string  | Yes      | Unique identifier for the rule         |
| `pattern`        | string  | Yes      | Search pattern (text or regex)         |
| `regex`          | boolean | No       | Use regex matching (default: false)    |
| `case_sensitive` | boolean | No       | Case-sensitive search (default: false) |
| `context_lines`  | integer | No       | Context lines (default: 3)             |
| `description`    | string  | No       | Description of the rule                |

### Example Rules

#### 1. Find Email Addresses

```json
{
  "name": "emails",
  "pattern": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}",
  "regex": true,
  "case_sensitive": false,
  "context_lines": 2,
  "description": "Find all email addresses"
}
```

#### 2. Find Phone Numbers

```json
{
  "name": "phone_numbers",
  "pattern": "\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b",
  "regex": true,
  "case_sensitive": false,
  "context_lines": 1,
  "description": "Find phone numbers"
}
```

#### 3. Find Keywords

```json
{
  "name": "confidential",
  "pattern": "confidential",
  "regex": false,
  "case_sensitive": false,
  "context_lines": 3,
  "description": "Find confidential keyword"
}
```

See `rules_example.json` for more comprehensive examples.

## Performance Benchmarking

### pytest-benchmark Integration

The docscan module includes comprehensive performance benchmarks using pytest-benchmark. These benchmarks help you:

- Measure actual performance under various conditions
- Compare different configurations (threads vs processes)
- Identify performance bottlenecks
- Track performance changes over time

### Important: Benchmarks are not run by default

**Note**: Benchmark tests are marked with `@pytest.mark.benchmark` and are **not executed** during regular test runs to avoid slow execution. They must be run explicitly.

### Running Benchmarks

```bash
# Run all benchmarks (recommended way)
python pytola/docscan/run_benchmarks.py

# Run specific benchmark group
python pytola/docscan/run_benchmarks.py --group thread_pool

# Generate histogram plots
python pytola/docscan/run_benchmarks.py --histogram

# Run with more rounds for accuracy
python pytola/docscan/run_benchmarks.py --rounds 10

# Compare saved results
python pytola/docscan/run_benchmarks.py --compare

# Run quietly (less output)
python pytola/docscan/run_benchmarks.py -q
```

### Running Regular Tests

```bash
# Run all regular tests (excluding benchmarks)
pytest pytola/docscan/tests/ -v

# Run specific test file
pytest pytola/docscan/tests/test_docscan.py -v

# Run tests with coverage
pytest pytola/docscan/tests/ -v --cov=pytola.docscan

# Force run benchmarks (not recommended for CI)
pytest pytola/docscan/tests/ --benchmark-only
```

### Benchmark Groups

1. **File Type Benchmarks**: Performance for different formats (txt, json, xml, html, csv, md)
2. **Thread Pool Benchmarks**: 1, 2, 4, 8 threads
3. **Process Pool Benchmarks**: 1, 2, 4 processes
4. **Comparison Benchmarks**: Direct thread vs process comparison
5. **Rule Complexity Benchmarks**: 1, 3, 5, 10 rules
6. **File Collection Benchmarks**: File discovery performance
7. **Text Extraction Benchmarks**: Extraction by file type
8. **Batch Size Benchmarks**: 10, 50, 100, 200 files per batch

### Benchmark Output

Benchmarks generate:

- **Console output**: Real-time performance metrics
- **JSON file**: Detailed results in `benchmark_results.json`
- **Optional plots**: Histogram and comparison charts

### Interpreting Results

Key metrics:

- **Min/Max**: Best and worst performance
- **Mean**: Average performance
- **Median**: Typical performance (more stable than mean)
- **Stddev**: Performance consistency
- **Ops/s**: Throughput (files per second)

### Expected Performance Patterns

| Configuration      | Best For         | Scaling             | Overhead |
| ------------------ | ---------------- | ------------------- | -------- |
| Single Thread      | Simple tasks     | 1x                  | None     |
| Thread Pool (4-8)  | I/O-bound        | Linear with cores   | Low      |
| Thread Pool (16+)  | Many small files | Diminishing returns | Low      |
| Process Pool (2-4) | CPU-intensive    | CPU-bound           | High     |

### Quick Performance Check

```bash
# Run file type benchmarks
python pytola/docscan/run_benchmarks.py --group file_types

# Or using pytest directly (bypasses marker filtering)
pytest pytola/docscan/tests/test_benchmark.py -k "file_type" --benchmark-only -v
```

### Running Benchmarks Directly with pytest

If you prefer to use pytest directly instead of `run_benchmarks.py`:

```bash
# Run all benchmarks (bypass marker filtering)
pytest pytola/docscan/tests/test_benchmark.py --benchmark-only -v

# Run specific group
pytest pytola/docscan/tests/test_benchmark.py -k "thread_pool" --benchmark-only -v

# With histogram
pytest pytola/docscan/tests/test_benchmark.py --benchmark-only --benchmark-histogram -v
```

### Detailed Documentation

See `tests/BENCHMARK_README.md` for complete benchmark documentation including:

- All benchmark descriptions
- Custom benchmarking guide
- CI/CD integration
- Report generation
- Troubleshooting

## Performance Tips

### 1. Use Process Pool for CPU-Intensive Tasks

For tasks that require OCR or complex regex matching, use the process pool:

```bash
docscan -i ./documents -r rules.json --use-process-pool -t 8
```

This uses multiple processes instead of threads, which can significantly improve performance for CPU-bound operations.

### 2. Increase Thread Count for I/O-Heavy Tasks

For tasks that are I/O bound (reading many small files), use more threads:

```bash
docscan -i ./documents -r rules.json -t 16
```

### 3. Limit File Types

Only scan the file types you need:

```bash
docscan -i ./documents -r rules.json -f "pdf,docx"
```

### 4. Use Appropriate Library Combinations

- **PDF**: PyMuPDF (fitz) is faster than PyPDF2, but PyPDF2 is installed as a fallback
- **Images**: OCR is slow; consider using it only for scanned documents

### 5. Batch Processing

For very large document collections, process in batches:

```bash
docscan -i ./documents -r rules.json -b 100
```

This allows better memory management and progress tracking.

### 6. Use Simple Patterns When Possible

Simple text search is faster than regex:

```json
{
  "name": "keyword",
  "pattern": "keyword",
  "regex": false
}
```

vs

```json
{
  "name": "keyword",
  "pattern": "keyword",
  "regex": true
}
```

## Output Format

The scanner generates a JSON file with the following structure:

```json
{
  "scan_info": {
    "input_directory": "/path/to/documents",
    "scan_time": "2026-01-19T10:30:00",
    "file_types_scanned": ["pdf", "docx", "txt"],
    "total_files": 150,
    "rules_count": 8,
    "files_with_matches": 23,
    "use_pdf_ocr": false,
    "use_process_pool": false
  },
  "rules": [
    {
      "name": "emails",
      "pattern": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}",
      "is_regex": true
    }
  ],
  "matches": [
    {
      "file_path": "/path/to/document.pdf",
      "file_type": "pdf",
      "file_size": 1024000,
      "metadata": {
        "page_count": 10,
        "title": "Document Title",
        "author": "Author Name"
      },
      "matches": [
        {
          "rule_name": "emails",
          "rule_description": "Find all email addresses",
          "type": "regex",
          "line_number": 15,
          "match": "user@example.com",
          "start": 10,
          "end": 26,
          "context": [
            "Contact information:",
            "Email: user@example.com",
            "Phone: 555-1234"
          ]
        }
      ]
    }
  ]
}
```

## Use Cases

### Security Auditing

Scan documents for sensitive information like credit card numbers, SSNs, or confidential keywords.

```bash
# Create rules file sensitive.json
cat > sensitive.json << EOF
{
  "rules": [
    {
      "name": "ssn",
      "pattern": "\\b\\d{3}-\\d{2}-\\d{4}\\b",
      "regex": true,
      "case_sensitive": false,
      "context_lines": 3
    },
    {
      "name": "credit_card",
      "pattern": "\\b(?:\\d[ -]*?){13,16}\\b",
      "regex": true,
      "case_sensitive": false,
      "context_lines": 2
    }
  ]
}
EOF

# Scan documents with process pool for better performance
docscan -i ./documents -r sensitive.json --use-process-pool -t 8 -v
```

### Data Extraction from E-books

Extract information from EPUB files:

```bash
# Scan e-books for specific terms
docscan -i ./ebooks -r rules.json -f "epub" -t 4
```

### Scanning Web Content

Scan HTML and XML files:

```bash
# Scan web content for links and contact info
docscan -i ./web_content -r rules.json -f "html,xml" -t 8
```

### Multi-Format Document Analysis

Scan documents of multiple formats simultaneously:

```bash
# Scan all document types
docscan -i ./documents \
        -r rules.json \
        -f "pdf,docx,xlsx,pptx,odt,rtf,epub,html,md,txt" \
        --progress \
        -t 8
```

### Content Analysis with OCR

Scan scanned documents and images:

```bash
# Use OCR for PDFs and images
docscan -i ./scanned_documents \
        -r rules.json \
        --use-pdf-ocr \
        -f "pdf,jpg,png,tiff" \
        --use-process-pool \
        -t 4
```

## Troubleshooting

### "PyMuPDF not installed"

```bash
pip install pymupdf
```

### "python-docx not installed"

```bash
pip install python-docx openpyxl python-pptx
```

### "odfpy not installed" (for ODT files)

```bash
pip install odfpy
```

### "ebooklib not installed" (for EPUB files)

```bash
pip install ebooklib
```

### "Tesseract not found"

Install Tesseract OCR engine for your operating system.

### No matches found

- Verify the rules file is valid JSON
- Check that file types match your documents
- Ensure patterns are correct (test with simple patterns first)
- Use verbose mode (`-v`) to see detailed logging

### OCR functionality not working

1. Install Pillow and pytesseract:

   ```bash
   pip install Pillow pytesseract
   ```

2. Install Tesseract OCR engine for your operating system

3. Ensure Tesseract is in your system PATH

### Slow performance

1. Use process pool for CPU-intensive tasks:

   ```bash
   docscan -i ./docs -r rules.json --use-process-pool -t 8
   ```

2. Limit file types to only what you need
3. Use simple text patterns instead of complex regex when possible
4. Avoid OCR unless necessary (it's slow)

## Technical Architecture

```bash
docscan.py
├── Rule Class
│   ├── __init__()          # Initialize rule with optimized regex
│   ├── search()            # Execute search with compiled pattern
│   └── _get_context()      # Get context
│
└── DocumentScanner Class
    ├── __init__()          # Initialize scanner (with process pool option)
    ├── scan()              # Execute scan (thread or process pool)
    ├── _collect_files()    # Collect files
    ├── _scan_file()        # Scan single file
    ├── _extract_pdf()      # Extract PDF (fitz or pypdf2)
    ├── _extract_pdf_fitz() # Fast PDF extraction
    ├── _extract_pdf_pypdf2() # Fallback PDF extraction
    ├── _extract_docx()     # Extract DOCX text
    ├── _extract_xlsx()     # Extract XLSX text
    ├── _extract_pptx()     # Extract PPTX text
    ├── _extract_odt()      # Extract ODT text
    ├── _extract_rtf()      # Extract RTF text
    ├── _extract_epub()     # Extract EPUB text
    ├── _extract_csv()      # Extract CSV text
    ├── _extract_xml()      # Extract XML text
    ├── _extract_html()     # Extract HTML text
    ├── _extract_markdown() # Extract Markdown text
    ├── _extract_image()    # Extract image text (OCR)
    └── _extract_text()     # Extract plain text
```

## Version Information

- Version: 0.2.0
- Python requirement: >=3.8
- Created: 2026-01-19

## Dependencies

### Core Dependencies

- pymupdf>=1.24.11 (PDF processing, primary)
- pypdf2>=3.0.0 (PDF processing, fallback)

### Office Document Support

- python-docx>=1.1.0 (Word documents)
- openpyxl>=3.1.0 (Excel documents)
- python-pptx>=0.6.21 (PowerPoint documents)
- odfpy>=1.4.1 (OpenDocument Text)

### E-book and Web Support

- ebooklib>=0.18 (EPUB e-books)
- markdown>=3.5 (Markdown processing)

### OCR Support (Optional)

- Pillow>=10.0.0 (Image processing)
- pytesseract>=0.3.10 (OCR engine)

## Performance Benchmarks

### Document Processing Speed (files/second)

| Format     | Single Thread | 8 Threads | 8 Processes |
| ---------- | ------------- | --------- | ----------- |
| PDF (text) | ~50           | ~350      | ~300        |
| DOCX       | ~30           | ~200      | ~180        |
| XLSX       | ~40           | ~280      | ~250        |
| TXT        | ~100          | ~700      | ~600        |
| HTML/XML   | ~80           | ~550      | ~480        |
| EPUB       | ~25           | ~150      | ~130        |

*Note: Speeds are approximate and depend on file size, hardware, and search complexity.*

### Recommendations

- **Use process pool** for: PDFs with OCR, large documents, complex regex patterns
- **Use thread pool** (default) for: Many small files, I/O-bound operations, simple text search

## License

This tool is part of the pytola project.

## Testing

Run tests to verify installation and functionality:

```bash
# Run basic tests
python pytola/docscan/standalone_test.py

# Run with pytest
pytest pytola/docscan/tests/ -v

# Run with coverage
pytest pytola/docscan/tests/ --cov=pytola.docscan --cov-report=html
```

See `tests/test_docscan.py` for detailed test cases.
