# MANSPIDER Integration Test Plan

## Overview

Create an end-to-end integration test that:
1. Spins up an SMB server using impacket's `smbserver.py`
2. Populates it with test files (various formats and encodings)
3. Runs MANSPIDER against it
4. Verifies all files are discovered and text is properly extracted

## Prerequisites

- impacket is already a dependency (used for SMB client in `man_spider/lib/smb.py`)
- Test data files already exist in `testdata/`
- pytest is available

## Architecture Understanding

Key files:
- `man_spider/manspider.py` - CLI entry point, `main()` and `go(options)`
- `man_spider/lib/spider.py` - MANSPIDER orchestrator class
- `man_spider/lib/spiderling.py` - Worker that crawls shares and processes files
- `man_spider/lib/parser/parser.py` - Text extraction via Kreuzberg (line 141: `extract_file_sync()`)
- `man_spider/lib/smb.py` - Impacket SMB client wrapper

Text extraction flow:
1. Spiderling downloads file to `/tmp/.manspider/` via `RemoteFile.get()`
2. `FileParser.extract_text()` calls `extract_file_sync()` from Kreuzberg
3. Content is matched against regex filters

## Task 1: Clean Up Kreuzberg Tests

Remove text encoding tests from `tests/test_kreuzberg.py` - text handling will be done elsewhere.

**File:** `tests/test_kreuzberg.py`

Keep only:
```python
@pytest.mark.parametrize("filename", [
    "test.docx",
    "test.pdf",
    "test.xlsx",
    "test.png",
    "test.doc",
    "test.xls",
])
```

Remove: test-ascii.txt, test-utf8.txt, test-utf8-bom.txt, test-utf16le.txt, test-utf16be.txt, test-utf16-bom.txt, test-latin1.txt, test-cp1252.txt

## Task 2: Add Text Encoding Detection to Parser

**File:** `man_spider/lib/parser/parser.py`

Before calling Kreuzberg's `extract_file_sync()`, detect if file is plain text using charset-normalizer.

```python
from charset_normalizer import from_path

def is_text_file(filepath):
    """Detect if file is plain text using charset-normalizer."""
    result = from_path(filepath)
    return result.best() is not None

def extract_text_file(filepath):
    """Extract text from plain text file, auto-detecting encoding."""
    result = from_path(filepath)
    best = result.best()
    return str(best) if best else None
```

Modify `extract_text()` method (around line 130):
1. First check `is_text_file()`
2. If True, use `extract_text_file()` directly (handles UTF-16, etc.)
3. If False, fall back to Kreuzberg for binary formats (docx, pdf, etc.)

**Note:** Check if there's existing encoding detection in `util.py` line 100-109 (`better_decode()`) that uses libmagic - may need to reconcile or replace.

## Task 3: Create SMB Integration Test

**File:** `tests/test_smb_integration.py`

### 3.1 SMB Server Setup

Use impacket's SimpleSMBServer:

```python
import tempfile
import shutil
import threading
import time
from pathlib import Path
from impacket.smbserver import SimpleSMBServer

class SMBTestServer:
    def __init__(self, share_path, port=4455):
        self.share_path = share_path
        self.port = port
        self.server = None
        self.thread = None

    def start(self):
        # Create SMB server with anonymous access
        self.server = SimpleSMBServer(listenAddress="127.0.0.1", listenPort=self.port)
        self.server.addShare("testshare", self.share_path, "Test Share")

        # Allow anonymous/guest access
        self.server.setSMB2Support(True)

        self.thread = threading.Thread(target=self.server.start)
        self.thread.daemon = True
        self.thread.start()
        time.sleep(0.5)  # Wait for server to start

    def stop(self):
        if self.server:
            self.server.stop()
```

### 3.2 Test Fixture

```python
import pytest

@pytest.fixture
def smb_server(tmp_path):
    """Spin up SMB server with test files."""
    # Copy test files to temp share directory
    testdata = Path(__file__).parent.parent / "testdata"
    share_path = tmp_path / "share"
    share_path.mkdir()

    # Copy all test files
    for f in testdata.iterdir():
        if f.is_file():
            shutil.copy(f, share_path / f.name)

    # Create subdirectory structure for depth testing
    subdir = share_path / "subdir"
    subdir.mkdir()
    shutil.copy(testdata / "test.docx", subdir / "nested.docx")

    server = SMBTestServer(str(share_path), port=4455)
    server.start()
    yield server, share_path
    server.stop()
```

### 3.3 Integration Tests

```python
def test_manspider_discovers_all_files(smb_server, tmp_path):
    """MANSPIDER finds all files on SMB share."""
    server, share_path = smb_server
    loot_dir = tmp_path / "loot"

    # Build options similar to CLI
    from man_spider.lib.spider import MANSPIDER

    # Create options object (see manspider.py for structure)
    options = create_test_options(
        targets=["127.0.0.1"],
        port=4455,
        loot=str(loot_dir),
        # Use guest/anonymous auth
        username="",
        password="",
    )

    spider = MANSPIDER(options)
    spider.start()

    # Verify files were found
    # Check loot directory or spider output
    assert (loot_dir / "test.docx").exists()
    # ... more assertions


def test_manspider_extracts_password_from_all_formats(smb_server, tmp_path):
    """MANSPIDER extracts 'Password123' from various file formats."""
    server, share_path = smb_server
    loot_dir = tmp_path / "loot"

    options = create_test_options(
        targets=["127.0.0.1"],
        port=4455,
        loot=str(loot_dir),
        content=["Password123"],  # Content filter
    )

    spider = MANSPIDER(options)
    spider.start()

    # All files containing Password123 should be in loot
    expected_files = [
        "test.docx", "test.pdf", "test.xlsx", "test.doc", "test.xls",
        "test-utf16le.txt", "test-utf8.txt",  # Text files
    ]
    for filename in expected_files:
        assert any(f.name == filename for f in loot_dir.rglob("*")), \
            f"{filename} not found in loot"


def test_manspider_handles_text_encodings(smb_server, tmp_path):
    """MANSPIDER correctly extracts text from UTF-16 and other encodings."""
    server, share_path = smb_server
    loot_dir = tmp_path / "loot"

    options = create_test_options(
        targets=["127.0.0.1"],
        port=4455,
        loot=str(loot_dir),
        content=["Password123"],
        extensions=[".txt"],  # Only text files
    )

    spider = MANSPIDER(options)
    spider.start()

    # All text encoding variants should match
    txt_files = list(loot_dir.rglob("*.txt"))
    assert len(txt_files) >= 8, f"Expected 8 txt files, got {len(txt_files)}"
```

### 3.4 Helper Function

```python
def create_test_options(**kwargs):
    """Create options object matching manspider CLI structure."""
    from argparse import Namespace

    defaults = {
        'targets': [],
        'port': 445,
        'username': '',
        'password': '',
        'domain': '',
        'hash': '',
        'loot': '/tmp/manspider_test_loot',
        'threads': 2,
        'maxdepth': 10,
        'content': [],
        'filenames': [],
        'extensions': [],
        'exclude_extensions': [],
        'dirnames': [],
        'exclude_dirnames': [],
        'shares': [],
        'exclude_shares': [],
        'max_filesize': '10M',
        'or_logic': False,
        'quiet': True,
        'no_download': False,
    }
    defaults.update(kwargs)
    return Namespace(**defaults)
```

## Task 4: Test for Binary Files with Embedded Text

Add tests for the binary test files that have "Password123" embedded:

```python
def test_manspider_finds_password_in_binary(smb_server, tmp_path):
    """MANSPIDER finds Password123 in arbitrary binary files."""
    server, share_path = smb_server
    loot_dir = tmp_path / "loot"

    options = create_test_options(
        targets=["127.0.0.1"],
        port=4455,
        loot=str(loot_dir),
        content=["Password123"],
        extensions=[".bin"],
    )

    spider = MANSPIDER(options)
    spider.start()

    # Binary files with embedded password should be found
    bin_files = list(loot_dir.rglob("*.bin"))
    assert len(bin_files) >= 1
```

## Implementation Order

1. **Task 1**: Clean up kreuzberg tests (5 min)
2. **Task 2**: Add charset-normalizer text detection to parser (30 min)
   - Explore existing `better_decode()` in util.py first
   - Modify `extract_text()` in parser.py
   - Run existing tests to verify no regression
3. **Task 3**: Create SMB integration test infrastructure (1-2 hours)
   - Set up SMB server fixture
   - Create helper functions
   - Write basic discovery test
   - Write content extraction test
4. **Task 4**: Add binary file detection tests (15 min)

## Key Files to Modify

| File | Change |
|------|--------|
| `tests/test_kreuzberg.py` | Remove text encoding parametrization |
| `man_spider/lib/parser/parser.py` | Add charset-normalizer detection before Kreuzberg |
| `tests/test_smb_integration.py` | New file - full integration test suite |
| `pyproject.toml` | May need to add charset-normalizer if not already dependency |

## Notes

- `charset-normalizer` is already installed (dependency of `requests`)
- impacket's smbserver requires root/elevated permissions on port 445, so use high port (4455)
- MANSPIDER uses multiprocessing - tests may need to handle process cleanup
- Check `man_spider/lib/util.py` line 100-109 for existing `better_decode()` function that uses libmagic for encoding detection - may want to replace with charset-normalizer approach

## Verification

After implementation, run:
```bash
uv run pytest tests/ -v
```

All tests should pass, including:
- Kreuzberg extraction tests (docx, pdf, xlsx, png, doc, xls)
- SMB integration tests (file discovery, content extraction, encoding handling)
