# CVE Extractor — Design Specification

---

## Design Overview

**Classes**: `CVERecord`, `GitHubPatch` (abstract), concrete patch subclasses

**Relationships**: One-way only. `CVERecord` is immutable data holder. `GitHubPatch` (abstract) contains `CommitPatch`, `FilePatch`, `PRPatch`, `ComparePatch` (concrete subclasses). No wrapper classes for output.

**Exceptions**: Standard Python exceptions only. No custom exception types. All public functions raise exceptions on errors; callers handle via try/except.

**Dependency roles**:
- Data holders: `CVERecord`, `GitHubPatch` (abstract), patch subclasses (pure, immutable)
- Helpers: All module-level functions (stateless, receive all configuration as parameters, no global state)
- Return values: Primitive types, tuples, dicts (no wrapper classes)

**Module organization**:
- `core.models` — CVE data models (`CVERecord`)
- `core.downloader` — CVE data acquisition (downloads, extracts nested archives)
- `core.filter` — CVE filtering by CWE type and keywords
- `core.github` — GitHub URL extraction and code type verification
- `core.extractor` — CVE information extraction to CSV from loaded objects
- `core.analyzer` — CVE distribution analysis from CSV

**Key design principles**:
1. All configuration passed as function parameters. No `Config` singleton.
2. Per-CVE functions process single `CVERecord` objects, not lists.
3. Only two bulk functions: `load_cves_from_directory()` (input) and `extract_cves_to_csv()` (output).
4. Callers load files once, iterate over objects, collect results explicitly.
5. No batch functions. No hidden loops in library code.

---

## Class / Type Specifications

### `CVERecord`

**Responsibility**: Immutable representation of a single CVE from JSON with parsed fields accessible as attributes.

**State/Fields**:

| Field | Type | Description |
|-------|------|-------------|
| `cve_id` | `str` | CVE identifier (e.g., `"CVE-2024-1234"`) |
| `cwe_types` | `list[str]` | CWE classifications (e.g., `["CWE-79", "CWE-89"]`) |
| `description` | `str` | Full vulnerability description |
| `affected_products` | `list[str]` | Affected software products |
| `references` | `list[str]` | Reference URLs (may contain GitHub patches) |
| `published_date` | `str` | Publication date (ISO format) |

**Methods**:

`__init__(cve_data: dict)`
- **Behavior**: Parse CVE JSON and initialize fields
- **Input**: Raw CVE JSON dictionary
- **Output**: None
- **Errors**: `KeyError` if required fields missing, `ValueError` if data malformed

`@staticmethod load_from_file(file_path: Path) -> CVERecord`
- **Behavior**: Load and parse CVE from JSON file
- **Input**: `file_path` — Path to CVE JSON file
- **Output**: `CVERecord` instance
- **Errors**: `OSError` on file read, `json.JSONDecodeError` on parse, `KeyError`/`ValueError` on schema

`matches_cwe_type(cwe_types: set[str]) -> bool`
- **Behavior**: Check if CVE has any CWE type in the provided set
- **Input**: `cwe_types` — Set of CWE identifiers to match
- **Output**: `True` if any match found, `False` otherwise
- **Errors**: None

`matches_keywords(keywords: set[str]) -> bool`
- **Behavior**: Check if description or product contains any keyword (case-insensitive)
- **Input**: `keywords` — Keywords to search
- **Output**: `True` if any match found, `False` otherwise
- **Errors**: None

**Example usage**:

```python
# Load once, reuse multiple times
cve = CVERecord.load_from_file(Path("CVE-2024-1234.json"))

# Check properties without reloading file
if cve.matches_cwe_type({"CWE-79"}):
    process(cve)
```

---

### `GitHubPatch` (Abstract)

**Responsibility**: Base class for different GitHub patch URL types. Extracted from CVE references.

**State/Fields**:

| Field | Type | Description |
|-------|------|-------------|
| `cve_id` | `str` | Source CVE identifier |
| `repository` | `str` | Repository in `owner/repo` format |

**Abstract Methods** (implemented by subclasses):

`get_patch_commit_sha() -> str`
- **Behavior**: Extract patch commit SHA from this patch instance
- **Output**: Commit SHA (e.g., `"abc123def456"`)
- **Errors**: Raises `ValueError` if SHA cannot be extracted

`get_patch_files() -> list[str]`
- **Behavior**: Return list of file paths modified by this patch (if available)
- **Output**: List of file paths; `[]` if not applicable for this patch type
- **Errors**: None

---

### `CommitPatch` (Subclass of `GitHubPatch`)

**Responsibility**: Represents a full commit patch (`github.com/owner/repo/commit/sha`).

**State/Fields** (inherits from parent):

| Field | Type | Description |
|-------|------|-------------|
| `commit_sha` | `str` | Commit SHA |

**Example**: `CommitPatch(cve_id="CVE-2024-1234", repository="owner/repo", commit_sha="abc123")`

---

### `FilePatch` (Subclass of `GitHubPatch`)

**Responsibility**: Represents a file-specific patch (`github.com/owner/repo/blob/sha/file/path`).

**State/Fields** (inherits from parent):

| Field | Type | Description |
|-------|------|-------------|
| `commit_sha` | `str` | Commit SHA |
| `file_path` | `str` | File path at that commit |

**Example**: `FilePatch(cve_id="CVE-2024-1234", repository="owner/repo", commit_sha="abc123", file_path="index.php")`

---

### `PRPatch` (Subclass of `GitHubPatch`)

**Responsibility**: Represents a pull request patch (`github.com/owner/repo/pull/number`).

**State/Fields** (inherits from parent):

| Field | Type | Description |
|-------|------|-------------|
| `pr_number` | `int` | Pull request number |
| `merge_commit_sha` | `str \| None` | Merge commit SHA (resolved via GitHub API) |

**Example**: `PRPatch(cve_id="CVE-2024-1234", repository="owner/repo", pr_number=1234, merge_commit_sha="abc123")`

---

### `ComparePatch` (Subclass of `GitHubPatch`)

**Responsibility**: Represents a compare/range patch (`github.com/owner/repo/compare/sha1...sha2`).

**State/Fields** (inherits from parent):

| Field | Type | Description |
|-------|------|-------------|
| `base_sha` | `str` | Base commit SHA (sha1) |
| `head_sha` | `str` | Head commit SHA (sha2) |

**Example**: `ComparePatch(cve_id="CVE-2024-1234", repository="owner/repo", base_sha="aaa111", head_sha="bbb222")`

---


## Function Specifications

### Downloader (`core.downloader`)

**`download_and_extract_cve_data(url: str, cache_dir: Path, target_dir: Path) -> Path`**

- **Responsibility**: Download and extract CVE List v5 dataset from URL
- **Behavior**:
  1. Check cache for valid file (7-day TTL)
  2. Download ZIP to cache if not valid
  3. Recursively extract nested ZIPs to target directory
  4. Validate ≥1 JSON file exists
  5. Return extraction directory
- **Input**:
  - `url` — CVE List v5 download URL
  - `cache_dir` — Directory for caching downloads
  - `target_dir` — Directory for extracted CVE files
- **Output**: `Path` to directory containing extracted CVE JSON files
- **Errors**: Raises `OSError` (file I/O), `urllib.error.URLError` (network), `zipfile.BadZipFile` (corrupt archive), `RuntimeError` (no JSON found)

---

### Filter (`core.filter`)

**`load_cves_from_directory(cve_dir: Path) -> list[CVERecord]`**

- **Responsibility**: Load all CVE JSON files from directory into CVERecord objects (bulk loading only)
- **Behavior**:
  1. Scan all JSON files in `cve_dir`
  2. Load each file via `CVERecord.load_from_file()`
  3. Collect valid records; skip malformed files with logging
  4. Return list of loaded CVERecord objects
- **Input**: `cve_dir` — Directory containing CVE JSON files
- **Output**: `list[CVERecord]` of loaded CVEs; `[]` if none valid
- **Errors**: Raises `OSError` on directory access; logs and skips per-file errors
- **Note**: This is the ONLY bulk function. All other functions operate per-object.

---

### GitHub Analysis (`core.github`)

**`extract_github_patch(cve: CVERecord) -> GitHubPatch | None`**

- **Responsibility**: Extract single GitHub patch object from CVE record
- **Behavior**:
  1. Search CVE references for GitHub URLs
  2. Prioritize URL types: commit > file > PR > compare (prefer direct commit info)
  3. Parse first valid URL to identify type
  4. Create appropriate patch subclass instance (`CommitPatch`, `FilePatch`, `PRPatch`, `ComparePatch`)
  5. Return patch object or `None` if no valid URLs found
- **Input**: `cve` — CVERecord object with references
- **Output**: Concrete `GitHubPatch` subclass instance; `None` if no valid URLs
- **Errors**: None (returns `None` for invalid/missing URLs, logs errors)

**`verify_patch_code_type(patch: GitHubPatch, language: str) -> bool`**

- **Responsibility**: Verify single patch's code type matches expected language
- **Behavior**:
  1. Extract patch commit SHA via `patch.get_patch_commit_sha()`
  2. Download and analyze modified files from GitHub for that commit
  3. Check file extensions/content against `language`
  4. Return `True` if match found, `False` otherwise
- **Input**:
  - `patch` — Concrete `GitHubPatch` subclass instance (e.g., `CommitPatch`, `FilePatch`)
  - `language` — Expected language (e.g., `"php"`, `"python"`)
- **Output**: `bool` — `True` if code type matches, `False` otherwise
- **Errors**: Raises `urllib.error.URLError` (network), returns `False` on GitHub API errors, file not found, detection uncertain

**`resolve_pre_patch_commit(patch: GitHubPatch) -> str | None`**

- **Responsibility**: Identify pre-patch commit SHA from a GitHub patch
- **Behavior**:
  1. Extract patch commit SHA from patch object
  2. Query GitHub API for commit details and parent
  3. If direct parent unavailable: fallback to nearest ancestor or branch head
  4. Return identified pre-patch commit SHA
- **Input**: `patch` — Concrete `GitHubPatch` subclass instance
- **Output**: Pre-patch commit SHA; `None` if identification failed
- **Errors**: Raises `urllib.error.URLError` (network), returns `None` on API errors

---

### Extraction (`core.extractor`)

**`extract_cves_to_csv(cves: list[CVERecord], output_file: Path) -> None`**

- **Responsibility**: Extract CVE information from loaded objects and write to CSV (bulk output operation)
- **Behavior**:
  1. Extract `cve_id`, `cve_type` (first CWE), `description` (truncate 200 chars) per object
  2. Sort all rows by CVE ID
  3. Write to CSV with columns: `cve_id`, `cve_type`, `description`
- **Input**:
  - `cves` — List of CVERecord objects (from prior filtering)
  - `output_file` — Output CSV path
- **Output**: `None` (writes to file as side effect)
- **Errors**: Raises `OSError` on write failure, `ValueError` on empty input
- **Note**: This is one of two bulk functions (other is `load_cves_from_directory`)

---

### Analyzer (`core.analyzer`)

**`analyze_distribution(csv_file: Path, min_count: int = 0) -> dict`**

- **Responsibility**: Analyze CVE distribution by type from CSV
- **Behavior**:
  1. Read CSV; validate `cve_id`, `cve_type` columns
  2. Group by type; extract year from CVE ID
  3. Calculate: count, percentage, year span per type
  4. Filter by `min_count` threshold
  5. Identify most common type
  6. Return analysis results as dictionary
- **Input**:
  - `csv_file` — CSV file with CVE data
  - `min_count` — Minimum count for inclusion (default `0`)
- **Output**: `dict` with keys:
  - `total_cves` (int) — Total CVEs analyzed
  - `stats` (dict[str, dict]) — Per-type stats: `{"CWE-79": {"count": 150, "percentage": 10.5, "year_span": "2018 - 2024"}, ...}`
  - `year_range` (str) — Overall year range
  - `most_common` (tuple[str, int]) — Most common type and count
- **Errors**: Raises `OSError` (file not found), `csv.Error`, `KeyError` (missing columns), `ValueError` (invalid CVE ID format)

---

## Exception / Error Types

No custom exception types. Standard Python exceptions raised directly.

| Exception | Raised In | Condition |
|-----------|-----------|-----------|
| `OSError` | All modules | File/directory I/O failures, missing files |
| `json.JSONDecodeError` | CVERecord, filter | Malformed CVE JSON |
| `KeyError` | CVERecord, analyzer | Missing required field/column |
| `ValueError` | CVERecord, analyzer | Invalid data format (e.g., CVE ID parsing) |
| `zipfile.BadZipFile` | downloader | Corrupted ZIP archive |
| `urllib.error.URLError` | downloader, github | Network request failures |
| `RuntimeError` | downloader | No JSON files found after extraction |
| `csv.Error` | analyzer | CSV parsing error |

---

## Validation Rules

### CVERecord

- `cve_id` must be non-empty string
- `cwe_types` list may be empty (unknown CWE)
- `description` may be empty but must be string
- `affected_products` and `references` may be empty lists
- All fields are immutable after construction

### Downloader

- Cache valid if: file exists AND size > 0 AND timestamp < 7 days
- Extraction success: ≥1 JSON file found via `**/*.json` glob
- Nested ZIP extraction handled recursively at all depths

### Filter

- CWE match: CVE has intersection with provided set via `matches_cwe_type()`
- Keyword match: case-insensitive substring in description or affected products via `matches_keywords()`
- Input list may be empty; returns empty result

### GitHub Patches

- `CommitPatch`: must have valid `repository` and `commit_sha`
- `FilePatch`: must have valid `repository`, `commit_sha`, and `file_path`
- `PRPatch`: must have valid `repository` and `pr_number`; `merge_commit_sha` resolved via API (may be `None`)
- `ComparePatch`: must have valid `repository`, `base_sha`, and `head_sha`
- All patches immutable after construction
- All patches have `cve_id` for tracing

### Code Type Verification (Per-Patch)

- Per-patch operation: caller iterates patches and tests each via `verify_patch_code_type()`
- Requires successful file download and extension/content analysis
- Returns `False` on network errors, API failures, file not found, or type mismatch (does not raise)

### Extractor

- Input CVE list must be non-empty (raises `ValueError`)
- Extracts `cve_id`, first CWE type from `cwe_types` list, description (up to 200 chars)
- Sorts all rows by CVE ID ascending before writing
- Raises `OSError` on write failure

### Analyzer

- CSV file must exist (raises `OSError` if not)
- CSV must have `cve_id` and `cve_type` columns (raises `KeyError` if missing)
- CVE year extracted from second segment (CVE-YYYY-...) (raises `ValueError` if invalid)
- Type included in results only if count ≥ `min_count`
- Returns `dict` with analysis results (no wrapper classes)

---

## Calling Patterns

**Load-Once, Iterate-Per-Object, Single-Patch Pattern**:

```python
# Step 1: Load CVEs once (ONLY bulk operation)
cves = load_cves_from_directory(cve_dir)

# Step 2: Caller iterates, tests per CVE object
cwe_filtered = [
    cve for cve in cves 
    if cve.matches_cwe_type({"CWE-79"})
]

# Step 3: Further filter by keywords
kw_filtered = [
    cve for cve in cwe_filtered
    if cve.matches_keywords({"xss", "injection"})
]

# Step 4: GitHub patch extraction
patches_by_cve = {
    cve.cve_id: patch for cve in kw_filtered
    if (patch := extract_github_patch(cve)) is not None
}


# Step 5: Code type verification
verified_by_code = {
    cve_id: patch for cve_id, patch in patches_by_cve.items()
    if verify_patch_code_type(patch, "php")
}

# Step 6: Pre-patch commit identification
verified_cves = [
    cve for cve in kw_filtered
    if cve.cve_id in verified_by_code and resolve_pre_patch_commit(verified_by_code[cve.cve_id]) is not None
]

# Step 7: Bulk extraction for output
extract_cves_to_csv(verified_cves, output_csv)  # Writes to CSV, no return value

# Step 8: Bulk analysis
analysis = analyze_distribution(output_csv, min_count=5)
# Returns: {
#   "total_cves": 500,
#   "stats": {
#     "CWE-79": {"count": 200, "percentage": 40.0, "year_span": "2018 - 2024"},
#     "CWE-89": {"count": 150, "percentage": 30.0, "year_span": "2019 - 2024"},
#     ...
#   },
#   "year_range": "2018 - 2024",
#   "most_common": ("CWE-79", 200)
# }
```

**Design Rules**: 
- Only `load_cves_from_directory()` and `extract_cves_to_csv()` handle bulk lists.
- All other functions operate on single objects: `CVERecord`, `GitHubPatch` (or subclass).
- One CVE has at most one patch (extracted from multiple URL formats representing same fix).
- Each patch type handled individually via concrete subclass instances.
- **No wrapper classes**: Return primitive types (dict, tuple, str, bool) not custom classes.
  - `extract_cves_to_csv()` returns `None` (side effect: file write)
  - `analyze_distribution()` returns `dict` with results

