# CVE Extractor — Concepts & Terminology

Multi-stage pipeline that acquires CVE data, filters by vulnerability classification, keywords, and code language, then downloads vulnerable repository code at pre-patch commits for security research.

---

## 1. Context

**Problem Statement**

CVE databases contain 200,000+ vulnerability records spanning all technology domains. Researchers need vulnerable code in its pre-patch state for pattern analysis, ML training, and security research. No existing tool automates the pipeline from raw CVE data to downloadable vulnerable repository archives filtered by language and vulnerability type.

**System Role**

CVE Extractor transforms raw CVE datasets into curated repository archives containing vulnerable code at the exact commit before patches were applied.

**Data Flow**

- **Inputs:** CVE List v5 URL, filtering criteria (CWE types, keywords, target language)
- **Outputs:** Repository code archives in vulnerable state (pre-patch commits)
- **Connections:** [CVE Source] → [Acquisition] → [Filtering Pipeline] → [Download] → [Vulnerable Archives]

```
[CVE List v5 URL]
    → Acquisition (download & extract)
    → Filtering Pipeline (CWE type → keywords → GitHub URLs → code type)
    → Download (pre-patch commit identification → archive download)
    → [Vulnerable Repository Archives]
```

**Scope Boundaries**

- **Owned:** CVE acquisition, multi-stage filtering, pre-patch commit identification, archive download, caching
- **Not Owned:** CVE JSON schema definition, GitHub API protocol, vulnerability analysis of downloaded code

---

## 2. Concepts

**Conceptual Diagram**

```
                 ┌─────────────────┐
                 │  CVE List v5    │
                 │  (Remote URL)   │
                 └────────┬────────┘
                          ▼
                 ┌─────────────────┐
                 │   Acquisition   │
                 │  Download &     │
                 │  Extract        │
                 └────────┬────────┘
                          ▼
              ┌───────────────────────┐
              │   Filtering Pipeline  │
              │  ┌─────────────────┐  │
              │  │ CWE Type Filter │  │
              │  └────────┬────────┘  │
              │           ▼           │
              │  ┌─────────────────┐  │
              │  │ Keyword Filter  │  │
              │  └────────┬────────┘  │
              │           ▼           │
              │  ┌─────────────────┐  │
              │  │ GitHub URL      │  │
              │  │ Extraction      │  │
              │  └────────┬────────┘  │
              │           ▼           │
              │  ┌─────────────────┐  │
              │  │ Code Type       │  │
              │  │ Verification    │  │
              │  └────────┬────────┘  │
              └───────────┼───────────┘
                          ▼
              ┌───────────────────────┐
              │      Download         │
              │  ┌─────────────────┐  │
              │  │ Pre-Patch       │  │
              │  │ Commit ID       │  │
              │  └────────┬────────┘  │
              │           ▼           │
              │  ┌─────────────────┐  │
              │  │ Archive         │  │
              │  │ Download        │  │
              │  └────────┬────────┘  │
              └───────────┼───────────┘
                          ▼
                 ┌─────────────────┐
                 │  Vulnerable     │
                 │  Repository     │
                 │  Archives       │
                 └─────────────────┘
```

**Core Concepts**

**CVE Record (Object)** — In-memory representation of a parsed CVE after loading from JSON file. Contains all CVE data as attributes: ID, CWE types, description, affected products, references, publication date. Loaded once from file, then reused for multiple filter operations without file I/O. Immutable after construction.

**Downloadable Artifact**. A remote CVE dataset from the official CVE List v5 release channel packaged as compressed archives. Represents the authoritative vulnerability source containing all CVE records. Downloaded once, cached locally, then extracted to directory of JSON files.

**Download Cache**. Local filesystem copy of previously downloaded artifacts with timestamp metadata. Implements a 7-day TTL validity window. Prevents redundant network requests and enables resumability across runs.

**Nested Archive Structure**. Archive files may contain nested ZIP files requiring recursive extraction to access actual content. Applies to both CVE List v5 archives (nested ZIPs before reaching JSON records) and GitHub-provided repository archives (nested compression layers). Handled transparently by extraction logic at arbitrary nesting depth.

**CWE Type Filter**. First filtering stage selecting CVEs by CWE classification (e.g., CWE-79 for XSS, CWE-89 for SQL injection). Operates on loaded CVERecord objects. Reduces 200,000+ records to a domain-relevant subset. Retains all CVEs with CWE type intersection.

**Keyword Filter**. Second filtering stage narrowing CVEs by searching descriptions and references for specific terms (e.g., "authentication", "injection", "remote code execution"). Matches case-insensitively in parsed CVERecord attributes. Operates on output of CWE type filtering.

**GitHub Patch URL**. A URL in CVE references pointing to a vulnerability fix on GitHub. Four valid formats exist: full commit URL, file-specific blob URL, pull request URL, and compare URL. Each format contains enough information to identify the patch commit SHA and target repository. Extracted per-CVE from CVERecord references and categorized by URL type.

**Code Type Verification**. Final filtering stage downloading and analyzing modified files from GitHub commits to verify code matches the expected language (e.g., PHP, Python). Eliminates false positives where a language is mentioned in CVERecord description but the actual patch modifies different code. Returns boolean match result per CVE.

**Pre-Patch Commit**. The commit immediately before a vulnerability fix was applied, representing the vulnerable code state. Identified by finding the parent of the patch commit via GitHub API. When the direct parent is unavailable (merge squash, rebase, force push), the system applies fallback strategies: nearest ancestor commit in the same branch, then main/master branch head as last resort.

**Vulnerable Repository Archive**. A repository code archive downloaded at the pre-patch commit state. Contains actual vulnerable code before the fix was applied. The primary deliverable of the entire system. Stored as extracted directory per CVE in the output location.

**Parameter Injection**. All configuration (URLs, paths, credentials) passed as function parameters. No global state or singleton Config. Each function receives only the parameters it needs. Enables testability, composability, and explicit dependency tracking.

---

## 3. Contracts & Flow

**Data Contracts**

- **Acquisition → Loading:** Directory of extracted CVE JSON files
- **Loading → CWE Filter:** `list[CVERecord]` — loaded CVE objects in memory
- **CWE Filter → Keyword Filter:** `list[CVERecord]` — CVEs matching CWE types
- **Keyword Filter → URL Extraction:** `list[CVERecord]` — CVEs matching keywords
- **URL Extraction → Code Verification:** `dict[cve_id: str, urls: dict]` — per-CVE patch URL mapping
- **Code Verification → Pre-Patch ID:** Set of verified CVE IDs with confirmed language match
- **Pre-Patch ID → Archive Download:** `dict[cve_id: str, sha: str]` — pre-patch commit SHA per CVE
- **Archive Download → Output:** `dict[cve_id: str, path: Path]` — downloaded archive directory per CVE

**Internal Processing Flow**

1. **Acquire** — Download CVE List v5 dataset from remote URL, recursively extract nested archives, validate JSON files exist in extraction directory
2. **Load Objects** — Read all CVE JSON files once, parse into CVERecord objects in memory, return list of loaded objects
3. **Filter by CWE Type** — Iterate loaded CVERecord objects, test CWE type match against requested set, return filtered list
4. **Filter by Keywords** — Iterate filtered CVERecord objects, test keyword match in description/products, return narrowed list
5. **Extract GitHub URLs** — Iterate filtered CVERecord objects, extract and categorize GitHub URLs from references, return per-CVE URL mapping
6. **Verify Code Type** — Iterate over CVE-to-URLs mapping, download GitHub patch content, check file type against expected language, collect verified CVEs
7. **Identify Pre-Patch Commits** — Iterate verified CVEs, extract patch commit SHA from URLs, query GitHub API for pre-patch parent, apply fallback strategies
8. **Download Archives** — Iterate pre-patch commit mapping, download repository archives, extract to output directory, verify integrity

> Technical specifications (function signatures, types, error handling) are in [design.md](design.md).

---

## 4. Scenarios

**Typical: PHP XSS Vulnerability Research**

Researcher targets PHP repositories with XSS vulnerabilities (CWE-79). Pipeline acquires 200,000+ CVEs, filters to ~2,000 CWE-79 records, narrows by keywords to ~500, extracts GitHub URLs for ~300, verifies PHP code for ~200, identifies pre-patch commits, and downloads 200 repository archives. Total: ~35 minutes on first run, ~2 minutes with cache.

**Boundary: Commit History Unavailable**

GitHub commit history is unavailable due to merge squash or repository cleanup. System detects missing parent commit, queries GitHub API for commit ancestors, falls back to nearest ancestor in the same branch. If complete history is unavailable, uses main/master branch head as final fallback. Processing continues for all remaining CVEs without interruption.

**Boundary: GitHub API Rate Limit**

Rate limit exceeded during code type verification or commit identification. System logs progress, caches all completed results in intermediate directory. On next run, pipeline resumes from cached state and processes only remaining CVEs.

**Interaction: Incremental Processing**

User reruns pipeline after partial completion or interruption. Acquisition returns cached directory immediately (within 7-day TTL). Each filtering stage reuses cached results. Download stage skips already-downloaded archives, processes only remaining CVEs. Efficient recovery with minimal redundant work.

---

## System Invariants

| Invariant | Property |
|-----------|----------|
| CVE Data Currency | Cached data ≤ 7 days old considered fresh; older triggers re-download |
| Patch Commit Robustness | Pre-patch commit identified even if direct parent unavailable via ancestor search |
| Code Type Accuracy | Repository code type verified via GitHub commit content; mismatches filtered out |
| GitHub API Resilience | Rate limit and network errors handled gracefully; partial results cached |
| Archive Integrity | Downloaded archives validated for extraction success before marking complete |
| Filtering Determinism | Same inputs with valid cache produce identical results across runs |
| Error Isolation | Failures in single CVE do not halt pipeline; recorded as absent in results |
