# CVE Extractor — Implementation Notes

## APIs

**GitHubKit** `github.rest.repos.get("owner", "repo")` — Fetch repository; returns Response[FullRepository]. Type-safe, fully typed.

**GitHubKit** `github.rest.repos.get_commit("owner", "repo", ref=sha)` — Fetch commit by SHA; returns Response[Commit] with `.files` list (File objects). Access `.filename`, `.status` properties.

**GitHubKit** `github.rest.repos.get_content("owner", "repo", path, headers={"Accept": "application/vnd.github.raw+json"})` — Fetch raw file content at specific commit; returns Response with `.text` property.

**GitHubKit** `github.rest.repos.download_tarball_archive("owner", "repo", ref=sha, stream=True)` — Stream repository archive at commit SHA; use `.iter_bytes(8192)` for memory efficiency.

**GitHubKit** `github.arequest(method, endpoint, **kwargs)` — Direct async API requests for custom endpoints not covered by generated methods.

**cwe-tree** `query.get_metadata(cwe_id)` — Fetch CWE metadata (dict); includes `id`, `name`, `abstract`, `layer`, `parents`, `children`. Returns None if invalid CWE ID.

**cwe-tree** `query.get_node(cwe_id)` — Retrieve single CWE node; accepts "CWE-732" or "732" format. Returns CweNode or None.

**cwe-tree** `query.get_children(cwe_id)` — Return set of direct child CWE IDs; empty set if none or invalid ID.

**cwe-tree** `query.get_parents(cwe_id)` — Return set of direct parent CWE IDs; empty set if none or invalid ID.

**cwe-tree** `query.get_layer(cwe_id)` — Return dict with layer mapping (depth in tree); empty dict if invalid ID.

**cwe-tree** `query.get_roots()` — Return list of root CweNode objects (nodes with no parents).

---

## Libraries

**GitHubKit** — Benchmark 79.9, High reputation; modern typed SDK for GitHub API v3+, fully typed responses, sync/async support, streaming downloads.

**cwe-tree** — Benchmark 77.4, High reputation; tree-based CWE hierarchy queries (metadata, parent/child traversal, node retrieval).

---

## Developer Instructions

**CVE JSON Parsing**: Load CVE records via `CVERecord.load_from_file(path)` — single-pass JSON load, immutable object; use `.matches_cwe_type()` and `.matches_keywords()` methods.

**GitHub Setup**: Create `GitHub(token="GITHUB_TOKEN")` instance once; reuse for all API calls. Environment variable GITHUB_TOKEN used automatically if set.

**GitHub Patch Extraction**: `extract_github_patch(cve)` returns single GitHubPatch subclass or None; prioritizes commit > file > PR > compare URL formats.

**Commit Content Analysis**: For code type verification, use `verify_patch_code_type(patch, language)` — downloads modified files via GitHubKit, checks extensions/content.

**List Comprehensions**: Chain filters declaratively (e.g., `[cve for cve in cves if cve.matches_cwe_type(...)]`); use walrus operator for complex conditions: `if (val := func(x)) is not None`.

**CWE Type Extraction**: Extract from CVE `containers.cna.problemTypes[0].descriptions[0]` for type/cweId; fallback to "Unknown" if missing. Use `cwe_tree.query.get_metadata(cwe_id)` to validate/enrich CWE.

**CWE Descendant Check**: cwe-tree has no built-in descent/is_descendant function. Implement DFS using `query.get_children()` or `query.get_parents()`: traverse parent/child relationships recursively to check if `cwe_x` is descendant of `cwe_y`. Cache results in module-level dict (CWE tree is static data).

**Repository Archive Download**: Stream to disk with `github.rest.repos.download_tarball_archive(owner, repo, ref=sha, stream=True)` — use `.iter_bytes(8192)` to avoid memory overload.

**Patch Commit SHA Extraction**: 
- Commit URL: direct SHA from path segment
- File URL: SHA from blob segment  
- PR URL: resolve via `github.rest.pulls.get(owner, repo, pr_number).merge_commit_sha`
- Compare URL: right-side SHA (`sha2`)

**Pre-Patch Commit Identification**: Query `github.rest.repos.get_commit(owner, repo, sha).parents[0].sha` for parent; if unavailable (merge squash), implement ancestor search or fallback to branch head.

**Rate Limit Handling**: GitHub allows 5000 req/hr (authenticated). For 1000+ CVEs, implement caching in `inter_dir`; store commit metadata and file lists to avoid re-queries.

---

## Design-Specific

### CVERecord (core.models)
- Immutable after construction; all fields set via `__init__`
- `load_from_file(path)` performs single JSON read, caches object
- Methods `matches_cwe_type()`, `matches_keywords()` use in-memory data, no re-reads

### GitHubPatch Hierarchy (core.models)
- Abstract base with `get_patch_commit_sha()`, `get_patch_files()` methods
- CommitPatch, FilePatch, PRPatch, ComparePatch concrete subclasses
- Immutable; merge_commit_sha (PRPatch) resolved once on creation

### GitHub Analysis (core.github)
- `extract_github_patch(cve)` prioritizes URL types; returns first valid or None
- `verify_patch_code_type(patch, language)` — returns False on errors, doesn't raise
- `resolve_pre_patch_commit(patch)` — raises URLError on network, returns None on API errors

### Filtering Pipeline (calling pattern)
- Load CVEs once: `load_cves_from_directory(cve_dir)` → `list[CVERecord]`
- Filter via comprehensions: `[cve for cve in cves if cve.matches_cwe_type(...)]`
- Complex logic via helper: `if (patch := extract_github_patch(cve)) is not None`
- Output operations (extract_cves_to_csv, analyze_distribution) are bulk; others are per-object

### Error Handling
- No custom exceptions; raise standard Python exceptions (OSError, ValueError, KeyError, etc.)
- Callers handle via try/except; functions return None or False for recoverable errors (not sentinel lists/dicts)

---

## Integration Notes

**CVE List v5 Format**: JSON files under extraction directory; each file = one CVE record; file stem = CVE ID (e.g., `CVE-2024-1234.json`).

**GitHubKit vs PyGithub**: Use GitHubKit only — modern, typed, better Benchmark (79.9), minimal overhead. No PyGithub needed.

**Multi-Stage Filtering**: Each stage preserves immutable CVERecord objects; filter by accumulating CVE ID sets or lists; no in-place modifications.
