# Data Directory

This directory contains all datasets and data artifacts for the **CVE Severity Classification** pipeline.  
See the main [README.md](../README.md) for the complete pipeline overview.

---

## Data Pipeline Flow

```
OSIDB → Enhanced JSONs → Multi-Strategy Scraper → Individual CVE dirs with robust commits
                ↓                       ↓
  train/test_kernel_cves.json
                                       ↓
                              cve_feature_extraction.py (48 Features)
                                       ↓
                              cve_dataset.csv
                                       ↓
                          split_datasets_for_train_test.py (Clean Split)
                                       ↓
            cve_training_dataset.csv + cve_testing_dataset.csv
                                       ↓
                               cve_smote_balancer.py
                                       ↓
                    balanced-training-dataset-through-smote.csv
                                       ↓
                   Aggressive XGBoost Training → models/
```

---

## Directory Structure

```
data/
├── ENHANCED LABELED DATA (from OSIDB with patch IDs)
│   ├── train_kernel_cves.json         # Training CVEs with manually researched patch IDs
│   └── test_kernel_cves.json          # Testing CVEs with patch IDs and clean separation
│
├── SCRAPED CVE DATA (created by cve_data_scraper.py)
│   ├── CVE-2022-49200                              # Individual CVE commit data
│   ├── CVE-2024-58093                              # Individual CVE commit data
│   ├── CVE-2025-21936                              # Individual CVE commit data
│   ├── CVE-2025-37853                              # Individual CVE commit data
│   └── CVE-2025-38215                              # [Sample: Only 5 CVEs shown]
│
├── ENHANCED FEATURE DATASETS (created by 48-feature extraction)
│   ├── cve_dataset.csv                # Complete sophisticated feature dataset
│   ├── cve_training_dataset.csv       # Clean training features (no data leakage)
│   ├── cve_testing_dataset.csv        # Clean testing features (proper separation)
│   └── balanced-training-dataset-through-smote.csv # SMOTE-balanced training data
│
├── GIT REPOSITORIES (auto-cloned by cve_data_scraper.py) [Not included here]
│   ├── linux_kernel_repo                           # Linux kernel source (~3GB)
│   └── linux_security_vulns                        # Security vulnerabilities repo
│
├── PREDICTION WORKSPACE (created by cve_predictor.py)
│   └── predict_data
│       └── CVE-2022-27666                          # Example prediction data
```
## Complete Data Story

### 1) Starting Point: Enhanced CVE Data with Patch IDs
We begin with enhanced CVE datasets that include manually researched patch IDs:

- [train_kernel_cves.json](train_kernel_cves.json) — Training CVEs with severity labels and 27+ manually added patch IDs
- [test_kernel_cves.json](test_kernel_cves.json) — Testing CVEs with clean separation for unbiased evaluation

These contain CVE IDs with severity labels (**IMPORTANT**, **MODERATE**, **LOW**) plus patch IDs for direct commit extraction.

---

### 2) Multi-Strategy CVE Data Scraping
[`cve_data_scraper.py`](../cve_data_scraper.py) uses a sophisticated 2-strategy approach:

- **Strategy 1**: Direct patch ID extraction (when available)
- **Strategy 2**: JSON references with fix-commit filtering
- **Enhanced Git Fetching**: Multi-branch commit retrieval with robust error handling
- **Result**: Higher success rate with quality commit data extraction

---

### 3) 48-Feature Extraction (Advanced ML Datasets)
[`cve_feature_extraction.py`](../cve_feature_extraction.py) implements proven Perl script logic with:

- **48 Sophisticated Features**: File path analysis, content analysis, author analysis, complex rules
- **Advanced Logic**: Kernel panic rules, feature interactions, TIPC special cases, CVE year thresholds
- **Business Rules**: Author prominence detection, content-based adjustments, security pattern recognition
- **Output**: [cve_dataset.csv](cve_dataset.csv) — battle-tested feature dataset
---
### 4) Clean Dataset Splitting (No Data Leakage)
[`split_datasets_for_train_test.py`](../split_datasets_for_train_test.py) ensures clean separation:

- **Clean Split**: Uses JSON files to maintain strict train/test separation
- **No Data Leakage**: Prevents training CVEs from appearing in test set
- **Outputs**: [cve_training_dataset.csv](cve_training_dataset.csv) and [cve_testing_dataset.csv](cve_testing_dataset.csv)

---

### 5) SMOTE Data Balancing
[`cve_smote_balancer.py`](../cve_smote_balancer.py) uses **SMOTE** to balance classes:

- **Input**: Clean training set with no data leakage
- **Output**: [balanced-training-dataset-through-smote.csv](balanced-training-dataset-through-smote.csv)
- **Result**: Proper representation of IMPORTANT/MODERATE/LOW classes

---

### 6) Aggressive Model Training & Clean Testing
[`xgboost_train.py`](../xgboost_train.py) trains an aggressive XGBoost model with:
- **Custom Class Weights**: 2.5x weight for MODERATE class
- **Enhanced Parameters**: 500 estimators, max_depth=8, regularization
- **Output**: [`models/`](../models/) directory

[`test_cve_model.py`](../test_cve_model.py) evaluates on [cve_testing_dataset.csv](cve_testing_dataset.csv) with no data leakage.
- **Output**: [`test-results/`](../test-results/) directory

---

### 7) Enhanced CVE Prediction
[`cve_predictor.py`](../cve_predictor.py) predicts severity with:
- **Enhanced Error Handling**: Helpful guidance for missing patch IDs
- **Robust Processing**: Handles CVEs with and without patch IDs
- **Model Loading**: Uses aggressive model from [`models/`](../models/)
- **Results**: Saves predictions to [predict_data](predict_data/) with detailed analysis

---

## Individual CVE Structure

```
CVE-YYYY-NNNNN/
├── metadata.json
├── CVE-YYYY-NNNNN.json
├── CVE-YYYY-NNNNN.mbox
└── commits/
    ├── {hash}.json
    ├── {hash}.patch
    └── {hash}_missing.json
```

---

## Features & Labels

- **Features:** 48 binary features from patch analysis.  
- **Labels:** IMPORTANT (0), MODERATE (1), LOW (2).

---

## Key Points

- **Sample vs Full Data:** Only 5 CVEs shown; full datasets are pre-computed.  
- **Processing Time:** Full extraction takes hours.  
- **Class Imbalance:** SMOTE balancing is essential.

---

_For complete usage instructions, see the main [../README.md](../README.md)._
