# Data Directory

This directory contains all datasets and data artifacts for the **CVE Severity Classification** pipeline.  
See the main [README.md](../README.md) for the complete pipeline overview.

---

## Data Pipeline Flow

```
OSIDB → kernel_cves.json → Split → train/test JSONs → cve_data_scraper.py → Individual CVE dirs
                                                             ↓
                                                   cve_feature_extraction.py
                                                             ↓
                                                        cve_dataset.csv
                                                             ↓
                                                split_datasets_for_train_test.py
                                                             ↓
                                             cve_training_dataset.csv + cve_testing_dataset.csv
                                                             ↓
                                                   cve_smote_balancer.py
                                                             ↓
                                             balanced-training-dataset-through-smote.csv
                                                             ↓
                                                       Model Training
```

---

## Directory Structure

```
data/
├── INITIAL LABELED DATA (from OSIDB)
│   ├── kernel_cves.json                           # Complete CVE list extracted from OSIDB
│   ├── train_kernel_cves.json                     # Training split (subset of kernel_cves.json)
│   └── test_kernel_cves.json                      # Testing split (subset of kernel_cves.json)
│
├── SCRAPED CVE DATA (created by cve_data_scraper.py)
│   ├── CVE-2022-49200                              # Individual CVE commit data
│   ├── CVE-2024-58093                              # Individual CVE commit data
│   ├── CVE-2025-21936                              # Individual CVE commit data
│   ├── CVE-2025-37853                              # Individual CVE commit data
│   └── CVE-2025-38215                              # [Sample: Only 5 CVEs shown]
│
├── FEATURE DATASETS (created by cve_feature_extraction.py)
│   ├── cve_dataset.csv                             # Sample dataset (5 CVEs demo)
│   ├── cve-training-dataset.csv                    # Full training features (pre-computed, created by split_datasets_for_train_test.py)
│   ├── cve_testing_dataset.csv                     # Full testing features (pre-computed, created by split_datasets_for_train_test.py)
│   └── balanced-training-dataset-through-smote.csv # SMOTE-balanced
│
├── GIT REPOSITORIES (auto-cloned by cve_data_scraper.py) [Not included here]
│   ├── linux_kernel_repo                           # Linux kernel source (~3GB)
│   └── linux_security_vulns                        # Security vulnerabilities repo
│
├── PREDICTION WORKSPACE (created by cve_predictor.py)
│   └── predict_data
│       └── CVE-2022-27666                          # Example prediction data
│
└── REPORTS & ANALYSIS (created by cve_data_scraper.py)
    ├── cve_analysis_report.json                    # Scraping statistics
    ├── sample_data.csv                             # Demo files
    └── sample_data.csv.numbers                     # Numbers spreadsheet version
```
## Complete Data Story

### 1) Starting Point: Labeled CVE Data from OSIDB
We begin with [kernel_cves.json](kernel_cves.json) — our complete list of labeled CVEs extracted from OSIDB.  
This contains CVE IDs with their severity labels (**IMPORTANT**, **MODERATE**, **LOW**).

- [train_kernel_cves.json](train_kernel_cves.json) and [test_kernel_cves.json](test_kernel_cves.json) are splits of the complete dataset.

---

### 2) CVE Data Scraping (Sample: 5 CVEs)
[`cve_data_scraper.py`](../cve_data_scraper.py) takes these labeled CVE lists and creates directories with commit messages, patches, and security advisories.

- 5 CVE directories shown are samples; running on all CVEs would take many hours.

---

### 3) Feature Extraction (Creates ML Datasets)
[`cve_feature_extraction.py`](../cve_feature_extraction.py) extracts **48 binary features** and produces:

- [cve_dataset.csv](cve_dataset.csv) — complete feature dataset
---
### 4) Dataset Splitting
[`split_datasets_for_train_test.py`](../split_datasets_for_train_test.py) splits the dataset based on JSON files:

- [cve_training_dataset.csv](cve_training_dataset.csv) — training set
- [cve_testing_dataset.csv](cve_testing_dataset.csv) — testing set

---

### 5) Data Balancing
[`cve_smote_balancer.py`](../cve_smote_balancer.py) uses **SMOTE** to balance classes, producing:

- [balanced-training-dataset-through-smote.csv](balanced-training-dataset-through-smote.csv)

---

### 6) Model Training & Testing
[`xgboost_train.py`](../xgboost_train.py) trains the model.  
[`test_cve_model.py`](../test_cve_model.py) evaluates on [cve_testing_dataset.csv](cve_testing_dataset.csv).

---

### 7) New CVE Prediction
[`cve_predictor.py`](../cve_predictor.py) predicts severity for new CVEs, saving results to [predict_data](predict_data/).

---

## Individual CVE Structure

```
CVE-YYYY-NNNNN/
├── metadata.json
├── CVE-YYYY-NNNNN.json
├── CVE-YYYY-NNNNN.mbox
└── commits/
    ├── {hash}.json
    ├── {hash}.patch
    └── {hash}_missing.json
```

---

## Features & Labels

- **Features:** 48 binary features from patch analysis.  
- **Labels:** IMPORTANT (0), MODERATE (1), LOW (2).

---

## Key Points

- **Sample vs Full Data:** Only 5 CVEs shown; full datasets are pre-computed.  
- **Processing Time:** Full extraction takes hours.  
- **Class Imbalance:** SMOTE balancing is essential.

---

_For complete usage instructions, see the main [../README.md](../README.md)._
