{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Common Criteria example\n",
    "\n",
    "This notebook illustrates basic functionality with the `CCDataset` class that holds Common Criteria dataset and of its sample `CommonCriteriaCert`.\n",
    "\n",
    "Note that there exists a front end to this functionality at [seccerts.org/cc](https://seccerts.org/cc/). Before reinventing the wheel, it's good idea to check our web. Maybe you don't even need to run the code, but just use our web instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sec_certs.dataset.common_criteria import CCDataset\n",
    "from sec_certs.dataset.common_criteria import CommonCriteriaCert\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get fresh dataset snapshot from mirror\n",
    "\n",
    "There's no need to do full processing of the dataset by yourself, unless you modified `sec-certs` code. You can simply fetch the processed version from the web. \n",
    "\n",
    "Note, however, that you won't be able to access the `pdf` and `txt` files of the certificates. You can only get the data that we extracted from it. \n",
    "\n",
    "Running the whole pipeline can get you the `pdf` and `txt` data. You can see how to do that in the last cell of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dset = CCDataset.from_web_latest()\n",
    "print(len(dset)) # Print number of certificates in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do some basic dataset serialization\n",
    "\n",
    "The dataset can be saved/loaded into/from `json`. Also, the dataset can be converted into a [pandas](https://pandas.pydata.org/) DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dump dataset into json and load it back\n",
    "dset.to_json(\"./cc_dset.json\")\n",
    "new_dset: CCDataset = CCDataset.from_json(\"./cc_dset.json\")\n",
    "assert dset == new_dset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn dataset into Pandas DataFrame\n",
    "df = dset.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple dataset manipulation\n",
    "\n",
    "The certificates of the dataset are stored in a dictionary that maps certificate's primary key (we call it `dgst`) to the `CommonCriteriaCert` object. The primary key of the certificate is simply a hash of the attributes that make the certificate unique.\n",
    "\n",
    "You can iterate over the dataset which is handy when selecting some subset of certificates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Iterate over certificates in dataset\n",
    "for cert in dset:\n",
    "    pass\n",
    "\n",
    "# Get certificates produced by Infineon manufacturer\n",
    "infineon_certs = [x for x in dset if \"Infineon\" in x.manufacturer]\n",
    "df_infineon = df.loc[df.manufacturer.str.contains(\"Infineon\", case=False)]\n",
    "\n",
    "# Get certificates with some CVE\n",
    "vulnerable_certs = [x for x in dset if x.heuristics.related_cves]\n",
    "df_vulnerable = df.loc[~df.related_cves.isna()]\n",
    "\n",
    "# Show CVE ids of some vulnerable certificate\n",
    "print(f\"{vulnerable_certs[0].heuristics.related_cves=}\")\n",
    "\n",
    "# Get certificates from 2015 and newer\n",
    "df_2015_and_newer = df.loc[df.year_from > 2014]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot distribution of years of certification\n",
    "df.year_from.value_counts().sort_index().plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dissect single certificate\n",
    "\n",
    "The `CommonCriteriaCert` is basically a data structure that holds all the data we keep about a certificate. Other classes (`CCDataset` or `model` package members) are used to transform and process the certificates. You can see all its attributes at [API docs](https://seccerts.org/docs/api/sample.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a certificate and print some attributes\n",
    "cert: CommonCriteriaCert = dset[\"bad93fb821395db2\"]\n",
    "print(f\"{cert.name=}\")\n",
    "print(f\"{cert.heuristics.cpe_matches=}\")\n",
    "print(f\"{cert.heuristics.report_references.directly_referencing=}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select all certificates from a dataset for which we detect at least one vulnerability.\n",
    "vulnerable_certs = [x for x in dset if x.heuristics.related_cves]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serialize single certificate\n",
    "\n",
    "Again, a certificate can be (de)serialized into/from json. It's also possible to construct pandas `Series` from a certificate as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "cert.to_json(\"./cert.json\")\n",
    "new_cert = cert.from_json(\"./cert.json\")\n",
    "assert cert == new_cert\n",
    "\n",
    "# Serialize as Pandas series\n",
    "ser = pd.Series(cert.pandas_tuple, index=cert.pandas_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assign dataset with CPE records and compute vulnerabilities\n",
    "\n",
    "*Note*: The data is already computed on dataset obtained with `from_web_latest()`, this is just for illustration. \n",
    "*Note*: This may likely not run in Binder, as the corresponding `CVEDataset` and `CPEDataset` instances take a lot of memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Automatically match CPEs and CVEs\n",
    "_, cpe_dset, _ = dset.compute_cpe_heuristics()\n",
    "dset.compute_related_cves()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create new dataset and fully process it\n",
    "\n",
    "The following piece of code roughly corresponds to `$ cc-certs all` CLI command -- it fully processes the CC pipeline. This will create a folder in current working directory where the outputs will be stored. \n",
    "\n",
    "*Warning*: It's not good idea to run this from notebook. It may take several hours to finnish. We recommend using `from_web_latest()` or turning this into a Python script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dset = CCDataset()\n",
    "dset.get_certs_from_web()\n",
    "dset.process_protection_profiles()\n",
    "dset.download_all_pdfs()\n",
    "dset.convert_all_pdfs()\n",
    "dset.analyze_certificates()"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "6386d1612879d92d026c363e7667e428bc38d86c5a080d58c3d70e7cd43df67d"
  },
  "kernelspec": {
   "display_name": "Python 3.8.1 ('certsvenv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
