{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering of scikit-learn toy data sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to:\n",
    "    \n",
    "  - [Notebook configuration](scikit_learn_datasets.ipynb#Notebook-configuration)\n",
    "  - [Data set generation](scikit_learn_datasets.ipynb#Data-set-generation)\n",
    "  \n",
    "  - CommonNN clustering using ...\n",
    "    - [... point coordiantes](scikit_learn_datasets.ipynb#CommonNN-clustering-using-data-point-coordiantes-as-input)\n",
    "    - [... pre-computed distances](scikit_learn_datasets.ipynb#CommonNN-clustering-with-pre-computed-distances)\n",
    "    - [... pre-computed neighbourhoods](scikit_learn_datasets.ipynb#CommonNN-clustering-with-pre-computed-neighbourhoods)\n",
    "    - [... pre-computed sorted neighbourhoods](scikit_learn_datasets.ipynb#Common-nearest-neighbours-clustering-with-pre-computed-sorted-neighbourhoods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.888075Z",
     "start_time": "2021-10-01T10:35:14.807916Z"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn import datasets\n",
    "from sklearn.metrics import pairwise_distances\n",
    "from sklearn.neighbors import KDTree\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "import cnnclustering\n",
    "from cnnclustering import cluster, hooks\n",
    "from cnnclustering import _types, _fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.895060Z",
     "start_time": "2021-10-01T10:35:16.890170Z"
    }
   },
   "outputs": [],
   "source": [
    "# Optional helper functionality from the benchmark module\n",
    "#     https://github.com/janjoswig/CommonNNClustering-Benchmark  \n",
    "sys.path.insert(0, \"../benchmark\")\n",
    "import helper_base"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print Python and package version information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.901386Z",
     "start_time": "2021-10-01T10:35:16.897096Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python:  3.8.8 (default, Mar 11 2021, 08:58:19)  [GCC 8.3.0]\n",
      "Packages:\n",
      "    matplotlib: 3.3.4\n",
      "    numpy: 1.20.1\n",
      "    sklearn: 0.24.1\n",
      "    cnnclustering: 0.4.3\n"
     ]
    }
   ],
   "source": [
    "# Version information\n",
    "print(\"Python: \", *sys.version.split(\"\\n\"))\n",
    "\n",
    "print(\"Packages:\")\n",
    "for package in [mpl, np, sklearn, cnnclustering]:\n",
    "    print(f\"    {package.__name__}: {package.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use [Matplotlib](https://matplotlib.org/) to create plots. The `matplotlibrc` file in the root directory of the `CommonNNClustering` repository is used to customize the appearance of the plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.912478Z",
     "start_time": "2021-10-01T10:35:16.903568Z"
    }
   },
   "outputs": [],
   "source": [
    "# Matplotlib configuration\n",
    "mpl.rc_file(\n",
    "    \"../../matplotlibrc\",\n",
    "    use_default_template=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.920775Z",
     "start_time": "2021-10-01T10:35:16.914551Z"
    }
   },
   "outputs": [],
   "source": [
    "# Axis property defaults for the plots\n",
    "ax_props = {\n",
    "    \"xlabel\": None,\n",
    "    \"ylabel\": None,\n",
    "    \"xlim\": (-2.5, 2.5),\n",
    "    \"ylim\": (-2.5, 2.5),\n",
    "    \"xticks\": (),\n",
    "    \"yticks\": (),\n",
    "    \"aspect\": \"equal\"\n",
    "}\n",
    "\n",
    "# Line plot property defaults\n",
    "line_props = {\n",
    "    \"linewidth\": 0,\n",
    "    \"marker\": '.',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-11T13:48:26.641740Z",
     "start_time": "2019-09-11T13:48:26.638371Z"
    }
   },
   "source": [
    "## Data set generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-11T13:48:56.480391Z",
     "start_time": "2019-09-11T13:48:56.477467Z"
    }
   },
   "source": [
    "To see the common-nearest-neighbours (CommonNN) clustering in action, let's have a look at a handful of basic 2D data sets from scikit-learn (like here in the [scikit-learn documentation](https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html)). We will cluster the data sets starting from different input data formats using the provided defaults. For more details see also the [Advanced usage](advanced_usage.ipynb) tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.929116Z",
     "start_time": "2021-10-01T10:35:16.922500Z"
    }
   },
   "outputs": [],
   "source": [
    "# Data set generation parameters\n",
    "np.random.seed(0)\n",
    "n_samples = 2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.955361Z",
     "start_time": "2021-10-01T10:35:16.930831Z"
    }
   },
   "outputs": [],
   "source": [
    "# Data set generation\n",
    "# Fit all datasets to the same value range\n",
    "#    using `data = StandardScaler().fit_transform(data)`\n",
    "\n",
    "# circles\n",
    "noisy_circles, _ = datasets.make_circles(\n",
    "    n_samples=n_samples,\n",
    "    factor=.5,\n",
    "    noise=.05\n",
    "    )\n",
    "noisy_circles = StandardScaler().fit_transform(noisy_circles)\n",
    "\n",
    "# moons\n",
    "noisy_moons, _ = datasets.make_moons(\n",
    "    n_samples=n_samples,\n",
    "    noise=.05\n",
    "    )\n",
    "noisy_moons = StandardScaler().fit_transform(noisy_moons)\n",
    "\n",
    "# blobs\n",
    "blobs, _ = datasets.make_blobs(\n",
    "    n_samples=n_samples,\n",
    "    random_state=8\n",
    "    )\n",
    "blobs = StandardScaler().fit_transform(blobs)\n",
    "\n",
    "# None\n",
    "no_structure = np.random.rand(\n",
    "    n_samples, 2\n",
    "    )\n",
    "no_structure = StandardScaler().fit_transform(no_structure)\n",
    "\n",
    "# aniso\n",
    "random_state = 170\n",
    "X, y = datasets.make_blobs(\n",
    "    n_samples=n_samples,\n",
    "    random_state=random_state\n",
    "    )\n",
    "\n",
    "transformation = [[0.6, -0.6], [-0.4, 0.8]]\n",
    "aniso = np.dot(X, transformation)\n",
    "aniso = StandardScaler().fit_transform(aniso)\n",
    "\n",
    "# varied\n",
    "varied, _ = datasets.make_blobs(\n",
    "    n_samples=n_samples,\n",
    "    cluster_std=[1.0, 2.5, 0.5],\n",
    "    random_state=random_state\n",
    "    )\n",
    "varied = StandardScaler().fit_transform(varied)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:16.962677Z",
     "start_time": "2021-10-01T10:35:16.958045Z"
    }
   },
   "outputs": [],
   "source": [
    "# Define cluster parameters\n",
    "dsets = [ # \"name\", set, **parameters\n",
    "    ('circles', noisy_circles, {\n",
    "        'radius_cutoff': 0.5,\n",
    "        'cnn_cutoff': 20,\n",
    "        'member_cutoff': 100,\n",
    "        'max_clusters': None\n",
    "        }),\n",
    "    ('moons', noisy_moons, {\n",
    "        'radius_cutoff': 0.5,\n",
    "        'cnn_cutoff': 20,\n",
    "        'member_cutoff': 2,\n",
    "        'max_clusters': None\n",
    "        }),\n",
    "    ('varied', varied, {\n",
    "        'radius_cutoff': 0.28,\n",
    "        'cnn_cutoff': 20,\n",
    "        'member_cutoff': 20,\n",
    "        'max_clusters': None\n",
    "        }),\n",
    "    ('aniso', aniso, {\n",
    "        'radius_cutoff': 0.29,\n",
    "        'cnn_cutoff': 30,\n",
    "        'member_cutoff': 5,\n",
    "        'max_clusters': None\n",
    "         }),\n",
    "    ('blobs', blobs, {\n",
    "        'radius_cutoff': 0.4,\n",
    "        'cnn_cutoff': 20,\n",
    "        'member_cutoff': 2,\n",
    "        'max_clusters': None\n",
    "        }),\n",
    "    ('None', no_structure, {\n",
    "        'radius_cutoff': 0.5,\n",
    "        'cnn_cutoff': 20,\n",
    "        'member_cutoff': 1,\n",
    "        'max_clusters': None\n",
    "        }),\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:17.284040Z",
     "start_time": "2021-10-01T10:35:16.964484Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 51980 characters]\n",
      "text/plain": [
       "<Figure size 750x450 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the original data sets\n",
    "fig, ax = plt.subplots(2, 3)\n",
    "Ax = ax.flatten()\n",
    "\n",
    "for count, (name, data, *_) in enumerate(dsets):    \n",
    "    Ax[count].plot(data[:, 0], data[:, 1], **line_props)\n",
    "    Ax[count].set(**ax_props)\n",
    "    Ax[count].set_title(f'{name}', fontsize=10, pad=4)\n",
    "    \n",
    "fig.subplots_adjust(\n",
    "    left=0, right=1, bottom=0, top=1, wspace=0.1, hspace=0.3 \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CommonNN clustering using data point coordiantes as input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:18.770507Z",
     "start_time": "2021-10-01T10:35:17.285736Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        100       None      2         0.500     0.000     00:00:0.162\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        2         None      2         0.500     0.000     00:00:0.155\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.280     20        20        None      3         0.338     0.114     00:00:0.205\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.290     30        5         None      3         0.319     0.050     00:00:0.159\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.400     20        2         None      3         0.334     0.001     00:00:0.269\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        1         None      1         1.000     0.000     00:00:0.135\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 105020 characters]\n",
      "text/plain": [
       "<Figure size 750x450 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(2, 3)\n",
    "Ax = ax.flatten()\n",
    "\n",
    "for count, (name, data, params) in enumerate(dsets):\n",
    "\n",
    "    # Default clustering initialisation\n",
    "    clustering = cluster.Clustering(data)\n",
    "\n",
    "    # Calculate neighbours brute force\n",
    "    clustering.fit(**params)\n",
    "    print()\n",
    "\n",
    "    clustering.evaluate(ax=Ax[count], annotate_pos=\"random\")\n",
    "    Ax[count].set(**ax_props)\n",
    "    Ax[count].set_title(f'{name}', fontsize=10, pad=4)\n",
    "    \n",
    "fig.subplots_adjust(\n",
    "    left=0, right=1, bottom=0, top=1, wspace=0.1, hspace=0.3 \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When *raw* input data is presented to a `Clustering` on creation without specifying anything else, the obtained `Clustering` object aggregates the needed clustering components, assuming that it got point coordinates as a sequence of sequences. The neighbourhood of a specific point will be collected brute-force by computing the (Euclidean) distances to all other points and comparing them to the radius cutoff. This will be the slowest possible approach but it has fairly conservative memory usage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:18.774739Z",
     "start_time": "2021-10-01T10:35:18.771897Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering(\n",
      "    input_data=InputDataExtComponentsMemoryview,\n",
      "    fitter=FitterExtBFS(\n",
      "        ngetter=NeighboursGetterExtBruteForce(\n",
      "            dgetter=DistanceGetterExtMetric(\n",
      "                metric=MetricExtEuclideanReduced\n",
      "                ),\n",
      "            sorted=False,\n",
      "            selfcounting=True\n",
      "            ),\n",
      "        na=NeighboursExtVectorCPPUnorderedSet,\n",
      "        nb=NeighboursExtVectorCPPUnorderedSet,\n",
      "        checker=SimilarityCheckerExtSwitchContains,\n",
      "        queue=QueueExtFIFOQueue\n",
      "        ),\n",
      "    predictor=None\n",
      "    )\n"
     ]
    }
   ],
   "source": [
    "# Clustering components used by default (\"coordinates\" recipe)\n",
    "print(helper_base.indent_at_parens(str(clustering)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CommonNN clustering with pre-computed distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:20.024033Z",
     "start_time": "2021-10-01T10:35:18.776264Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        100       None      2         0.500     0.000     00:00:0.077\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        2         None      2         0.500     0.000     00:00:0.093\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.280     20        20        None      3         0.338     0.114     00:00:0.118\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.290     30        5         None      3         0.319     0.050     00:00:0.087\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.400     20        2         None      3         0.334     0.001     00:00:0.201\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        1         None      1         1.000     0.000     00:00:0.074\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 105324 characters]\n",
      "text/plain": [
       "<Figure size 750x450 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(2, 3)\n",
    "Ax = ax.flatten()\n",
    "\n",
    "for count, (name, data, params) in enumerate(dsets):\n",
    "    \n",
    "    # Default clustering initialisation\n",
    "    clustering = cluster.Clustering(data)  \n",
    "    \n",
    "    # Pre-compute distances and choose the corresponding recipe \n",
    "    distances = pairwise_distances(data)\n",
    "    clustering_dist = cluster.Clustering(\n",
    "        distances,\n",
    "        registered_recipe_key=\"distances\"\n",
    "        )   \n",
    "    \n",
    "    # Use pre-computed distances\n",
    "    clustering_dist.fit(**params)\n",
    "    clustering._labels = clustering_dist._labels\n",
    "    print()\n",
    "\n",
    "    clustering.evaluate(ax=Ax[count], annotate_pos=\"random\")\n",
    "    Ax[count].set(**ax_props)\n",
    "    Ax[count].set_title(f'{name}', fontsize=10, pad=4)\n",
    "    \n",
    "fig.subplots_adjust(\n",
    "    left=0, right=1, bottom=0, top=1, wspace=0.1, hspace=0.3 \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When *raw* input data is presented to a `Clustering` on creation in terms of a dense distance matrix, the \"distances\" recipe can be chosen to make the obtained `Clustering` object aggregate the needed clustering components. The neighbourhood of a specific point will be collected brute-force by looking up distances from the input data. This will be somewhat faster as no distances need to be calculated during the clustering but it has a high memory demand. It also allows to leverage smart external methods for the task of the distance calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:20.028330Z",
     "start_time": "2021-10-01T10:35:20.025635Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering(\n",
      "    input_data=InputDataExtComponentsMemoryview,\n",
      "    fitter=FitterExtBFS(\n",
      "        ngetter=NeighboursGetterExtBruteForce(\n",
      "            dgetter=DistanceGetterExtMetric(\n",
      "                metric=MetricExtPrecomputed\n",
      "                ),\n",
      "            sorted=False,\n",
      "            selfcounting=True\n",
      "            ),\n",
      "        na=NeighboursExtVectorCPPUnorderedSet,\n",
      "        nb=NeighboursExtVectorCPPUnorderedSet,\n",
      "        checker=SimilarityCheckerExtSwitchContains,\n",
      "        queue=QueueExtFIFOQueue\n",
      "        ),\n",
      "    predictor=None\n",
      "    )\n"
     ]
    }
   ],
   "source": [
    "# Clustering components used by the \"distance\" recipe\n",
    "print(helper_base.indent_at_parens(str(clustering_dist)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CommonNN clustering with pre-computed neighbourhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:21.128677Z",
     "start_time": "2021-10-01T10:35:20.029597Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        100       None      2         0.500     0.000     00:00:0.036\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        2         None      2         0.500     0.000     00:00:0.052\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.280     20        20        None      3         0.338     0.114     00:00:0.064\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.290     30        5         None      3         0.319     0.050     00:00:0.039\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.400     20        2         None      3         0.334     0.001     00:00:0.149\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        1         None      1         1.000     0.000     00:00:0.032\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 105160 characters]\n",
      "text/plain": [
       "<Figure size 750x450 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(2, 3)\n",
    "Ax = ax.flatten()\n",
    "\n",
    "for count, (name, data, params) in enumerate(dsets):\n",
    "    \n",
    "    clustering = cluster.Clustering(data)  \n",
    "    \n",
    "    # Pre-compute neighbourhoods and choose the corresponding recipe\n",
    "    tree = KDTree(data)\n",
    "    neighbourhoods = tree.query_radius(\n",
    "        data, r=params[\"radius_cutoff\"], return_distance=False\n",
    "        )\n",
    "    \n",
    "    clustering_neighbourhoods = cluster.Clustering(\n",
    "        neighbourhoods,\n",
    "        preparation_hook=hooks.prepare_neighbourhoods,\n",
    "        registered_recipe_key=\"neighbourhoods\"\n",
    "        )\n",
    "    \n",
    "    # Use pre-computed neighbourhoods\n",
    "    clustering_neighbourhoods.fit(**params)\n",
    "    clustering._labels = clustering_neighbourhoods._labels\n",
    "    print()\n",
    "\n",
    "    clustering.evaluate(ax=Ax[count], annotate_pos=\"random\")\n",
    "    Ax[count].set(**ax_props)\n",
    "    Ax[count].set_title(f'{name}', fontsize=10, pad=4)\n",
    "    \n",
    "fig.subplots_adjust(\n",
    "    left=0, right=1, bottom=0, top=1, wspace=0.1, hspace=0.3 \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When *raw* input data is presented to a `Clustering` on creation in terms of pre-computed neighbourhoods, the \"neighbourhoods\" recipe can be chosen to make the obtained `Clustering` object aggregate the needed clustering components. The neighbourhood of a specific point will looked up from the input data. This will be considerably faster and it has a lower memory demand compared to the \"distances\" case. It also allows to leverage smart external methods for the task of the neighbourhood calculation.\n",
    "\n",
    "Note, that the \"neighbourhoods\" recipe requires the *raw* neighbourhoods in the concrete format of a matrix in which the neighbourhoods of individual points are padded to the length of the largest neighbourhood. If neighbourhoods are presented in terms of a sequence of sequences (with varying length) as one would get from a tree query to `sklearn.neighbors.KDTree`, this can be transformed into the suitable format using the preparation hook `hooks.prepare_neighbourhoods`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:21.133193Z",
     "start_time": "2021-10-01T10:35:21.130254Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering(\n",
      "    input_data=InputDataExtNeighbourhoodsMemoryview,\n",
      "    fitter=FitterExtBFS(\n",
      "        ngetter=NeighboursGetterExtLookup(\n",
      "            sorted=False,\n",
      "            selfcounting=True\n",
      "            ),\n",
      "        na=NeighboursExtVectorCPPUnorderedSet,\n",
      "        nb=NeighboursExtVectorCPPUnorderedSet,\n",
      "        checker=SimilarityCheckerExtSwitchContains,\n",
      "        queue=QueueExtFIFOQueue\n",
      "        ),\n",
      "    predictor=None\n",
      "    )\n"
     ]
    }
   ],
   "source": [
    "# Clustering components used by the \"distance\" recipe\n",
    "print(helper_base.indent_at_parens(str(clustering_neighbourhoods)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Common-nearest-neighbours clustering with pre-computed sorted neighbourhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:21.974270Z",
     "start_time": "2021-10-01T10:35:21.134473Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        100       None      2         0.500     0.000     00:00:0.003\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        2         None      2         0.500     0.000     00:00:0.004\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.280     20        20        None      3         0.338     0.114     00:00:0.006\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.290     30        5         None      3         0.319     0.050     00:00:0.004\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.400     20        2         None      3         0.334     0.001     00:00:0.011\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "#points   r         c         min       max       #clusters %largest  %noise    time     \n",
      "2000      0.500     20        1         None      1         1.000     0.000     00:00:0.004\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 104752 characters]\n",
      "text/plain": [
       "<Figure size 750x450 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(2, 3)\n",
    "Ax = ax.flatten()\n",
    "\n",
    "for count, (name, data, params) in enumerate(dsets):\n",
    "\n",
    "    clustering = cluster.Clustering(data)  \n",
    "    \n",
    "    # Pre-compute and sort neighbourhoods and choose the corresponding recipe\n",
    "    tree = KDTree(data)\n",
    "    neighbourhoods = tree.query_radius(\n",
    "        data, r=params[\"radius_cutoff\"], return_distance=False\n",
    "        )\n",
    "\n",
    "    for n in neighbourhoods:\n",
    "        n.sort()\n",
    "\n",
    "    # Custom initialisation using a clustering builder\n",
    "    clustering_neighbourhoods_sorted = cluster.Clustering(\n",
    "        neighbourhoods,\n",
    "        preparation_hook=hooks.prepare_neighbourhoods,\n",
    "        registered_recipe_key=\"sorted_neighbourhoods\"\n",
    "        )\n",
    "    \n",
    "    # Use pre-computed neighbourhoods\n",
    "    clustering_neighbourhoods_sorted.fit(**params)\n",
    "    clustering._labels = clustering_neighbourhoods_sorted._labels\n",
    "    print()\n",
    "\n",
    "    clustering.evaluate(ax=Ax[count], annotate_pos=\"random\")\n",
    "    Ax[count].set(**ax_props)\n",
    "    Ax[count].set_title(f'{name}', fontsize=10, pad=4)\n",
    "    \n",
    "fig.subplots_adjust(\n",
    "    left=0, right=1, bottom=0, top=1, wspace=0.1, hspace=0.3 \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pre-computed neighbourhood information can be additionally sorted. While this has the same memory demand as using unsorted neighbourhoods, it offers the fastest clustering. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-10-01T10:35:21.978554Z",
     "start_time": "2021-10-01T10:35:21.975762Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clustering(\n",
      "    input_data=InputDataExtNeighbourhoodsMemoryview,\n",
      "    fitter=FitterExtBFS(\n",
      "        ngetter=NeighboursGetterExtLookup(\n",
      "            sorted=True,\n",
      "            selfcounting=True\n",
      "            ),\n",
      "        na=NeighboursExtVector,\n",
      "        nb=NeighboursExtVector,\n",
      "        checker=SimilarityCheckerExtScreensorted,\n",
      "        queue=QueueExtFIFOQueue\n",
      "        ),\n",
      "    predictor=None\n",
      "    )\n"
     ]
    }
   ],
   "source": [
    "# Clustering components used by the \"distance\" recipe\n",
    "print(helper_base.indent_at_parens(str(clustering_neighbourhoods_sorted)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cnnclustering38",
   "language": "python",
   "name": "cnnclustering38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "164.986px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
