{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `h3ronpy`\n",
    "\n",
    "[![PyPI version](https://img.shields.io/pypi/v/h3ronpy)](https://pypi.python.org/pypi/h3ronpy/) [![dependency status](https://deps.rs/repo/github/nmandery/h3ronpy/status.svg)](https://deps.rs/repo/github/nmandery/h3ronpy) [![DOI](https://zenodo.org/badge/402118389.svg)](https://zenodo.org/badge/latestdoi/402118389)\n",
    "\n",
    "Python extension for the [H3](https://h3geo.org) geospatial indexing system exposing some functionalities of the [h3ron](https://github.com/nmandery/h3ron) rust crates to the python language and integrating with the [numpy](https://numpy.org/), [pandas](https://pandas.pydata.org/),\n",
    "[geopandas](https://geopandas.org/), [rasterio](https://rasterio.readthedocs.io/en/latest/) and [gdal](https://gdal.org/) libraries. \n",
    "\n",
    "One goal is to not duplicate any functions already implemented by the [official H3 python bindings](https://github.com/uber/h3-py).\n",
    "\n",
    "This library is in parts parallelized using [rayon](https://github.com/rayon-rs/rayon). The number of threads can be controlled as \n",
    "described in [the rayon FAQ](https://github.com/rayon-rs/rayon/blob/master/FAQ.md#how-many-threads-will-rayon-spawn)\n",
    "\n",
    "\n",
    "## Installation\n",
    "\n",
    "### via PyPI\n",
    "\n",
    "Regarding linux: to install the `manylinux_2_24` wheels, a fairly recent version of `pip` is required. So in case `pip` only finds the source distribution, upgrade `pip` using\n",
    "\n",
    "```\n",
    "pip install --upgrade pip\n",
    "```\n",
    "\n",
    "### Building manually\n",
    "\n",
    "To build this extension from source, you will need:\n",
    "\n",
    "* Rust. Install the latest version using [rustup](https://rustup.rs/)\n",
    "* A C compiler for the libh3 sources, for example `clang`\n",
    "* Python 3.x and the corresponding C headers\n",
    "* The dependencies from the [requirements.dev.txt](./requirements.dev.txt) file.\n",
    "\n",
    "On Ubuntu most system-level dependencies should be available after running rustup and \n",
    "\n",
    "```shell\n",
    "sudo apt-get install python3-dev build-essential clang\n",
    "```\n",
    "\n",
    "The build is done using [maturin](https://github.com/PyO3/maturin)\n",
    "\n",
    "There are three main commands:\n",
    "\n",
    "* `maturin publish` builds the crate into python packages and publishes them to pypi.\n",
    "* `maturin build` builds the wheels and stores them in a folder (`../target/wheels` by default), but doesn't upload them. It's possible to upload those with [twine](https://github.com/pypa/twine).\n",
    "* `maturin develop` builds the crate and installs it as a python module directly in the current virtualenv.\n",
    "\n",
    "To build the extension just use the `maturin build --release` command for an optimized build.\n",
    "\n",
    "\n",
    "## Logging\n",
    "\n",
    "This library uses rusts [`log` crate](https://docs.rs/log) together with the [`env_logger` crate](https://docs.rs/env_logger). \n",
    "This means that logging to `stdout` can be  controlled via environment variables. Set `RUST_LOG` to `debug`, `error`, `info`, \n",
    "`warn`, or `trace` for the corresponding log output.\n",
    "\n",
    "For more fine-grained logging settings, see the documentation of `env_logger`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "The following sections depend on the libraries from the `requirements.documentation.txt` file being installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import rasterio\n",
    "from rasterio.plot import show\n",
    "import numpy as np\n",
    "import h3.api.numpy_int as h3\n",
    "from scipy import ndimage\n",
    "import geopandas as gpd\n",
    "\n",
    "# increase the plot size\n",
    "pyplot.rcParams['figure.dpi'] = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting raster data to H3\n",
    "\n",
    "#### Prepare a dataset using rasterio first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "src = rasterio.open(\"data/europe-and-north-africa.tif\")\n",
    "print(src.colorinterp)\n",
    "\n",
    "green = src.read(2)\n",
    "blue = src.read(3)\n",
    "print(green.shape)\n",
    "\n",
    "show(src)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do image processing, like this messy extraction and smoothing of the greenish pixels done here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "vegetation_mask = (green < 250) & (blue < 50)\n",
    "ocean_mask = (green >= 6) & (green <= 14) & (blue >= 47) & (blue <= 54)\n",
    "vegetation_nodata_value = 0\n",
    "\n",
    "vegetation = np.full(green.shape, 10, dtype=\"int8\")\n",
    "vegetation[ocean_mask] = vegetation_nodata_value\n",
    "vegetation[vegetation_mask] = 20\n",
    "#pyplot.imshow(vegetation, cmap='Greens')\n",
    "\n",
    "# smooth a bit to remove single pixels\n",
    "vegetation = ndimage.gaussian_filter(vegetation, sigma=.7)\n",
    "vegetation[vegetation <= 5] = vegetation_nodata_value\n",
    "vegetation[(vegetation > 0) & (vegetation < 15)] = 1\n",
    "vegetation[vegetation >= 15] = 2\n",
    "vegetation[ocean_mask] = vegetation_nodata_value\n",
    "\n",
    "vegetation_plot_args = dict(cmap='Greens', vmin=0, vmax=2)\n",
    "\n",
    "pyplot.imshow(vegetation, **vegetation_plot_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "vegetation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert the raster numpy array to H3\n",
    "\n",
    "Find the h3 resolution to use. See also the docstrings of the used functions and of the `h3ronpy.raster` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from h3ronpy.raster import nearest_h3_resolution\n",
    "\n",
    "h3_res = nearest_h3_resolution(vegetation.shape, src.transform, search_mode=\"smaller_than_pixel\")\n",
    "print(f\"Using H3 resolution {h3_res}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ... directly into a pandas (geo-)dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from h3ronpy.raster import raster_to_dataframe\n",
    "\n",
    "print(\"conversion start\")\n",
    "vegetation_h3_df = raster_to_dataframe(vegetation, src.transform, h3_res, nodata_value=vegetation_nodata_value, compacted=True, geo=True)\n",
    "print(\"conversion done\")\n",
    "\n",
    "print(\"plotting ... this may take a bit\")\n",
    "vegetation_h3_df.plot(column=\"value\", linewidth=0.2, edgecolor=\"black\", **vegetation_plot_args)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using GDAL instead of rasterio\n",
    "\n",
    "To not repeat any steps from above, we just save the processed data using `rasterio` and load it again using `gdal`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ... TBW ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert vectordata to H3 indexes\n",
    "\n",
    "The conversion of GeoDataFrames is parallelized using the available CPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import geopandas as gpd\n",
    "from h3ronpy import vector, util\n",
    "\n",
    "world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))\n",
    "africa = world[world[\"continent\"] == \"Africa\"]\n",
    "africa.plot(column=\"name\")\n",
    "\n",
    "# convert to h3 cells and build a new geodataframe\n",
    "gdf = util.h3index_column_to_geodataframe(vector.geodataframe_to_h3(africa, 3))\n",
    "gdf.plot(column=\"name\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Polygon to H3 conversion is based on the H3 `polyfill` function, which fills based on the containment of the H3 cells centroid in the polygon geometry. Depending on the shape of the geometry the resulting cells may look like below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fill_namibia(**kw):\n",
    "    namibia = world[world[\"name\"] == \"Namibia\"]\n",
    "\n",
    "    cell_ax = util.h3index_column_to_geodataframe(vector.geodataframe_to_h3(namibia, 3, **kw)).plot()\n",
    "\n",
    "    return namibia.plot(ax=cell_ax, facecolor=(0,0,0,0), edgecolor='black')\n",
    "\n",
    "fill_namibia()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `intersecting` argument extends the polyfill a bit to include all cells which intersect with the polygon geometry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fill_namibia(intersecting=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting H3 indexes to vector geometries\n",
    "\n",
    "The main functionality already exists with the `LinkedPolygons` in the H3 library itself. This library here just integrates this and adds some polish - like the smoothing - around it.\n",
    "\n",
    "Additionally the polygon objects implement the [python geo interface](https://gist.github.com/sgillies/2217756), so they are compatible to most mapping libraries - see the geopandas plotting below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from h3ronpy import Polygon\n",
    "\n",
    "squares_and_rings = np.concatenate([\n",
    "    h3.hex_ring(h3.geo_to_h3(12.3, 15.4, 10), 6),\n",
    "    h3.hex_ring(h3.geo_to_h3(12.31, 15.39, 10), 10),\n",
    "    h3.hex_ring(h3.geo_to_h3(12.29, 15.395, 10), 8),\n",
    "    h3.polyfill_polygon([\n",
    "        (12.25, 15.28),\n",
    "        (12.25, 15.37),\n",
    "        (12.35, 15.37),\n",
    "        (12.35, 15.28), \n",
    "        (12.25, 15.28)\n",
    "    ], 10, lnglat_order=False)\n",
    "])\n",
    "sqr_geoms = Polygon.from_h3indexes(squares_and_rings) # returns a list of polygons\n",
    "gpd.GeoDataFrame({}, geometry=sqr_geoms).plot(edgecolor=\"black\", linewidth=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a cleaner look, the generated vector geometries can also be smoothed using a variation of [Chaikins smoothing algorithm](http://www.idav.ucdavis.edu/education/CAGDNotes/Chaikins-Algorithm/Chaikins-Algorithm.html). This includes a non-destructive simplification where the number of vertices gets reduced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sqr_smoothed = Polygon.from_h3indexes(squares_and_rings, smoothen=True)\n",
    "gpd.GeoDataFrame({}, geometry=sqr_smoothed).plot(edgecolor=\"black\", linewidth=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also the `align_to_h3_resolution` variant of this which will generate polygons not exceeding the area of a given parent resolution.\n",
    "\n",
    "Corners will be aligned to the corners of the parent resolution when they are less than an\n",
    "edge length away from them. This is to avoid gaps when `smoothen` is set to true."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "box_indexes = h3.polyfill_polygon([\n",
    "        (12.25, 15.30),\n",
    "        (12.25, 15.35),\n",
    "        (12.30, 15.35),\n",
    "        (12.30, 15.30), \n",
    "        (12.25, 15.30)\n",
    "], 10)\n",
    "\n",
    " # aling the geometries to h3 resolution 2 where complete edges touch. This reduced the number of vertices \n",
    "# a fair amount\n",
    "box_aligned_geoms = Polygon.from_h3indexes_aligned(box_indexes, 7, smoothen=True)\n",
    "gpd.GeoDataFrame({}, geometry=box_aligned_geoms).plot(edgecolor=\"black\", linewidth=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorized operations on h3indexes\n",
    "\n",
    "The `h3ronpy.op` module contains vectorized functions for working with h3indexes. Currently mostly for working with k-rings - more will propably be added in the future."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
