{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyrax Demonstration\n",
    "\n",
    "For this demonstration we'll walk through a simplified version of a typical machine learning workflow supported by Hyrax."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mtauraso/miniforge3/envs/hyrax/lib/python3.10/site-packages/ignite/handlers/checkpoint.py:16: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.\n",
      "  from torch.distributed.optim import ZeroRedundancyOptimizer\n"
     ]
    }
   ],
   "source": [
    "import hyrax\n",
    "import pooch\n",
    "import subprocess\n",
    "\n",
    "import chromadb\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from pathlib import Path\n",
    "from IPython.display import IFrame\n",
    "\n",
    "from hyrax.config_utils import find_most_recent_results_dir\n",
    "from mpr_demo_plotting import sort_objects_by_median_distance, plot_grid, plot_umap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download a sample HSC dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path = pooch.retrieve(\n",
    "    # DOI for Example HSC dataset\n",
    "    url=\"doi:10.5281/zenodo.14498536/hsc_demo_data.zip\",\n",
    "    known_hash=\"md5:1be05a6b49505054de441a7262a09671\",\n",
    "    fname=\"example_hsc_new.zip\",\n",
    "    path=\"../../data\",\n",
    "    processor=pooch.Unzip(extract_dir=\".\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset is comprised of approximately 993 cutouts from the Hyper Suprime Cam survey.\n",
    "Each cutout includes i, r and g bands and is 8 arcseconds on a side.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and configure a Hyrax object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:37:13,094 hyrax:INFO] Runtime Config read from: /Users/mtauraso/src/fibad/src/hyrax/hyrax_default_config.toml\n"
     ]
    }
   ],
   "source": [
    "h = hyrax.Hyrax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An instance of the `Hyrax` class will be used through out this demo.\n",
    "Under the hood when it is created, it will:\n",
    "-  Load the configuration file specified (here it's using the built in default).\n",
    "-  Parse the configuration file for external libraries and add those to the appropriate registries.\n",
    "-  Prepare logging for the system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the location of the data to use for training\n",
    "h.config[\"general\"][\"data_dir\"] = \"../../data/hsc_8asec_1000\"\n",
    "\n",
    "# Specify the dataset class that represents the data\n",
    "h.config[\"data_set\"][\"name\"] = \"HSCDataSet\"\n",
    "h.config[\"data_set\"][\"train_size\"] = 0.8\n",
    "h.config[\"data_set\"][\"validate_size\"] = 0.2\n",
    "h.config[\"data_set\"][\"test_size\"] = 0.0\n",
    "\n",
    "# Select the model to use for training\n",
    "h.config[\"model\"][\"name\"] = \"HyraxAutoencoder\"\n",
    "\n",
    "# Set the number of epochs and batch size for training.\n",
    "h.config[\"train\"][\"epochs\"] = 20\n",
    "h.config[\"data_loader\"][\"batch_size\"] = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default configuration needs a few tweaks to work for this demo.\n",
    "We've updated the location of our sample data, and specified which model we want to train.\n",
    "\n",
    "The configuration is represented as nested python dictionary. This allows for easy manipulation in a notebook via the `.config` attribute of the hyrax instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:37:18,063 hyrax.data_sets.hsc_data_set:INFO] Checking file dimensions to determine standard cutout size...\n",
      "[2025-04-18 16:37:18,066 hyrax.data_sets.fits_image_dataset:INFO] FitsImageDataSet has 993 objects\n",
      "[2025-04-18 16:37:18,075 hyrax.data_sets.hsc_data_set:INFO] Processed 993 objects for pruning\n",
      "[2025-04-18 16:37:18,076 hyrax.data_sets.fits_image_dataset:INFO] Preloading FitsImageDataSet cache...\n",
      "[2025-04-18 16:37:18,097 hyrax.models.model_registry:INFO] Using criterion: torch.nn.CrossEntropyLoss with default arguments.\n",
      "2025-04-18 16:37:18,115 ignite.distributed.auto.auto_dataloader INFO: Use data loader kwargs for dataset '<hyrax.data_sets.hsc': \n",
      "\t{'sampler': <hyrax.pytorch_ignite.SubsetSequentialSampler object at 0x1666aee00>, 'batch_size': 32, 'shuffle': False, 'pin_memory': False}\n",
      "2025-04-18 16:37:18,118 ignite.distributed.auto.auto_dataloader INFO: Use data loader kwargs for dataset '<hyrax.data_sets.hsc': \n",
      "\t{'sampler': <hyrax.pytorch_ignite.SubsetSequentialSampler object at 0x1666afb80>, 'batch_size': 32, 'shuffle': False, 'pin_memory': False}\n",
      "/Users/mtauraso/miniforge3/envs/hyrax/lib/python3.10/site-packages/ignite/handlers/tqdm_logger.py:127: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train': (<torch.utils.data.dataloader.DataLoader object at 0x16666e020>, [131, 613, 731, 646, 4, 920, 347, 616, 359, 49, 18, 937, 510, 539, 139, 24, 851, 382, 324, 963, 899, 293, 778, 417, 924, 650, 619, 866, 446, 505, 541, 747, 958, 575, 902, 547, 898, 370, 519, 310, 471, 789, 197, 115, 732, 507, 55, 183, 258, 90, 956, 172, 430, 166, 401, 860, 658, 728, 448, 840, 729, 768, 982, 917, 330, 890, 779, 396, 476, 153, 488, 588, 167, 843, 486, 798, 916, 208, 579, 882, 631, 376, 171, 93, 644, 465, 640, 105, 199, 499, 643, 551, 515, 305, 657, 754, 745, 369, 933, 634, 562, 668, 782, 825, 590, 853, 805, 481, 674, 531, 762, 299, 148, 7, 688, 398, 38, 452, 314, 381, 497, 427, 113, 912, 679, 70, 485, 328, 973, 689, 609, 671, 797, 741, 221, 767, 276, 533, 457, 864, 637, 500, 555, 715, 431, 295, 473, 64, 513, 981, 662, 969, 984, 584, 243, 460, 255, 520, 641, 378, 683, 765, 390, 820, 372, 989, 716, 811, 743, 245, 753, 89, 114, 695, 812, 914, 603, 425, 257, 455, 591, 196, 835, 615, 918, 677, 22, 706, 786, 92, 810, 307, 412, 288, 99, 218, 934, 10, 304, 0, 358, 136, 104, 149, 182, 605, 439, 854, 157, 213, 355, 77, 249, 983, 190, 296, 3, 726, 459, 626, 976, 826, 300, 274, 472, 150, 669, 23, 402, 879, 846, 656, 365, 834, 98, 144, 227, 107, 407, 684, 67, 327, 137, 72, 909, 385, 29, 202, 857, 341, 164, 518, 35, 286, 428, 36, 17, 558, 896, 755, 53, 730, 462, 599, 742, 872, 819, 749, 230, 960, 867, 16, 97, 814, 126, 194, 311, 804, 335, 850, 816, 298, 343, 703, 60, 292, 203, 537, 483, 594, 580, 878, 192, 760, 318, 532, 554, 395, 932, 406, 714, 241, 974, 253, 5, 287, 573, 709, 290, 795, 52, 121, 908, 818, 926, 885, 238, 400, 821, 240, 802, 266, 681, 102, 186, 39, 399, 884, 133, 736, 498, 456, 151, 334, 458, 895, 392, 845, 85, 169, 769, 530, 941, 187, 803, 582, 572, 59, 758, 967, 188, 88, 246, 475, 101, 722, 265, 635, 278, 433, 645, 583, 667, 985, 901, 309, 655, 163, 970, 461, 317, 526, 875, 712, 801, 223, 824, 602, 841, 28, 326, 180, 342, 574, 117, 58, 642, 435, 110, 859, 815, 316, 178, 19, 639, 761, 490, 50, 491, 319, 8, 527, 509, 678, 205, 833, 12, 138, 721, 734, 852, 877, 432, 454, 373, 281, 991, 262, 776, 832, 589, 959, 665, 746, 638, 65, 248, 349, 553, 128, 529, 966, 629, 200, 71, 447, 947, 118, 849, 611, 892, 817, 346, 225, 873, 284, 283, 73, 362, 512, 31, 780, 502, 717, 675, 659, 336, 790, 988, 54, 146, 474, 315, 87, 664, 793, 231, 944, 979, 450, 320, 576, 492, 598, 693, 968, 122, 724, 750, 388, 251, 708, 608, 348, 627, 557, 925, 308, 185, 277, 306, 103, 883, 686, 680, 175, 224, 597, 40, 285, 938, 333, 270, 842, 130, 546, 766, 303, 504, 910, 607, 79, 62, 751, 331, 43, 606, 380, 69, 542, 48, 713, 569, 207, 379, 254, 61, 612, 226, 942, 236, 161, 886, 467, 651, 501, 263, 337, 434, 75, 235, 108, 141, 76, 787, 229, 697, 661, 568, 503, 891, 239, 112, 222, 975, 514, 748, 889, 132, 201, 20, 242, 210, 622, 921, 289, 453, 377, 544, 344, 744, 904, 350, 618, 928, 397, 949, 540, 905, 727, 799, 496, 663, 487, 596, 560, 83, 470, 950, 352, 135, 922, 81, 426, 957, 649, 647, 414, 410, 911, 493, 32, 585, 366, 561, 707, 96, 962, 856, 556, 443, 691, 345, 604, 863, 628, 870, 508, 267, 156, 887, 586, 391, 666, 489, 545, 360, 291, 21, 940, 595, 781, 986, 420, 191, 806, 363, 415, 592, 429, 549, 422, 214, 567, 837, 943, 528, 600, 468, 100, 624, 109, 735, 621, 955, 858, 733, 237, 571, 129, 409, 536, 676, 690, 830, 176, 419, 394, 927, 862, 915, 275, 581, 654, 987, 313, 405, 868, 913, 636, 479, 322, 41, 548, 80, 393, 566, 56, 961, 124, 757, 247, 897, 523, 250, 174, 173, 159, 198, 978, 698, 623, 340, 259, 418, 535, 752, 339, 294, 906, 441, 2, 440, 127, 368, 383, 445, 763, 935, 791, 375, 329, 367, 84, 91, 145, 482, 948, 907, 219, 653, 244, 301, 181, 66, 601, 437, 931, 361, 152, 711, 389, 794, 964, 865, 785, 51, 323, 808, 670, 232, 34, 756, 170, 829, 617, 796, 652, 692, 438, 177, 900, 228, 861, 936, 564, 351, 261, 280, 952, 552, 234, 570, 919, 506, 704, 836, 534, 699, 86, 687, 923, 827, 423, 233, 685, 694, 332, 386, 813, 673, 577, 155, 633]), 'validate': (<torch.utils.data.dataloader.DataLoader object at 0x1666af8b0>, [630, 593, 116, 364, 807, 847, 45, 876, 992, 209, 74, 972, 463, 893, 625, 63, 416, 720, 216, 282, 47, 710, 831, 464, 256, 939, 484, 449, 929, 881, 9, 951, 954, 771, 371, 495, 696, 632, 620, 68, 855, 106, 408, 823, 119, 357, 160, 738, 525, 930, 770, 784, 700, 723, 777, 701, 424, 162, 442, 404, 945, 268, 26, 838, 42, 521, 788, 614, 82, 44, 828, 774, 977, 6, 844, 140, 773, 903, 587, 740, 702, 953, 15, 563, 271, 125, 273, 264, 120, 648, 772, 354, 980, 511, 211, 610, 444, 217, 466, 33, 143, 11, 387, 134, 325, 154, 725, 871, 739, 272, 550, 158, 25, 792, 94, 413, 880, 252, 193, 338, 946, 195, 37, 179, 165, 451, 220, 524, 478, 469, 297, 874, 30, 269, 204, 312, 142, 279, 206, 839, 718, 990, 759, 147, 783, 800, 971, 212, 123, 737, 168, 477, 672, 559, 302, 95, 822, 14, 321, 682, 13, 517, 764, 353, 215, 578, 480, 1, 869, 775, 809, 894, 189, 705, 516, 719, 27, 565, 384, 660, 421, 848, 494, 78, 260, 374, 965, 543, 522, 57, 356, 411, 403, 888, 538, 111, 46, 436, 184])}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025/04/18 16:37:18 WARNING mlflow.system_metrics.system_metrics_monitor: Skip logging GPU metrics because creating `GPUMonitor` failed with error: Failed to initialize NVML, skip logging GPU metrics: NVML Shared Library Not Found.\n",
      "2025/04/18 16:37:18 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.\n",
      "[2025-04-18 16:37:18,776 hyrax.pytorch_ignite:INFO] Training model on device: mps\n",
      "[2025-04-18 16:37:21,703 hyrax.data_sets.fits_image_dataset:INFO] Processed 992 objects\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4f30a72a1864c9d915b058b20dd341f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 25\n",
      "\tepoch: 1\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ab77e29b8324cfcb35f7233d05300bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 50\n",
      "\tepoch: 2\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4701f00c3c84cc4b8590ff68aad2e9a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 75\n",
      "\tepoch: 3\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b4ee91e8b4ce4dc182cf963459632473",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 100\n",
      "\tepoch: 4\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea1ae5ac911d46c59ea840d60a7568b2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 125\n",
      "\tepoch: 5\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2500c3ba8640485b81959e5b6d21006a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 150\n",
      "\tepoch: 6\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0d00fa3d56043ec8d3b9b8100fd225b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 175\n",
      "\tepoch: 7\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9060d1dcc13401e9fb8e91a4782cda7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 200\n",
      "\tepoch: 8\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3044fc853a7e49559891faf3046359ca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 225\n",
      "\tepoch: 9\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8ed96dd2f3e545c2a2973a69bb8bc72f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 250\n",
      "\tepoch: 10\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7695a7bac3aa45159c2b556fc6550d16",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 275\n",
      "\tepoch: 11\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0b8a6c7665446a4abbfdb57ecfcf9c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 300\n",
      "\tepoch: 12\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ed168525c0c45fcaf1a230d86877b20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 325\n",
      "\tepoch: 13\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "516446a8f6ae46d4950c9997e713ce24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 350\n",
      "\tepoch: 14\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eafe48a10f7f4613b57851e5cf6e4040",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 375\n",
      "\tepoch: 15\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ef974363f634d03bfb65b10ce402788",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 400\n",
      "\tepoch: 16\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "64cc55226f094b3e8628edec34d37a6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 425\n",
      "\tepoch: 17\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e9a974624b64a9daa277fcfc40d01b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 450\n",
      "\tepoch: 18\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3d4d506fd83c442094efbbe0cde4d06f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 475\n",
      "\tepoch: 19\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af1fabf05e8845829bfa1c4a99b084bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:37:35,531 hyrax.pytorch_ignite:INFO] Total training time: 16.75[s]\n",
      "[2025-04-18 16:37:35,531 hyrax.pytorch_ignite:INFO] Latest checkpoint saved as: /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163718-train-_hkr/checkpoint_epoch_20.pt\n",
      "[2025-04-18 16:37:35,531 hyrax.pytorch_ignite:INFO] Best metric checkpoint saved as: /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163718-train-_hkr/checkpoint_20_loss=-174.2037.pt\n",
      "2025/04/18 16:37:35 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...\n",
      "2025/04/18 16:37:35 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "State:\n",
      "\titeration: 500\n",
      "\tepoch: 20\n",
      "\tepoch_length: 25\n",
      "\tmax_epochs: 20\n",
      "\toutput: <class 'dict'>\n",
      "\tbatch: <class 'torch.Tensor'>\n",
      "\tmetrics: <class 'dict'>\n",
      "\tdataloader: <class 'torch.utils.data.dataloader.DataLoader'>\n",
      "\tseed: <class 'NoneType'>\n",
      "\ttimes: <class 'dict'>\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:37:35,575 hyrax.train:INFO] Finished Training\n",
      "[2025-04-18 16:37:35,762 hyrax.model_exporters:INFO] Exported model to ONNX format: /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163718-train-_hkr/example_model_opset_20.onnx\n"
     ]
    }
   ],
   "source": [
    "h.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we call `.train()` to train the model there's a lot going on under the hood:\n",
    "- The model is automatically loaded onto the fastest hardware available.\n",
    "- A data loader is instantiated and configured to load batches of data to the same hardware.\n",
    "- A new timestamped directory is created under the configured results directory where all output is saved.\n",
    "- The configuration becomes immutable and a copy is saved for reproducibility.\n",
    "- The model and system metrics start being logged for review in both TensorBoard and MLFlow.\n",
    "- Checkpoints are saved automatically both at the last epoch and at the epoch with the lowest loss value.\n",
    "- Finally the model weights file is saved.\n",
    "\n",
    "Training time depends heavily on the hardware available, model, and training parameters.\n",
    "For a point of reference training takes about 40s for this case:\n",
    "- Model trained: Hyrax autoencoder\n",
    "- Dataset and size: Example HSC data, 993 samples, 96x96 pixel cutouts\n",
    "- Number of epochs: 20\n",
    "- Batch size: 32\n",
    "- Hardware: Desktop with GTX 1660 Super GPU\n",
    "\n",
    "While we train on only about 1000 samples here, Hyrax training has scaled up to over 1M samples on an HPC system with access to multiple GPUs without requiring the user to make any code changes.\n",
    "To do so, the command line interface of Hyrax was used to work within a Slurm environment like so:\n",
    "```\n",
    ">> hyrax train --runtime-config ./results/<timestamped_directory>/runtime_config.toml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quickly evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-7a26b6cd99fb35e6\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-7a26b6cd99fb35e6\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6006;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%reload_ext tensorboard\n",
    "%tensorboard --logdir {h.config['general']['results_dir']}"
   ]
  },
  {
   "attachments": {
    "tensorboard_screenshot.JPG": {
     "image/jpeg": "[encoded data, 149380 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preceding cell will start TensorBoard, to perform simple evaluation of trained models. While TensorBoard can run easily in a notebook, when that notebook is rendered to HTML (for demonstration or documentation) the server backing the TensorBoard UI isn't included in the rendering. If the cell above was run locally, the resulting UI would look similar to the following screen shot.\n",
    "\n",
    "![tensorboard_screenshot.JPG](attachment:tensorboard_screenshot.JPG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a new model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "from hyrax.models.model_registry import hyrax_model\n",
    "\n",
    "\n",
    "@hyrax_model  # This decorator registers the model with the Hyrax framework\n",
    "class TrialAutoencoder(nn.Module):\n",
    "    def __init__(self, config, shape):\n",
    "        super().__init__()\n",
    "        self.config = config\n",
    "\n",
    "        # Encoder\n",
    "        self.encoder = nn.Sequential(\n",
    "            nn.Conv2d(3, 16, kernel_size=3, stride=2, padding=1),  # (16, 48, 48)\n",
    "            nn.ReLU(),\n",
    "            nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1),  # (32, 24, 24)\n",
    "            nn.ReLU(),\n",
    "            nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1),  # (64, 12, 12)\n",
    "            nn.ReLU(),\n",
    "            nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=1),  # (128, 6, 6)\n",
    "            nn.ReLU(),\n",
    "        )\n",
    "\n",
    "        # Decoder\n",
    "        self.decoder = nn.Sequential(\n",
    "            nn.ConvTranspose2d(128, 64, kernel_size=3, stride=2, padding=1, output_padding=1),  # (64, 12, 12)\n",
    "            nn.ReLU(),\n",
    "            nn.ConvTranspose2d(64, 32, kernel_size=3, stride=2, padding=1, output_padding=1),  # (32, 24, 24)\n",
    "            nn.ReLU(),\n",
    "            nn.ConvTranspose2d(32, 16, kernel_size=3, stride=2, padding=1, output_padding=1),  # (16, 48, 48)\n",
    "            nn.ReLU(),\n",
    "            nn.ConvTranspose2d(16, 3, kernel_size=3, stride=2, padding=1, output_padding=1),  # (3, 96, 96)\n",
    "            nn.Sigmoid(),  # Normalize output to [0, 1]\n",
    "        )\n",
    "\n",
    "    def _eval_encoder(self, x):\n",
    "        return self.encoder(x)\n",
    "\n",
    "    def _eval_decoder(self, x):\n",
    "        return self.decoder(x)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self._eval_encoder(x)\n",
    "\n",
    "    def train_step(self, x):\n",
    "        z = self._eval_encoder(x)\n",
    "        x_hat = self._eval_decoder(z)\n",
    "\n",
    "        loss = self.criterion(x, x_hat)\n",
    "        loss = loss.sum(dim=[1, 2, 3]).mean(dim=[0])\n",
    "        self.optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        self.optimizer.step()\n",
    "        return {\"loss\": loss.item()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "New models can be written in a notebook for easier experimentation.\n",
    "Above, an autoencoder is written for comparison against the builtin `HyraxAutoencoder` where the only difference is that the builtin autoencoder uses `nn.GeLU` while `nn.ReLU` is used here.\n",
    "\n",
    "Note that the class is decorated with `@hyrax_model`, this decorator automatically provides several conveniences for the user:\n",
    "- The model class is registered with Hyrax for training.\n",
    "- Default loss and optimizer functions are provided.\n",
    "- Support for defining loss and optimizer functions in the Hyrax configuration.\n",
    "- Methods to save and load model weights are provided included.\n",
    "- Automatic verification that required methods were implemented.\n",
    "\n",
    "In addition to the `@hyrax_model`, other decorators to support extensibility and reduce boilerplate code have been developed including:\n",
    "- `@hyrax_dataset` - for rapid development of new data set interfaces\n",
    "- `@hyrax_verb` - for new core actions i.e. `h.custom_train(...)`, `h.bespoke_predict(...)`, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the newly defined model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-03-26 13:52:40,724 hyrax.data_sets.hsc_data_set:INFO] Processed 993 objects for pruning\n",
      "[2025-03-26 13:52:40,725 hyrax.data_sets.hsc_data_set:INFO] Checking file dimensions to determine standard cutout size...\n",
      "[2025-03-26 13:52:40,727 hyrax.data_sets.hsc_data_set:INFO] HSC Data set loader has 993 objects\n",
      "[2025-03-26 13:52:40,728 hyrax.data_sets.hsc_data_set:INFO] Preloading HSCDataSet cache...\n",
      "[2025-03-26 13:52:40,740 hyrax.models.model_registry:INFO] Using criterion: torch.nn.MSELoss with arguments: {'reduction': 'none'}.\n",
      "[2025-03-26 13:52:40,748 hyrax.models.model_registry:INFO] Using optimizer: torch.optim.Adam with arguments: {'lr': 0.001}.\n",
      "2025-03-26 13:52:40,753 ignite.distributed.auto.auto_dataloader INFO: Use data loader kwargs for dataset '<hyrax.data_sets.hsc': \n",
      "\t{'sampler': <torch.utils.data.sampler.SubsetRandomSampler object at 0x7fa2a01edb80>, 'batch_size': 32, 'pin_memory': True}\n",
      "2025-03-26 13:52:40,761 ignite.distributed.auto.auto_dataloader INFO: Use data loader kwargs for dataset '<hyrax.data_sets.hsc': \n",
      "\t{'sampler': <torch.utils.data.sampler.SubsetRandomSampler object at 0x7fa2681a0d10>, 'batch_size': 32, 'pin_memory': True}\n",
      "2025/03/26 13:52:41 INFO mlflow.system_metrics.system_metrics_monitor: Started monitoring system metrics.\n",
      "[2025-03-26 13:52:41,993 hyrax.pytorch_ignite:INFO] Training model on device: cuda\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "012bfd04c60a4d1cbfdb9b71bcbd846e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-03-26 13:52:54,145 hyrax.data_sets.hsc_data_set:INFO] Processed 992 objects\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4609119b3f1142ea85b28490cfb43bda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "00b419ef16194dcea6583227abb39ba4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "abd7c935a532464981a918e14ba6ff2d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6b7554bcdaf34131a479c3fe234e1383",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6dbd6688211f4793a528bb9c20f2051e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9f0ce1e26cf74483bdace6c6e858fc95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4e175a660133450dbb57ea9187dfb83f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18d430d34eb444a4a2270928584aa9d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17775558184e4d9c85dbfaa8e0e5e6bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a15bf975b3a4981b653463ed871c572",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40ef482d9ccd4520a843e83641fc60e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc1b797e6a294ef6b27bf5e202af200c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87cf04ea44e041a696b910b6805118c8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f9fbf9b41982435882721c4fad12e4a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e457bd70e5f47238d15501ef70fb22e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d51f0aba8b747fd9245539ceeb90507",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61ab857ffd71420db579537b212599fa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c3a237a6e34144aeb6991918a9aa1487",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ddf400d7f92943fcbd7cc6cb8f74db76",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  4%|4         | 1/25 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-03-26 13:53:07,286 hyrax.pytorch_ignite:INFO] Total training time: 25.29[s]\n",
      "[2025-03-26 13:53:07,287 hyrax.pytorch_ignite:INFO] Latest checkpoint saved as: /home/drew/code/fibad/docs/pre_executed/results/20250326-135240-train-idOv/checkpoint_epoch_20.pt\n",
      "[2025-03-26 13:53:07,287 hyrax.pytorch_ignite:INFO] Best metric checkpoint saved as: /home/drew/code/fibad/docs/pre_executed/results/20250326-135240-train-idOv/checkpoint_18_loss=-153.1286.pt\n",
      "2025/03/26 13:53:07 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...\n",
      "2025/03/26 13:53:07 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!\n",
      "[2025-03-26 13:53:07,299 hyrax.train:INFO] Finished Training\n",
      "[2025-03-26 13:53:07,367 hyrax.model_exporters:INFO] Exported model to ONNX format: /home/drew/code/fibad/docs/pre_executed/results/20250326-135240-train-idOv/example_model_opset_20.onnx\n"
     ]
    }
   ],
   "source": [
    "# Specify that we now want to train the model defined in this notebook\n",
    "h.config[\"model\"][\"name\"] = \"TrialAutoencoder\"\n",
    "\n",
    "# Define loss and optimizer functions for easy experimentation\n",
    "h.config[\"criterion\"][\"name\"] = \"torch.nn.MSELoss\"\n",
    "h.config[\"torch.nn.MSELoss\"] = {\"reduction\": \"none\"}\n",
    "\n",
    "h.config[\"optimizer\"][\"name\"] = \"torch.optim.Adam\"\n",
    "h.config[\"torch.optim.Adam\"] = {\"lr\": 1e-3}\n",
    "\n",
    "# train the new model\n",
    "h.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the adjustments that were made to `h.config` before beginning the training.\n",
    "First the model name is updated to be the name of the new model class defined in this notebook.\n",
    "Recall that the `@hyrax_model` decorator will register the model with Hyrax, this is what allows us to refer to the model by the class name.\n",
    "\n",
    "Additionally we defined the loss and optimizer functions in the config to be used by the model.\n",
    "Hyrax supports this feature to allow easy experimentation and evaluation of the performance of these hyperparameters.\n",
    "\n",
    "All of the same actions are being taken as before when we call `h.train()`.\n",
    "Recall that under the hood Hyrax is:\n",
    "- Identifying and using the most performant hardware available to train.\n",
    "- Creating a new timestamped directory where all output is saved.\n",
    "- Freezing the configuration and saving a copy for reproducibility.\n",
    "- Logging model and system metrics for review in both TensorBoard and MLFlow.\n",
    "- Generating checkpoints automatically both at the last epoch and at the epoch with the lowest loss value.\n",
    "- Saving the model weights file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare multiple models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"1000\"\n",
       "            src=\"http://localhost:8080\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fa2a206b740>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Start the MLFlow UI server\n",
    "backend_store_uri = f\"file://{Path(h.config['general']['results_dir']).resolve() / 'mlflow'}\"\n",
    "mlflow_ui_process = subprocess.Popen(\n",
    "    [\"mlflow\", \"ui\", \"--backend-store-uri\", backend_store_uri, \"--port\", \"8080\"],\n",
    "    stdout=subprocess.PIPE,\n",
    "    stderr=subprocess.PIPE,\n",
    ")\n",
    "\n",
    "# Display the MLFlow UI in an IFrame in the notebook\n",
    "IFrame(src=\"http://localhost:8080\", width=\"100%\", height=1000)"
   ]
  },
  {
   "attachments": {
    "mlflow_screenshot.JPG": {
     "image/jpeg": "[encoded data, 164412 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyrax automatically logs training information for model evaluation.\n",
    "Here we see an in-notebook instance of the MLFlow UI.\n",
    "Typically, the UI would be started from the command line and viewed in a browser to avoid having to scroll back and forth in a notebook.\n",
    "The typical command to do so would look like: `mlflow ui --backend-store-uri <results_directory/mlflow>`\n",
    "\n",
    "While MLFlow can run in a notebook, when that notebook is rendered to HTML (for demonstration or documentation) the server backing the MLFlow UI isn't included in the rendering. If the cell above was run locally, the resulting UI would look similar to the following screen shot.\n",
    "\n",
    "![mlflow_screenshot.JPG](attachment:mlflow_screenshot.JPG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:37:46,721 hyrax.data_sets.hsc_data_set:INFO] Checking file dimensions to determine standard cutout size...\n",
      "[2025-04-18 16:37:46,723 hyrax.data_sets.fits_image_dataset:INFO] FitsImageDataSet has 993 objects\n",
      "[2025-04-18 16:37:46,734 hyrax.data_sets.hsc_data_set:INFO] Processed 993 objects for pruning\n",
      "[2025-04-18 16:37:46,734 hyrax.data_sets.fits_image_dataset:INFO] Preloading FitsImageDataSet cache...\n",
      "[2025-04-18 16:37:46,743 hyrax.models.model_registry:INFO] Using criterion: torch.nn.CrossEntropyLoss with default arguments.\n",
      "[2025-04-18 16:37:46,746 hyrax.verbs.infer:INFO] data set has length 993\n",
      "2025-04-18 16:37:46,753 ignite.distributed.auto.auto_dataloader INFO: Use data loader kwargs for dataset '<hyrax.data_sets.hsc': \n",
      "\t{'sampler': None, 'batch_size': 512, 'shuffle': False, 'pin_memory': False}\n",
      "[2025-04-18 16:37:46,832 hyrax.verbs.infer:INFO] Saving inference results at: /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163746-infer-ynj3\n",
      "[2025-04-18 16:37:47,160 hyrax.pytorch_ignite:INFO] Evaluating model on device: mps\n",
      "[2025-04-18 16:37:47,165 hyrax.pytorch_ignite:INFO] Total epochs: 1\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ddf1aeadfbdb48eaa2667e87a3c6f59f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       " 50%|#####     | 1/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:37:50,578 hyrax.data_sets.fits_image_dataset:INFO] Processed 992 objects\n",
      "[2025-04-18 16:37:50,777 hyrax.pytorch_ignite:INFO] Total evaluation time: 3.62[s]\n",
      "[2025-04-18 16:37:50,818 hyrax.verbs.infer:INFO] Inference Complete.\n",
      "[2025-04-18 16:37:50,952 hyrax.config_utils:WARNING] Cannot find default_config.toml for torch.\n",
      "[2025-04-18 16:37:50,952 hyrax.config_utils:WARNING] Cannot find default_config.toml for torch.\n",
      "[2025-04-18 16:37:52,955 hyrax.config_utils:WARNING] Cannot find default_config.toml for umap.\n",
      "[2025-04-18 16:37:52,974 hyrax.data_sets.hsc_data_set:INFO] Checking file dimensions to determine standard cutout size...\n",
      "[2025-04-18 16:37:52,976 hyrax.data_sets.fits_image_dataset:INFO] FitsImageDataSet has 993 objects\n",
      "[2025-04-18 16:37:52,985 hyrax.data_sets.hsc_data_set:INFO] Processed 993 objects for pruning\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<hyrax.data_sets.inference_dataset.InferenceDataSet at 0x169c15660>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Update the data set splits to be 100% test data\n",
    "h.config[\"data_set\"][\"test_size\"] = 1.0\n",
    "h.config[\"data_set\"][\"train_size\"] = 0.0\n",
    "h.config[\"data_set\"][\"validate_size\"] = 0.0\n",
    "\n",
    "# Increase batch size for faster inference\n",
    "h.config[\"data_loader\"][\"batch_size\"] = 512\n",
    "\n",
    "# Run inference\n",
    "h.infer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this demo, we'll pretend that of all the models we trained, the last one performed best.\n",
    "We'll now use that model to run inference.\n",
    "Note that by default, Hyrax will find the weights of the last successfully trained model for inference, but of course, a different set of weights can be specified in the configuration.\n",
    "\n",
    "First we make a small update to the data set splits, setting `test_size` to 100% and the other splits to 0%.\n",
    "We also increase the batch size in order to make better use of the available GPU memory.\n",
    "\n",
    "Finally we run inference over the dataset using the trained model weights with `h.infer()`.\n",
    "As with training, Hyrax is doing a lot behind the scenes on behalf of the user including:\n",
    "- Identifying and using the most performant hardware available.\n",
    "- Creating a new timestamped directory where all output is saved.\n",
    "- Freezing the configuration and saving a copy for reproducibility.\n",
    "- Saving the results of inference in batched .npy files.\n",
    "- Optionally persisting the results to a vector database.\n",
    "\n",
    "Again, while predicting the latent space for only 1000 samples here, Hyrax inference has scaled up to over 1M samples on an HPC system with access to multiple GPUs without requiring any code changes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the results of inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20596 characters]",
      "text/plain": [
       "<Figure size 1200x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Establish a connection to the database containing the inference results\n",
    "results_dir = find_most_recent_results_dir(h.config, \"infer\")\n",
    "client = chromadb.PersistentClient(path=str(results_dir))\n",
    "collection = client.get_collection(\"shard_0\")\n",
    "\n",
    "# For each entry in the database, find the L2 norm distance to the k nearest neighbors\n",
    "all_embeddings = collection.get(include=[\"embeddings\"])\n",
    "all_nn = collection.query(query_embeddings=all_embeddings[\"embeddings\"], n_results=5)\n",
    "\n",
    "# Calculate the median distance to the k nearest neighbor for each entry\n",
    "median_dist_all_nn = np.median(all_nn[\"distances\"], axis=1)\n",
    "\n",
    "# Plot a histogram of the median distances\n",
    "plt.figure(figsize=(12, 6))\n",
    "plt.hist(median_dist_all_nn, bins=100, range=(0, 30_000))\n",
    "plt.xlabel(\"Median distance to k nearest neighbors\")\n",
    "plt.ylabel(\"Count\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With inference complete, we can begin to explore the results.\n",
    "Here we make use of the built in vector database that enables fast, approximate, similarity search.\n",
    "The vector database was populated automatically while running inference.\n",
    "\n",
    "> Aside: What is a \"vector database\"?\n",
    ">\n",
    "> A vector database is one which is optimized to store vectors and maintains an index that can be exploited to provide rapid look up of similar vectors.\n",
    "> The similarity between vectors is defined by the distance between them, where the definition of distance is configurable.\n",
    "> By default, Hyrax uses squared L2 norm.\n",
    "> Thus, given a latent space vector for a given object, the database provides an efficient way to find the k most similar vectors.\n",
    "\n",
    "By making use of our vector database, we have:\n",
    "- Efficiently found the L2 norm distance to each of the k nearest neighbors for every vector produced by inference.\n",
    "- Calculated the median of the distances to the k nearest neighbors.\n",
    "- Plotted the histogram of those median values.\n",
    "\n",
    "There appears to be a long tail in the distribution of values, indicating that there are a small number of objects with latent space vectors that have median L2 norm distances much greater than the average."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine a few objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_object = sort_objects_by_median_distance(\n",
    "    all_embeddings, median_dist_all_nn, data_directory=h.config[\"general\"][\"data_dir\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we order the results of the median calculation and transform the results a bit to make it easier to visualize (gather filename and median distance values for plotting).\n",
    "Next we plot the first and last 16 objects in the sorted list.\n",
    "The first 16 objects are those that correspond to near the peak of the histogram.\n",
    "The last 16 objects are those in the long tail of the histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 720532 characters]",
      "text/plain": [
       "<Figure size 2000x2500 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_grid(sorted_object[:16])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 532380 characters]",
      "text/plain": [
       "<Figure size 2000x2500 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_grid(sorted_object[-16:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examine an embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:38:02,075 hyrax.data_sets.inference_dataset:INFO] Using most recent results dir /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163746-infer-ynj3 for lookup. Use the [results] inference_dir config to set a directory or pass it to this verb.\n",
      "[2025-04-18 16:38:02,107 hyrax.config_utils:WARNING] Cannot find default_config.toml for torch.\n",
      "[2025-04-18 16:38:02,108 hyrax.config_utils:WARNING] Cannot find default_config.toml for torch.\n",
      "[2025-04-18 16:38:02,108 hyrax.config_utils:WARNING] Cannot find default_config.toml for umap.\n",
      "[2025-04-18 16:38:02,130 hyrax.data_sets.hsc_data_set:INFO] Checking file dimensions to determine standard cutout size...\n",
      "[2025-04-18 16:38:02,132 hyrax.data_sets.fits_image_dataset:INFO] FitsImageDataSet has 993 objects\n",
      "[2025-04-18 16:38:02,141 hyrax.data_sets.hsc_data_set:INFO] Processed 993 objects for pruning\n",
      "[2025-04-18 16:38:02,142 hyrax.verbs.umap:INFO] Saving UMAP results to /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163802-umap-BMkM\n",
      "OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16ac76427b794f2fbc7585bd043319fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating lower dimensional representation using UMAP::   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mtauraso/miniforge3/envs/hyrax/lib/python3.10/site-packages/ignite/handlers/checkpoint.py:16: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.\n",
      "  from torch.distributed.optim import ZeroRedundancyOptimizer\n",
      "/Users/mtauraso/miniforge3/envs/hyrax/lib/python3.10/site-packages/ignite/handlers/checkpoint.py:16: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.\n",
      "  from torch.distributed.optim import ZeroRedundancyOptimizer\n",
      "OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.\n",
      "OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 42664 characters]",
      "text/plain": [
       "<Figure size 1200x1200 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h.umap()\n",
    "plot_umap(find_most_recent_results_dir(h.config, \"umap\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we are using the output of the latest inference run to inform a UMAP fitter and then plot the resulting lower dimensional space.\n",
    "With only about 1000 samples, it's hard to visually identify obvious groupings in the 2D space.\n",
    "\n",
    "While fitting and transforming is made fairly straightforward with the `umap-learn` API, additional data plumbing is provided by FIBAD.\n",
    "We ensure that file output of the inference step allows for efficient fitting and transformation with UMAP and all of the data plumbing to read from inference and write results is taken care of.\n",
    "\n",
    "To support further exploration of the embedding space, FIBAD includes an early implementation of an interactive visualization tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactive visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2025-04-18 16:38:18,811 hyrax.data_sets.inference_dataset:INFO] Using most recent results dir /Users/mtauraso/src/fibad/docs/pre_executed/results/20250418-163802-umap-BMkM for lookup. Use the [results] inference_dir config to set a directory or pass it to this verb.\n",
      "[2025-04-18 16:38:18,836 hyrax.config_utils:WARNING] Cannot find default_config.toml for torch.\n",
      "[2025-04-18 16:38:18,836 hyrax.config_utils:WARNING] Cannot find default_config.toml for torch.\n",
      "[2025-04-18 16:38:18,837 hyrax.config_utils:WARNING] Cannot find default_config.toml for umap.\n",
      "[2025-04-18 16:38:18,856 hyrax.data_sets.hsc_data_set:INFO] Checking file dimensions to determine standard cutout size...\n",
      "[2025-04-18 16:38:18,858 hyrax.data_sets.fits_image_dataset:INFO] FitsImageDataSet has 993 objects\n",
      "[2025-04-18 16:38:18,867 hyrax.data_sets.hsc_data_set:INFO] Processed 993 objects for pruning\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<script type=\"esms-options\">{\"shimMode\": true}</script><style>*[data-root-id],\n",
       "*[data-root-id] > * {\n",
       "  box-sizing: border-box;\n",
       "  font-family: var(--jp-ui-font-family);\n",
       "  font-size: var(--jp-ui-font-size1);\n",
       "  color: var(--vscode-editor-foreground, var(--jp-ui-font-color1));\n",
       "}\n",
       "\n",
       "/* Override VSCode background color */\n",
       ".cell-output-ipywidget-background:has(\n",
       "    > .cell-output-ipywidget-background > .lm-Widget > *[data-root-id]\n",
       "  ),\n",
       ".cell-output-ipywidget-background:has(> .lm-Widget > *[data-root-id]) {\n",
       "  background-color: transparent !important;\n",
       "}\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  const force = true;\n  const py_version = '3.6.2'.replace('rc', '-rc.').replace('.dev', '-dev.');\n  const reloading = false;\n  const Bokeh = root.Bokeh;\n\n  // Set a timeout for this load but only if we are not already initializing\n  if (typeof (root._bokeh_timeout) === \"undefined\" || (force || !root._bokeh_is_initializing)) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks;\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n    if (js_modules == null) js_modules = [];\n    if (js_exports == null) js_exports = {};\n\n    root._bokeh_onload_callbacks.push(callback);\n\n    if (root._bokeh_is_loading > 0) {\n      // Don't load bokeh if it is still initializing\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    } else if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n      // There is nothing to load\n      run_callbacks();\n      return null;\n    }\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n    window._bokeh_on_load = on_load\n\n    function on_error(e) {\n      const src_el = e.srcElement\n      console.error(\"failed to load \" + (src_el.href || src_el.src));\n    }\n\n    const skip = [];\n    if (window.requirejs) {\n      window.requirejs.config({'packages': {}, 'paths': {}, 'shim': {}});\n      root._bokeh_is_loading = css_urls.length + 0;\n    } else {\n      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n    }\n\n    const existing_stylesheets = []\n    const links = document.getElementsByTagName('link')\n    for (let i = 0; i < links.length; i++) {\n      const link = links[i]\n      if (link.href != null) {\n        existing_stylesheets.push(link.href)\n      }\n    }\n    for (let i = 0; i < css_urls.length; i++) {\n      const url = css_urls[i];\n      const escaped = encodeURI(url)\n      if (existing_stylesheets.indexOf(escaped) !== -1) {\n        on_load()\n        continue;\n      }\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }    var existing_scripts = []\n    const scripts = document.getElementsByTagName('script')\n    for (let i = 0; i < scripts.length; i++) {\n      var script = scripts[i]\n      if (script.src != null) {\n        existing_scripts.push(script.src)\n      }\n    }\n    for (let i = 0; i < js_urls.length; i++) {\n      const url = js_urls[i];\n      const escaped = encodeURI(url)\n      if (skip.indexOf(escaped) !== -1 || existing_scripts.indexOf(escaped) !== -1) {\n        if (!window.requirejs) {\n          on_load();\n        }\n        continue;\n      }\n      const element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (let i = 0; i < js_modules.length; i++) {\n      const url = js_modules[i];\n      const escaped = encodeURI(url)\n      if (skip.indexOf(escaped) !== -1 || existing_scripts.indexOf(escaped) !== -1) {\n        if (!window.requirejs) {\n          on_load();\n        }\n        continue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (const name in js_exports) {\n      const url = js_exports[name];\n      const escaped = encodeURI(url)\n      if (skip.indexOf(escaped) >= 0 || root[name] != null) {\n        if (!window.requirejs) {\n          on_load();\n        }\n        continue;\n      }\n      var element = document.createElement('script');\n      element.onerror = on_error;\n      element.async = false;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      element.textContent = `\n      import ${name} from \"${url}\"\n      window.${name} = ${name}\n      window._bokeh_on_load()\n      `\n      document.head.appendChild(element);\n    }\n    if (!js_urls.length && !js_modules.length) {\n      on_load()\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  const js_urls = [\"https://cdn.holoviz.org/panel/1.5.4/dist/bundled/reactiveesm/es-module-shims@^1.10.0/dist/es-module-shims.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-3.6.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.6.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.6.2.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.6.2.min.js\", \"https://cdn.holoviz.org/panel/1.5.4/dist/panel.min.js\"];\n  const js_modules = [];\n  const js_exports = {};\n  const css_urls = [];\n  const inline_js = [    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {} // ensure no trailing comma for IE\n  ];\n\n  function run_inline_js() {\n    if ((root.Bokeh !== undefined) || (force === true)) {\n      for (let i = 0; i < inline_js.length; i++) {\n        try {\n          inline_js[i].call(root, root.Bokeh);\n        } catch(e) {\n          if (!reloading) {\n            throw e;\n          }\n        }\n      }\n      // Cache old bokeh versions\n      if (Bokeh != undefined && !reloading) {\n        var NewBokeh = root.Bokeh;\n        if (Bokeh.versions === undefined) {\n          Bokeh.versions = new Map();\n        }\n        if (NewBokeh.version !== Bokeh.version) {\n          Bokeh.versions.set(NewBokeh.version, NewBokeh)\n        }\n        root.Bokeh = Bokeh;\n      }\n    } else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    }\n    root._bokeh_is_initializing = false\n  }\n\n  function load_or_wait() {\n    // Implement a backoff loop that tries to ensure we do not load multiple\n    // versions of Bokeh and its dependencies at the same time.\n    // In recent versions we use the root._bokeh_is_initializing flag\n    // to determine whether there is an ongoing attempt to initialize\n    // bokeh, however for backward compatibility we also try to ensure\n    // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n    // before older versions are fully initialized.\n    if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n      // If the timeout and bokeh was not successfully loaded we reset\n      // everything and try loading again\n      root._bokeh_timeout = Date.now() + 5000;\n      root._bokeh_is_initializing = false;\n      root._bokeh_onload_callbacks = undefined;\n      root._bokeh_is_loading = 0\n      console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n      load_or_wait();\n    } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n      setTimeout(load_or_wait, 100);\n    } else {\n      root._bokeh_is_initializing = true\n      root._bokeh_onload_callbacks = []\n      const bokeh_loaded = root.Bokeh != null && (root.Bokeh.version === py_version || (root.Bokeh.versions !== undefined && root.Bokeh.versions.has(py_version)));\n      if (!reloading && !bokeh_loaded) {\n        if (root.Bokeh) {\n          root.Bokeh = undefined;\n        }\n        console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n      }\n      load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n        console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n        run_inline_js();\n      });\n    }\n  }\n  // Give older versions of the autoload script a head-start to ensure\n  // they initialize before we start loading newer version.\n  setTimeout(load_or_wait, 100)\n}(window));",
      "application/vnd.holoviews_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": "\nif ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n}\n\n\n    function JupyterCommManager() {\n    }\n\n    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        comm_manager.register_target(comm_id, function(comm) {\n          comm.on_msg(msg_handler);\n        });\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n          comm.onMsg = msg_handler;\n        });\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n          var messages = comm.messages[Symbol.asyncIterator]();\n          function processIteratorResult(result) {\n            var message = result.value;\n            console.log(message)\n            var content = {data: message.data, comm_id};\n            var buffers = []\n            for (var buffer of message.buffers || []) {\n              buffers.push(new DataView(buffer))\n            }\n            var metadata = message.metadata || {};\n            var msg = {content, buffers, metadata}\n            msg_handler(msg);\n            return messages.next().then(processIteratorResult);\n          }\n          return messages.next().then(processIteratorResult);\n        })\n      }\n    }\n\n    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n      if (comm_id in window.PyViz.comms) {\n        return window.PyViz.comms[comm_id];\n      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n        if (msg_handler) {\n          comm.on_msg(msg_handler);\n        }\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n        comm.open();\n        if (msg_handler) {\n          comm.onMsg = msg_handler;\n        }\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        var comm_promise = google.colab.kernel.comms.open(comm_id)\n        comm_promise.then((comm) => {\n          window.PyViz.comms[comm_id] = comm;\n          if (msg_handler) {\n            var messages = comm.messages[Symbol.asyncIterator]();\n            function processIteratorResult(result) {\n              var message = result.value;\n              var content = {data: message.data};\n              var metadata = message.metadata || {comm_id};\n              var msg = {content, metadata}\n              msg_handler(msg);\n              return messages.next().then(processIteratorResult);\n            }\n            return messages.next().then(processIteratorResult);\n          }\n        }) \n        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n          return comm_promise.then((comm) => {\n            comm.send(data, metadata, buffers, disposeOnDone);\n          });\n        };\n        var comm = {\n          send: sendClosure\n        };\n      }\n      window.PyViz.comms[comm_id] = comm;\n      return comm;\n    }\n    window.PyViz.comm_manager = new JupyterCommManager();\n    \n\n\nvar JS_MIME_TYPE = 'application/javascript';\nvar HTML_MIME_TYPE = 'text/html';\nvar EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\nvar CLASS_NAME = 'output';\n\n/**\n * Render data to the DOM node\n */\nfunction render(props, node) {\n  var div = document.createElement(\"div\");\n  var script = document.createElement(\"script\");\n  node.appendChild(div);\n  node.appendChild(script);\n}\n\n/**\n * Handle when a new output is added\n */\nfunction handle_add_output(event, handle) {\n  var output_area = handle.output_area;\n  var output = handle.output;\n  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n    return\n  }\n  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n  if (id !== undefined) {\n    var nchildren = toinsert.length;\n    var html_node = toinsert[nchildren-1].children[0];\n    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n    var scripts = [];\n    var nodelist = html_node.querySelectorAll(\"script\");\n    for (var i in nodelist) {\n      if (nodelist.hasOwnProperty(i)) {\n        scripts.push(nodelist[i])\n      }\n    }\n\n    scripts.forEach( function (oldScript) {\n      var newScript = document.createElement(\"script\");\n      var attrs = [];\n      var nodemap = oldScript.attributes;\n      for (var j in nodemap) {\n        if (nodemap.hasOwnProperty(j)) {\n          attrs.push(nodemap[j])\n        }\n      }\n      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n      oldScript.parentNode.replaceChild(newScript, oldScript);\n    });\n    if (JS_MIME_TYPE in output.data) {\n      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n    }\n    output_area._hv_plot_id = id;\n    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n      window.PyViz.plot_index[id] = Bokeh.index[id];\n    } else {\n      window.PyViz.plot_index[id] = null;\n    }\n  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n    var bk_div = document.createElement(\"div\");\n    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n    var script_attrs = bk_div.children[0].attributes;\n    for (var i = 0; i < script_attrs.length; i++) {\n      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n    }\n    // store reference to server id on output_area\n    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n  }\n}\n\n/**\n * Handle when an output is cleared or removed\n */\nfunction handle_clear_output(event, handle) {\n  var id = handle.cell.output_area._hv_plot_id;\n  var server_id = handle.cell.output_area._bokeh_server_id;\n  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n  if (server_id !== null) {\n    comm.send({event_type: 'server_delete', 'id': server_id});\n    return;\n  } else if (comm !== null) {\n    comm.send({event_type: 'delete', 'id': id});\n  }\n  delete PyViz.plot_index[id];\n  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n    var doc = window.Bokeh.index[id].model.document\n    doc.clear();\n    const i = window.Bokeh.documents.indexOf(doc);\n    if (i > -1) {\n      window.Bokeh.documents.splice(i, 1);\n    }\n  }\n}\n\n/**\n * Handle kernel restart event\n */\nfunction handle_kernel_cleanup(event, handle) {\n  delete PyViz.comms[\"hv-extension-comm\"];\n  window.PyViz.plot_index = {}\n}\n\n/**\n * Handle update_display_data messages\n */\nfunction handle_update_output(event, handle) {\n  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n  handle_add_output(event, handle)\n}\n\nfunction register_renderer(events, OutputArea) {\n  function append_mime(data, metadata, element) {\n    // create a DOM node to render to\n    var toinsert = this.create_output_subarea(\n    metadata,\n    CLASS_NAME,\n    EXEC_MIME_TYPE\n    );\n    this.keyboard_manager.register_events(toinsert);\n    // Render to node\n    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n    render(props, toinsert[0]);\n    element.append(toinsert);\n    return toinsert\n  }\n\n  events.on('output_added.OutputArea', handle_add_output);\n  events.on('output_updated.OutputArea', handle_update_output);\n  events.on('clear_output.CodeCell', handle_clear_output);\n  events.on('delete.Cell', handle_clear_output);\n  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n\n  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n    safe: true,\n    index: 0\n  });\n}\n\nif (window.Jupyter !== undefined) {\n  try {\n    var events = require('base/js/events');\n    var OutputArea = require('notebook/js/outputarea').OutputArea;\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  } catch(err) {\n  }\n}\n",
      "application/vnd.holoviews_load.v0+json": ""
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.holoviews_exec.v0+json": "",
      "text/html": [
       "<div id='049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9'>\n",
       "  <div id=\"ec2672a7-1134-422d-8713-104445b02604\" data-root-id=\"049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9\" style=\"display: contents;\"></div>\n",
       "</div>\n",
       "<script type=\"application/javascript\">(function(root) {\n",
       "  var docs_json = {\"52b33356-2971-48b2-b638-eb7f6293693f\":{\"version\":\"3.6.2\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"panel.models.browser.BrowserInfo\",\"id\":\"049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9\"},{\"type\":\"object\",\"name\":\"panel.models.comm_manager.CommManager\",\"id\":\"12f856ce-3cf1-4802-9196-683cea6200b1\",\"attributes\":{\"plot_id\":\"049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9\",\"comm_id\":\"4f380d77dc0941d48d1a0b30bd10510a\",\"client_comm_id\":\"c85fce9e87294a86a865d28b7875e886\"}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"gap\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"TemplateEditor1\",\"properties\":[{\"name\":\"layout\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"ReactiveESM1\",\"properties\":[{\"name\":\"esm_constants\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}}]},{\"type\":\"model\",\"name\":\"JSComponent1\",\"properties\":[{\"name\":\"esm_constants\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}}]},{\"type\":\"model\",\"name\":\"ReactComponent1\",\"properties\":[{\"name\":\"esm_constants\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}}]},{\"type\":\"model\",\"name\":\"AnyWidgetComponent1\",\"properties\":[{\"name\":\"esm_constants\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}}]},{\"type\":\"model\",\"name\":\"request_value1\",\"properties\":[{\"name\":\"fill\",\"kind\":\"Any\",\"default\":\"none\"},{\"name\":\"_synced\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_request_sync\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
       "  var render_items = [{\"docid\":\"52b33356-2971-48b2-b638-eb7f6293693f\",\"roots\":{\"049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9\":\"ec2672a7-1134-422d-8713-104445b02604\"},\"root_ids\":[\"049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9\"]}];\n",
       "  var docs = Object.values(docs_json)\n",
       "  if (!docs) {\n",
       "    return\n",
       "  }\n",
       "  const py_version = docs[0].version.replace('rc', '-rc.').replace('.dev', '-dev.')\n",
       "  async function embed_document(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    await Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "    for (const render_item of render_items) {\n",
       "      for (const root_id of render_item.root_ids) {\n",
       "\tconst id_el = document.getElementById(root_id)\n",
       "\tif (id_el.children.length && id_el.children[0].hasAttribute('data-root-id')) {\n",
       "\t  const root_el = id_el.children[0]\n",
       "\t  root_el.id = root_el.id + '-rendered'\n",
       "\t  for (const child of root_el.children) {\n",
       "            // Ensure JupyterLab does not capture keyboard shortcuts\n",
       "            // see: https://jupyterlab.readthedocs.io/en/4.1.x/extension/notebook.html#keyboard-interaction-model\n",
       "\t    child.setAttribute('data-lm-suppress-shortcuts', 'true')\n",
       "\t  }\n",
       "\t}\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "  function get_bokeh(root) {\n",
       "    if (root.Bokeh === undefined) {\n",
       "      return null\n",
       "    } else if (root.Bokeh.version !== py_version) {\n",
       "      if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
       "\treturn null\n",
       "      }\n",
       "      return root.Bokeh.versions.get(py_version);\n",
       "    } else if (root.Bokeh.version === py_version) {\n",
       "      return root.Bokeh\n",
       "    }\n",
       "    return null\n",
       "  }\n",
       "  function is_loaded(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    return (Bokeh != null && Bokeh.Panel !== undefined)\n",
       "  }\n",
       "  if (is_loaded(root)) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (is_loaded(root)) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else if (document.readyState == \"complete\") {\n",
       "        attempts++;\n",
       "        if (attempts > 200) {\n",
       "          clearInterval(timer);\n",
       "\t  var Bokeh = get_bokeh(root)\n",
       "\t  if (Bokeh == null || Bokeh.Panel == null) {\n",
       "            console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
       "\t  } else {\n",
       "\t    console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
       "\t    embed_document(root)\n",
       "\t  }\n",
       "        }\n",
       "      }\n",
       "    }, 25, root)\n",
       "  }\n",
       "})(window);</script>"
      ]
     },
     "metadata": {
      "application/vnd.holoviews_exec.v0+json": {
       "id": "049a21ec-0ac5-43e1-ad37-ed8ed85bc1f9"
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<div class=\"logo-block\">\n",
       "<img src='data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEAAAABACAYAAACqaXHeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAAB+wAAAfsBxc2miwAAABl0RVh0U29mdHdhcmUAd3d3Lmlua3NjYXBlLm9yZ5vuPBoAAA6zSURB\n",
       "VHic7ZtpeFRVmsf/5966taWqUlUJ2UioBBJiIBAwCZtog9IOgjqACsogKtqirT2ttt069nQ/zDzt\n",
       "tI4+CrJIREFaFgWhBXpUNhHZQoKBkIUASchWla1S+3ar7r1nPkDaCAnZKoQP/D7mnPOe9/xy76n3\n",
       "nFSAW9ziFoPFNED2LLK5wcyBDObkb8ZkxuaoSYlI6ZcOKq1eWFdedqNzGHQBk9RMEwFAASkk0Xw3\n",
       "ETacDNi2vtvc7L0ROdw0AjoSotQVkKSvHQz/wRO1lScGModBFbDMaNRN1A4tUBCS3lk7BWhQkgpD\n",
       "lG4852/+7DWr1R3uHAZVQDsbh6ZPN7CyxUrCzJMRouusj0ipRwD2uKm0Zn5d2dFwzX1TCGhnmdGo\n",
       "G62Nna+isiUqhkzuKrkQaJlPEv5mFl2fvGg2t/VnzkEV8F5ioioOEWkLG86fvbpthynjdhXYZziQ\n",
       "x1hC9J2NFyi8vCTt91Fh04KGip0AaG9zuCk2wQCVyoNU3Hjezee9bq92duzzTmxsRJoy+jEZZZYo\n",
       "GTKJ6SJngdJqAfRzpze0+jHreUtPc7gpBLQnIYK6BYp/uGhw9YK688eu7v95ysgshcg9qSLMo3JC\n",
       "4jqLKQFBgdKDPoQ+Pltb8dUyQLpeDjeVgI6EgLIQFT5tEl3rn2losHVsexbZ3EyT9wE1uGdkIPcy\n",
       "BGxn8QUq1QrA5nqW5i2tLqvrrM9NK6AdkVIvL9E9bZL/oyfMVd/jqvc8LylzRBKDJSzIExwhQzuL\n",
       "QYGQj4rHfFTc8mUdu3E7yoLtbTe9gI4EqVgVkug2i5+uXGo919ixbRog+3fTbQ8qJe4ZOYNfMoTI\n",
       "OoshUNosgO60AisX15aeI2PSIp5KiFLI9ubb1vV3Qb2ltwLakUCDAkWX7/nHKRmmGIl9VgYsUhJm\n",
       "2NXjKYADtM1ygne9QQDIXlk49FBstMKx66D1v4+XuQr7vqTe0VcBHQlRWiOCbmmSYe2SqtL6q5rJ\n",
       "zsTb7lKx3FKOYC4DoqyS/B5bvLPxvD9Qtf6saxYLQGJErmDOdOMr/zo96km1nElr8bmPOBwI9COv\n",
       "HnFPRIwmkSOv9kcAS4heRsidOkpeWBgZM+UBrTFAXNYL5Vf2ii9c1trNzpYdaoVil3WIc+wdk+gQ\n",
       "noie3ecCcxt9ITcLAPWt/laGEO/9U6PmzZkenTtsSMQ8uYywJVW+grCstAvCIaAdArAsIWkRDDs/\n",
       "KzLm2YcjY1Lv0UdW73HabE9n6V66cxSzfEmuJssTpKGVp+0vHq73FwL46eOjpMpbRAnNmJFrGJNu\n",
       "Ukf9Yrz+3rghiumCKNXXWPhLYcjxGsIpoCMsIRoFITkW8AuyM8jC1+/QLx4bozCEJIq38+1rtpR6\n",
       "V/yzb8eBlRb3fo5l783N0CWolAzJHaVNzkrTzlEp2bQ2q3TC5gn6wpnoQAmwSiGh2GitnTmVMc5O\n",
       "UyfKWUKCIsU7+fZDKwqdT6DDpvkzAX4/+AMFjk0tDp5GRXLpQ2MUmhgDp5gxQT8+Y7hyPsMi8uxF\n",
       "71H0oebujHALECjFKaW9Lm68n18wXp2kVzIcABytD5iXFzg+WVXkegpAsOOYziqo0OkK76GyquC3\n",
       "ltZAzMhhqlSNmmWTE5T6e3IN05ITFLM4GdN0vtZ3ob8Jh1NAKXFbm5PtLU/eqTSlGjkNAJjdgn/N\n",
       "aedXa0tdi7+t9G0FIF49rtMSEgAs1kDLkTPO7ebm4IUWeyh1bKomXqlgMG6kJmHcSM0clYLJ8XtR\n",
       "1GTnbV3F6I5wCGikAb402npp1h1s7LQUZZSMIfALFOuL3UUrfnS8+rez7v9qcold5tilgHbO1fjK\n",
       "9ubb17u9oshxzMiUBKXWqJNxd+fqb0tLVs4lILFnK71H0Ind7uiPgACVcFJlrb0tV6DzxqqTIhUM\n",
       "CwDf1/rrVhTa33/3pGPxJYdQ2l2cbgVcQSosdx8uqnDtbGjh9SlDVSMNWhlnilfqZk42Th2ZpLpf\n",
       "xrHec5e815zrr0dfBZSwzkZfqsv+1FS1KUknUwPARVvItfKUY+cn57yP7qv07UE3p8B2uhUwLk09\n",
       "e0SCOrK+hbdYHYLjRIl71wWzv9jpEoeOHhGRrJAzyEyNiJuUqX0g2sBN5kGK6y2Blp5M3lsB9Qh4\n",
       "y2Ja6x6+i0ucmKgwMATwhSjdUu49tKrQ/pvN5d53ml2CGwCmJipmKjgmyuaXzNeL2a0AkQ01Th5j\n",
       "2DktO3Jyk8f9vcOBQHV94OK+fPumJmvQHxJoWkaKWq9Vs+yUsbq0zGT1I4RgeH2b5wef7+c7bl8F\n",
       "eKgoHVVZa8ZPEORzR6sT1BzDUAD/d9F78e2Tzv99v8D+fLVTqAKAsbGamKey1Mt9Ann4eH3gTXTz\n",
       "idWtAJ8PQWOk7NzSeQn/OTHDuEikVF1R4z8BQCy+6D1aWRfY0tTGG2OM8rRoPaeIj5ZHzJxszElN\n",
       "VM8K8JS5WOfv8mzRnQAKoEhmt8gyPM4lU9SmBK1MCQBnW4KONT86v1hZ1PbwSXPw4JWussVjtH9Y\n",
       "NCoiL9UoH/6PSu8jFrfY2t36erQHXLIEakMi1SydmzB31h3GGXFDFNPaK8Rme9B79Ixrd0WN+1ij\n",
       "NRQ/doRmuFLBkHSTOm5GruG+pFjFdAmorG4IXH1Qua6ASniclfFtDYt+oUjKipPrCQB7QBQ2lrgP\n",
       "fFzm+9XWUtcqJ3/5vDLDpJ79XHZk3u8nGZ42qlj1+ydtbxysCezrydp6ugmipNJ7WBPB5tydY0jP\n",
       "HaVNzs3QzeE4ZpTbI+ZbnSFPbVOw9vsfnVvqWnirPyCNGD08IlqtYkh2hjZ5dErEQzoNm+6ykyOt\n",
       "Lt5/PQEuSRRKo22VkydK+vvS1XEKlhCJAnsqvcVvH7f/ZU2R67eXbMEGAMiIV5oWZWiWvz5Fv2xG\n",
       "sjqNJQRvn3Rs2lji/lNP19VjAQDgD7FHhujZB9OGqYxRkZxixgRDVlqS6uEOFaJUVu0rPFzctrnF\n",
       "JqijImVp8dEKVWyUXDk92zAuMZ6bFwpBU1HrOw6AdhQgUooChb0+ItMbWJitSo5Ws3IAOGEOtL53\n",
       "0vHZih9sC4vtofZ7Qu6523V/fmGcds1TY3V36pUsBwAbSlxnVh2xLfAD/IAIMDf7XYIkNmXfpp2l\n",
       "18rkAJAy9HKFaIr/qULkeQQKy9zf1JgDB2uaeFNGijo5QsUyacNUUTOnGO42xSnv4oOwpDi1zYkc\n",
       "efUc3I5Gk6PhyTuVKaOGyLUAYPGIoY9Pu/atL/L92+4q9wbflRJ2Trpm/jPjdBtfnqB/dIThcl8A\n",
       "KG7hbRuKnb8qsQsVvVlTrwQAQMUlf3kwJI24Z4JhPMtcfng5GcH49GsrxJpGvvHIaeem2ma+KSjQ\n",
       "lIwUdYyCY8j4dE1KzijNnIP2llF2wcXNnsoapw9XxsgYAl6k+KzUXbi2yP3KR2ecf6z3BFsBICdW\n",
       "nvnIaG3eHybqX7vbpEqUMT+9OL4Qpe8VON7dXuFd39v19FoAABRVePbGGuXTszO0P7tu6lghUonE\n",
       "llRdrhArLvmKdh9u29jcFiRRkfLUxBiFNiqSU9icoZQHo5mYBI1MBgBH6wMNb+U7Pnw337H4gi1Y\n",
       "ciWs+uks3Z9fztUvfzxTm9Ne8XXkvQLHNytOOZeiD4e0PgkAIAYCYknKUNUDSXEKzdWNpnil7r4p\n",
       "xqkjTarZMtk/K8TQ6Qve78qqvXurGwIJqcOUKfUWHsm8KGvxSP68YudXq4pcj39X49uOK2X142O0\n",
       "Tz5/u/7TVybqH0rSya6ZBwD21/gubbrgWdDgEOx9WUhfBaC2ibcEBYm7a7x+ukrBMNcEZggyR0TE\n",
       "T8zUPjikQ4VosQZbTpS4vqizBKvqmvjsqnpfzaZyx9JPiz1/bfGKdgD45XB1zoIMzYbfTdS/NClB\n",
       "Gct0USiY3YL/g0LHy/uq/Ef6uo5+n0R/vyhp17Klpge763f8rMu6YU/zrn2nml+2WtH+Z+5IAAFc\n",
       "2bUTdTDOSNa9+cQY7YLsOIXhevEkCvzph7a8laecz/Un/z4/Ae04XeL3UQb57IwU9ZDr9UuKVajv\n",
       "nxp1+1UVIo/LjztZkKH59fO3G/JemqCfmaCRqbqbd90ZZ8FfjtkfAyD0J/9+C2h1hDwsSxvGjNDc\n",
       "b4zk5NfrSwiQblLHzZhg+Jf4aPlUwpDqkQqa9nimbt1/TDH8OitGMaQnj+RJS6B1fbF7SY1TqO5v\n",
       "/v0WAADl1f7zokgS7s7VT2DZ7pegUjBM7mjtiDZbcN4j0YrHH0rXpCtY0qPX0cVL0rv5jv/ZXend\n",
       "0u/EESYBAFBU4T4Qa5TflZOhTe7pmKpaP8kCVUVw1+yhXfJWvn1P3hnXi33JsTN6PnP3hHZ8Z3/h\n",
       "aLHzmkNPuPj7Bc/F/Q38CwjTpSwQXgE4Vmwry9tpfq/ZFgqFMy4AVDtCvi8rvMvOmv0N4YwbVgEA\n",
       "sPM72/KVnzfspmH7HQGCRLG2yL1+z8XwvPcdCbsAANh+xPzstgMtxeGKt+6MK3/tacfvwhWvIwMi\n",
       "oKEBtm0H7W+UVfkc/Y1V0BhoPlDr/w1w/eu1vjIgAgDg22OtX6/eYfnEz/focrZTHAFR+PSs56/7\n",
       "q32nwpjazxgwAQCwcU/T62t3WL7r6/jVRa6/byp1rei+Z98ZUAEAhEPHPc8fKnTU9nbgtnOe8h0l\n",
       "9hcGIqmODLQAHCy2Xti6v/XNRivf43f4fFvIteu854+VHnR7q9tfBlwAAGz+pnndB9vM26UebAe8\n",
       "SLHujPOTPVW+rwY+sxskAAC2HrA8t2Vvc7ffP1r9o+vwR2dcr92InIAbKKC1FZ5tB1tf+/G8p8sv\n",
       "N/9Q5zd/XR34LYCwV5JdccMEAMDBk45DH243r/X4xGvqxFa/GNpS7n6rwOwNWwHVE26oAADYurf1\n",
       "zx/utOzt+DMKYM0p17YtZZ5VNzqfsB2HewG1WXE8PoZ7gOclbTIvynZf9JV+fqZtfgs/8F/Nu5rB\n",
       "EIBmJ+8QRMmpU7EzGRsf2FzuePqYRbzh/zE26EwdrT10f6r6o8HOYzCJB9Dpff8tbnGLG8L/A/WE\n",
       "roTBs2RqAAAAAElFTkSuQmCC'\n",
       "     style='height:25px; border-radius:12px; display: inline-block; float: left; vertical-align: middle'></img>\n",
       "\n",
       "\n",
       "  <img src='data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACMAAAAjCAYAAAAe2bNZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAK6wAACusBgosNWgAAABx0RVh0U29mdHdhcmUAQWRvYmUgRmlyZXdvcmtzIENTNui8sowAAAf9SURBVFiFvZh7cFTVHcc/59y7793sJiFAwkvAYDRqFWwdraLVlj61diRYsDjqCFbFKrYo0CltlSq1tLaC2GprGIriGwqjFu10OlrGv8RiK/IICYECSWBDkt3s695zTv9IAtlHeOn0O7Mzu797z+/3Ob/z+p0VfBq9doNFljuABwAXw2PcvGHt6bgwxhz7Ls4YZNVXxxANLENwE2D1W9PAGmAhszZ0/X9gll5yCbHoOirLzmaQs0F6F8QMZq1v/8xgNm7DYwwjgXJLYL4witQ16+sv/U9HdDmV4WrKw6B06cZC/RMrM4MZ7xz61DAbtzEXmAvUAX4pMOVecg9/MFFu3j3Gz7gQBLygS2RGumBkL0cubiFRsR3LzVBV1UMk3IrW73PT9C2lYOwhQB4ClhX1AuKpjLcV27oEjyUpNUJCg1CvcejykWTCXyQgzic2HIIBjg3pS6+uRLKAhumZvD4U+tq0jTrgkVKQQtLekfTtxIPAkhTNF6G7kZm7aPp6M9myKVQEoaYaIhEQYvD781DML/RfBGNZXAl4irJiwBa07e/y7cQnBaJghIX6ENl2GR/fGCBoz6cm5qeyEqQA5ZYA5x5eeiV0Qph4gjFAUSwAr6QllQgcxS/Jm25Cr2Tmpsk03XI9NfI31FTZBEOgVOk51adqDBNPCNPSRlkiDXbBEwOU2WxH+I7itQZ62g56OjM33suq1YsZHVtGZSUI2QdyYgkgOthQNIF7BIGDnRAJgJSgj69cUx1gB8PkOGwL4E1gPrM27gIg7NlGKLQApc7BmEnAxP5g/rw4YqBrCDB5xHkw5rdR/1qTrN/hKNo6YUwVDNpFsnjYS8RbidBPcPXFP6R6yfExuOXmN4A3jv1+8ZUwgY9D2OWjUZE6lO88jDwHI8ZixGiMKSeYTBamCoDk6kDAb6y1OcH1a6KpD/fZesoFw5FlIXAVCIiH4PxrV+p2npVDToTBmtjY8t1swh2V61E9KqWiyuPEjM8dbfxuvfa49Zayf9R136Wr8mBSf/T7bNteA8zwaGEUbFpckWwq95n59dUIywKl2fbOIS5e8bWSu0tJ1a5redAYfqkdjesodFajcgaVNWhXo1C9SrkN3Usmv3UMJrc6/DDwkwEntkEJLe67tSLhvyzK8rHDQWleve5CGk4VZEB1r+5bg2E2si+Y0QatDK6jUVkX5eg2YYlp++ZM+rfMNYamAj8Y7MAVWFqaR1f/t2xzU4IHjybBtthzuiAASqv7jTF7jOqDMAakFHgDNsFyP+FhwZHBmH9F7cutIYkQCylYYv1AZSqsn1/+bX51OMMjPSl2nAnM7hnjOx2v53YgNWAzHM9Q/9l0lQWPSCBSyokAtOBC1Rj+w/1Xs+STDp4/E5g7Rs2zm2+oeVd7PUuHKDf6A4r5EsPT5K3gfCnBXNUYnvGzb+KcCczYYWOnLpy4eOXuG2oec0PBN8XQQAnpvS35AvAykr56rWhPBiV4MvtceGLxk5Mr6A1O8IfK7rl7xJ0r9kyumuP4fa0lMqTBLJIAJqEf1J3qE92lMBndlyfRD2YBghHC4hlny7ASqCeWo5zaoDdIWfnIefNGTb9fC73QDfhyBUCNOxrGPSUBfPem9us253YTV+3mcBbdkUYfzmHiLqZbYdIGHHON2ZlemXouaJUOO6TqtdHEQuXYY8Yt+EbDgmlS6RdzkaDTv2P9A3gICiq93sWhb5mc5wVhuU3Y7m5hOc3So7qFT3SLgOXHb/cyOfMn7xROegoC/PTcn3v8gbKPgDopJFk3R/uBPWQiwQ+2/GJevRMObLUzqe/saJjQUQTTftEVMW9tWxPgAocwcj9abNcZe7s+6t2R2xXZG7zyYLp8Q1PiRBBHym5bYuXi8Qt+/LvGu9f/5YDAxABsaRNPH6Xr4D4Sk87a897SOy9v/fKwjoF2eQel95yDESGEF6gEMwKhLwKus3wOVjTtes7qzgLdXTMnNCNoEpbcrtNuq6N7Xh/+eqcbj94xQkp7mdKpW5XbtbR8Z26kgMCAf2UU5YEovRUVRHbu2b3vK1UdDFkDCyMRQxbpdv8nhKAGIa7QaQedzT07fFPny53R738JoVYBdVrnsNx9XZ9v33UeGO+AA2MMUkgqQ5UcdDLZSFeVgONnXeHqSAC5Ew1BXwko0D1Zct3dT1duOjS3MzZnEUJtBuoQAq3SGOLR4ekjn9NC5nVOaYXf9lETrUkmOJy3pOz8OKIb2A1cWhJCCEzOxU2mUPror+2/L3yyM3pkM7jTjr1nBOgkGeyQ7erxpdJsMAS9wb2F9rzMxNY1K2PMU0WtZV82VU8Wp6vbKJVo9Lx/+4cydORdxCCQ/kDGTZCWsRpLu7VD7bfKqL8V2orKTp/PtzaXy42jr6TwAuisi+7JolUG4wY+8vyrISCMtRrLKWpvjAOqx/QGhp0rjRo5xD3x98CWQuOQN8qumRMmI7jKZPUEpzNVZsj4Zbaq1to5tZZsKIydLWojhIXrJnES79EaOzv3du2NytKuxzJKAA6wF8xqEE8s2jo/1wd/khslQGxd81Zg62Bbp31XBH+iETt7Y3ELA0iU6iGDlQ5mexe0VEx4a3x8V1AaYwFJgTiwaOsDmeK2J8nMUOqsnB1A+dcA04ucCYt0urkjmflk9iT2v30q/gZn5rQPvor4n9Ou634PeBzoznes/iot/7WnClKoM/+zCIjH5kwT8ChQjTHPIPTjFV3PpU/Hx+DM/A9U3IXI4SPCYAAAAABJRU5ErkJggg=='\n",
       "       style='height:15px; border-radius:12px; display: inline-block; float: left'></img>\n",
       "  \n",
       "\n",
       "\n",
       "\n",
       "\n",
       "</div>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c2c20420047499983e8b450a0736e99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BokehModel(combine_events=True, render_bundle={'docs_json': {'0417ab3c-8701-4fa6-8088-449d6c92a042': {'version…"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h.config[\"visualize\"][\"fields\"] = [\"ra\", \"dec\"]\n",
    "h.visualize(width=1000, height=1000)"
   ]
  },
  {
   "attachments": {
    "umap_visualization.JPG": {
     "image/jpeg": "[encoded data, 213660 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Hyrax visualization tooling utilizes Holoviews, Datashader as well as an efficient tree structure to be able to display millions of points.\n",
    "It allows for panning, zooming as well as lasso and box selections.\n",
    "When selecting points, the resulting object ids are displayed are displayed in the associated table.\n",
    "\n",
    "While this is an early version of interactive visualization, it has been scaled up to millions of data points.\n",
    "The next steps for this tooling will be to support deeper interactivity, namely:\n",
    "-  Automatically displaying the object selected in the table\n",
    "-  Leveraging the vector db to identify similar objects\n",
    "-  Supporting three dimensional UMAP output\n",
    "\n",
    "\n",
    "This visualization runs in a notebook but when rendered to HTML (for demonstration or documentation) the server backing the interactive visual isn't packaged with the rendering. If the cell above was run locally, the resulting UI would look similar to the following screen shot.\n",
    "\n",
    "![umap_visualization.JPG](attachment:umap_visualization.JPG)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlflow_ui_process.terminate()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fibad",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "nbsphinx": {
   "orphan": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
