{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8389ad39",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import pandas as pd\n",
    "import shutil\n",
    "from seeq import spy\n",
    "\n",
    "# Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script\n",
    "spy.options.compatibility = 193"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "450f1f4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Logged in to <strong>http://localhost:34216</strong> as <strong>agent_api_key</strong>.<br>Seeq Server Version: <strong>R65.0.0-SNAPSHOT</strong><br>Seeq SDK Module Version: <strong>65.0.0</strong> @ C:\\dev\\develop\\sdk\\pypi\\seeq\\sdk<br>Seeq SPy Module Version: <strong>191.1</strong> @ C:\\dev\\develop\\sdk\\pypi\\seeq\\spy</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Log into Seeq Server if you're not using Seeq Data Lab:\n",
    "spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "594cf76f",
   "metadata": {},
   "source": [
    "# Workbook Jobs\n",
    "\n",
    "In [spy.workbooks.ipynb](spy.workbooks.ipynb), you can learn to push and pull workbooks (Workbench Analyses and Organizer Topics) to/from the Seeq service/server using SPy.\n",
    "\n",
    "You may need to do something \"in bulk,\" in one of the following scenarios:\n",
    "\n",
    "- Re-mapping references (e.g. historian tags/signals) from one datasource to another, or one asset tree to another\n",
    "- Transferring work from one Seeq service/server to another, possibly including data\n",
    "\n",
    "The set of functions in the `spy.workbooks.job` module are suitable for this work. Each function operates within a \"job folder\" that captures the state of the job. Unlike `spy.workbooks.pull()` and `spy.workbooks.push()`, the equivalent commands in `spy.workbooks.job` do not require all workbooks to be held in memory. This allows very large jobs to be executed (as long as there is sufficient disk space). All parts of the process are _resumable_, and SPy will pick up where it left off if the operation is interrupted for any reason (e.g. a network error).\n",
    "\n",
    "This notebook will walk through the use of this module, referencing the scenarios above. In general, commands are executed in the following order:\n",
    "\n",
    "1. `spy.workbooks.job.pull()`\n",
    "2. `spy.workbooks.job.data.pull()` (optional)\n",
    "3. `spy.workbooks.job.push()`\n",
    "4. `spy.workbooks.job.data.push()` (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960e90ac",
   "metadata": {},
   "source": [
    "## Establish the Job Folder\n",
    "\n",
    "The parameter that defines a job is a _job folder_. It is the first argument for all job functions, and it is managed entirely by SPy. The folder is laid out in an intuitive way that allows you to inspect it, and, in some troubleshooting cases, make modifications yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dd4dd9bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "job_folder = 'Output/My First Workbooks Job'\n",
    "\n",
    "# Remove the job folder so that old file/artifacts don't affect the tutorial\n",
    "if os.path.exists(job_folder):\n",
    "    shutil.rmtree(job_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c48f1a6",
   "metadata": {},
   "source": [
    "## Let's Make Something to Work With...\n",
    "\n",
    "We need some Analyses/Topics to work with for the purposes of demonstrating the functionality, so let's make sure the example workbooks have been pushed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aa290d62",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Push successful</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: left;\">Type</td><td style=\"background-color: #EEFFEE; text-align: left;\">Workbook Type</td><td style=\"background-color: #EEFFEE; text-align: left;\">Count</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: left;\">Errors</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td><td style=\"background-color: #EEFFEE; text-align: left;\">Pushed Workbook ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">URL</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">D833DC83-9A38-48DE-BF45-EB787E9E8375</td><td style=\"text-align: left; vertical-align: top;\">Example Analysis</td><td style=\"text-align: left; vertical-align: top;\">Workbook</td><td style=\"text-align: left; vertical-align: top;\">Analysis</td><td style=\"text-align: right; vertical-align: top;\">53</td><td style=\"vertical-align: top;\">00:00:02.56</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-AF85-7580-8F20-D91393026BA1</td><td style=\"text-align: left; vertical-align: top;\"><a target=\"_blank\" href=\"http://localhost:34216/0EECC4B1-AF3C-71A0-A425-D0461BC34EDB/workbook/0EECC4B1-AF85-7580-8F20-D91393026BA1/worksheet/0EECC4B1-BD71-ECC0-AED5-ED45BE2F739E\">link</a></td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">811B1488-297A-4FD2-AE7C-A1FE0E3B3641</td><td style=\"text-align: left; vertical-align: top;\">Example Topic</td><td style=\"text-align: left; vertical-align: top;\">Workbook</td><td style=\"text-align: left; vertical-align: top;\">Topic</td><td style=\"text-align: right; vertical-align: top;\">5</td><td style=\"vertical-align: top;\">00:00:00.60</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-C7E3-6230-B51A-DC819224BB74</td><td style=\"text-align: left; vertical-align: top;\"><a target=\"_blank\" href=\"http://localhost:34216/0EECC4B1-AF3C-71A0-A425-D0461BC34EDB/workbook/0EECC4B1-C7E3-6230-B51A-DC819224BB74/worksheet/0EECC4B1-C8B0-7370-92EC-AFB1F4510B04\">link</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Name</th>\n",
       "      <th>Type</th>\n",
       "      <th>Workbook Type</th>\n",
       "      <th>Count</th>\n",
       "      <th>Time</th>\n",
       "      <th>Errors</th>\n",
       "      <th>Result</th>\n",
       "      <th>Pushed Workbook ID</th>\n",
       "      <th>URL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>D833DC83-9A38-48DE-BF45-EB787E9E8375</td>\n",
       "      <td>Example Analysis</td>\n",
       "      <td>Workbook</td>\n",
       "      <td>Analysis</td>\n",
       "      <td>53</td>\n",
       "      <td>0:00:02.557404</td>\n",
       "      <td>0</td>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4B1-AF85-7580-8F20-D91393026BA1</td>\n",
       "      <td>http://localhost:34216/0EECC4B1-AF3C-71A0-A425...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>811B1488-297A-4FD2-AE7C-A1FE0E3B3641</td>\n",
       "      <td>Example Topic</td>\n",
       "      <td>Workbook</td>\n",
       "      <td>Topic</td>\n",
       "      <td>5</td>\n",
       "      <td>0:00:00.602075</td>\n",
       "      <td>0</td>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4B1-C7E3-6230-B51A-DC819224BB74</td>\n",
       "      <td>http://localhost:34216/0EECC4B1-AF3C-71A0-A425...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID              Name      Type  \\\n",
       "0  D833DC83-9A38-48DE-BF45-EB787E9E8375  Example Analysis  Workbook   \n",
       "1  811B1488-297A-4FD2-AE7C-A1FE0E3B3641     Example Topic  Workbook   \n",
       "\n",
       "  Workbook Type Count            Time Errors   Result  \\\n",
       "0      Analysis    53  0:00:02.557404      0  Success   \n",
       "1         Topic     5  0:00:00.602075      0  Success   \n",
       "\n",
       "                     Pushed Workbook ID  \\\n",
       "0  0EECC4B1-AF85-7580-8F20-D91393026BA1   \n",
       "1  0EECC4B1-C7E3-6230-B51A-DC819224BB74   \n",
       "\n",
       "                                                 URL  \n",
       "0  http://localhost:34216/0EECC4B1-AF3C-71A0-A425...  \n",
       "1  http://localhost:34216/0EECC4B1-AF3C-71A0-A425...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_workbooks = spy.workbooks.load('Support Files/Example Export.zip')\n",
    "spy.workbooks.push(example_workbooks,\n",
    "                   path='SPy Documentation Examples >> Workbook Job Import',\n",
    "                   label=f'{spy.session.user.name} Workbook Job Example', \n",
    "                   refresh=False, \n",
    "                   errors='raise')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ca5e659",
   "metadata": {},
   "source": [
    "## Pulling Workbooks\n",
    "\n",
    "Start the job cycle by issuing the `spy.workbooks.job.pull()` to grab a set of workbooks and write them to disk.\n",
    "\n",
    "As with `spy.workbooks.pull()`, we create a DataFrame full of workbooks to pull by using the `spy.workbooks.search()` function. Then we can supply that DataFrame to `spy.workbooks.job.pull()`, which takes many of the same parameters as `spy.workbooks.pull()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dcd02efa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Query successful.</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">Count</td><td style=\"background-color: #EEFFEE; text-align: right;\">Time</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">Results</td><td style=\"text-align: right; vertical-align: top;\">2</td><td style=\"vertical-align: top;\">00:00:00.07</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Archived</th>\n",
       "      <th>Created At</th>\n",
       "      <th>ID</th>\n",
       "      <th>Name</th>\n",
       "      <th>Path</th>\n",
       "      <th>Pinned</th>\n",
       "      <th>Search Folder ID</th>\n",
       "      <th>Type</th>\n",
       "      <th>Updated At</th>\n",
       "      <th>Workbook Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2024-02-15 21:42:24.216606+00:00</td>\n",
       "      <td>0EECC4B1-AF85-7580-8F20-D91393026BA1</td>\n",
       "      <td>Example Analysis</td>\n",
       "      <td>SPy Documentation Examples &gt;&gt; Workbook Job Import</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0EECC4B1-AF3C-71A0-A425-D0461BC34EDB</td>\n",
       "      <td>Workbook</td>\n",
       "      <td>2024-02-15 21:42:27.423060+00:00</td>\n",
       "      <td>Analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2024-02-15 21:42:26.771990300+00:00</td>\n",
       "      <td>0EECC4B1-C7E3-6230-B51A-DC819224BB74</td>\n",
       "      <td>Example Topic</td>\n",
       "      <td>SPy Documentation Examples &gt;&gt; Workbook Job Import</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0EECC4B1-AF3C-71A0-A425-D0461BC34EDB</td>\n",
       "      <td>Workbook</td>\n",
       "      <td>2024-02-15 21:42:27.453059200+00:00</td>\n",
       "      <td>Topic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Archived                          Created At  \\\n",
       "0       NaN    2024-02-15 21:42:24.216606+00:00   \n",
       "1       NaN 2024-02-15 21:42:26.771990300+00:00   \n",
       "\n",
       "                                     ID              Name  \\\n",
       "0  0EECC4B1-AF85-7580-8F20-D91393026BA1  Example Analysis   \n",
       "1  0EECC4B1-C7E3-6230-B51A-DC819224BB74     Example Topic   \n",
       "\n",
       "                                                Path  Pinned  \\\n",
       "0  SPy Documentation Examples >> Workbook Job Import     NaN   \n",
       "1  SPy Documentation Examples >> Workbook Job Import     NaN   \n",
       "\n",
       "                       Search Folder ID      Type  \\\n",
       "0  0EECC4B1-AF3C-71A0-A425-D0461BC34EDB  Workbook   \n",
       "1  0EECC4B1-AF3C-71A0-A425-D0461BC34EDB  Workbook   \n",
       "\n",
       "                           Updated At Workbook Type  \n",
       "0    2024-02-15 21:42:27.423060+00:00      Analysis  \n",
       "1 2024-02-15 21:42:27.453059200+00:00         Topic  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "workbooks_df = spy.workbooks.search({\n",
    "    'Path': 'SPy Documentation Examples >> Workbook Job Import'\n",
    "})\n",
    "\n",
    "# Store these in variables that we'll use later\n",
    "example_analysis_workbook_id = workbooks_df[workbooks_df['Name'] == 'Example Analysis'].iloc[0]['ID']\n",
    "example_topic_workbook_id = workbooks_df[workbooks_df['Name'] == 'Example Topic'].iloc[0]['ID']\n",
    "\n",
    "workbooks_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3ee73c6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Pull successful</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Path</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: left;\">Workbook Type</td><td style=\"background-color: #EEFFEE; text-align: right;\">Count</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: right;\">Errors</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-AF85-7580-8F20-D91393026BA1</td><td style=\"text-align: left; vertical-align: top;\">SPy Documentation Examples >> Workbook Job Import</td><td style=\"text-align: left; vertical-align: top;\">Example Analysis</td><td style=\"text-align: left; vertical-align: top;\">Analysis</td><td style=\"text-align: right; vertical-align: top;\">53</td><td style=\"vertical-align: top;\">00:00:01.57</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-C7E3-6230-B51A-DC819224BB74</td><td style=\"text-align: left; vertical-align: top;\">SPy Documentation Examples >> Workbook Job Import</td><td style=\"text-align: left; vertical-align: top;\">Example Topic</td><td style=\"text-align: left; vertical-align: top;\">Topic</td><td style=\"text-align: right; vertical-align: top;\">9</td><td style=\"vertical-align: top;\">00:00:00.44</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Path</th>\n",
       "      <th>Name</th>\n",
       "      <th>Workbook Type</th>\n",
       "      <th>Count</th>\n",
       "      <th>Time</th>\n",
       "      <th>Errors</th>\n",
       "      <th>Result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0EECC4B1-AF85-7580-8F20-D91393026BA1</td>\n",
       "      <td>SPy Documentation Examples &gt;&gt; Workbook Job Import</td>\n",
       "      <td>Example Analysis</td>\n",
       "      <td>Analysis</td>\n",
       "      <td>53</td>\n",
       "      <td>0:00:01.573021</td>\n",
       "      <td>0</td>\n",
       "      <td>Success</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0EECC4B1-C7E3-6230-B51A-DC819224BB74</td>\n",
       "      <td>SPy Documentation Examples &gt;&gt; Workbook Job Import</td>\n",
       "      <td>Example Topic</td>\n",
       "      <td>Topic</td>\n",
       "      <td>9</td>\n",
       "      <td>0:00:00.440331</td>\n",
       "      <td>0</td>\n",
       "      <td>Success</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  \\\n",
       "0  0EECC4B1-AF85-7580-8F20-D91393026BA1   \n",
       "1  0EECC4B1-C7E3-6230-B51A-DC819224BB74   \n",
       "\n",
       "                                                Path              Name  \\\n",
       "0  SPy Documentation Examples >> Workbook Job Import  Example Analysis   \n",
       "1  SPy Documentation Examples >> Workbook Job Import     Example Topic   \n",
       "\n",
       "  Workbook Type  Count            Time  Errors   Result  \n",
       "0      Analysis     53  0:00:01.573021       0  Success  \n",
       "1         Topic      9  0:00:00.440331       0  Success  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spy.workbooks.job.pull(job_folder, workbooks_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96808342",
   "metadata": {},
   "source": [
    "As mentioned earlier, jobs are _resumable_. If you execute the above cell again, you will see that the **Result** column indicates `Already pulled`.\n",
    "\n",
    "If you would like to force a job to redo its work, supply the `resume=False` argument. You can also inspect the job folder's `Workbooks` subfolder and selectively delete workbook folders therein to force SPy to re-pull workbooks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4b0abce",
   "metadata": {},
   "source": [
    "## Pushing Workbooks\n",
    "\n",
    "As mentioned above, there are two primary scenarios where you want to push workbooks in bulk:\n",
    "\n",
    "- Re-mapping references (e.g. historian tags/signals) from one datasource to another, or one asset tree to another\n",
    "- Transferring work from one Seeq service/server to another, possibly including data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c1701c8",
   "metadata": {},
   "source": [
    "### Datasource Maps\n",
    "\n",
    "In either case, it's important to understand the concept of _datasource maps_. These are JSON files that contain instructions for SPy as it maps the identifiers in the pulled workbook definitions to identifiers on the target system. These maps can incorporate relatively complex Regular Expression specifications that allow you to re-orient workbooks from one set of input data to another.\n",
    "\n",
    "The `spy.workbooks.job.pull()` command will create a `Datasource Maps` folder inside the job folder. There will be one file for every datasource that was encountered during the pull operation -- if a workbook touched a datasource in some way, there will be a file for it.\n",
    "\n",
    "Here's what a typical file looks like:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"Datasource Class\": \"Time Series CSV Files\",\n",
    "    \"Datasource ID\": \"Example Data\",\n",
    "    \"Datasource Name\": \"Example Data\",\n",
    "    \"Item-Level Map Files\": [],\n",
    "    \"RegEx-Based Maps\": [\n",
    "        {\n",
    "            \"Old\": {\n",
    "                \"Type\": \"(?<type>.*)\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Data ID\": \"(?<data_id>.*)\"\n",
    "            },\n",
    "            \"New\": {\n",
    "                \"Type\": \"${type}\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Data ID\": \"${data_id}\"\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "\n",
    "You can make modifications to these files by loading them into an editor, including Jupyter's text editor. Generally the most common action is to add or change entries in the `RegEx-Based Maps` block. That section is a _list_ of _dictionaries_ that each have an `Old` and a `New` subsection. Within the `Old` block, you can specify properties to match on. The key is the property name and the value is a [regular expression](https://en.wikipedia.org/wiki/Regular_expression), often employing a _capture group_. In the example above, the `Data ID` field is matching using the `.*` regex and storing it in a capture group called `data_id`. The `New` block then contains the properties and values to search upon to \"map\" to target items. In the example above, the `\"Data ID\": \"${data_id}\"` specification just means that the Data ID is being used \"as-is\" without any alteration.\n",
    "\n",
    "(If you happen to be familiar with [Connector Property Transforms](https://telemetry.seeq.com/support-link/kb/latest/cloud/connector-property-transforms), this regex approach may feel familiar.)\n",
    "\n",
    "Let's look at a more complicated example:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"Datasource Class\": \"Time Series CSV Files\",\n",
    "    \"Datasource ID\": \"Example Data\",\n",
    "    \"Datasource Name\": \"Example Data\",\n",
    "    \"Item-Level Map Files\": [],\n",
    "    \"RegEx-Based Maps\": [\n",
    "        {\n",
    "            \"Old\": {\n",
    "                \"Type\": \"(?<type>.*)\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Data ID\": \"(?<data_id>.*)\"\n",
    "            },\n",
    "            \"New\": {\n",
    "                \"Type\": \"${type}\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Data ID\": \"${data_id}\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"Old\": {\n",
    "                \"Type\": \"(?<type>.*)\",\n",
    "                \"Path\": \"Example >> Cooling Tower 1\",\n",
    "                \"Asset\": \"Area (?<subarea>[ABC])\",\n",
    "                \"Name\": \"(?<name>.*)\"\n",
    "            },\n",
    "            \"New\": {\n",
    "                \"Type\": \"${type}\",\n",
    "                \"Path\": \"Example >> Cooling Tower 2\",\n",
    "                \"Asset\": \"Area ${subarea}\",\n",
    "                \"Name\": \"${name}\"\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "\n",
    "In this example there are two RegEx-Based Maps specified. The first map is identical to the previous example, and it will be used first-- if there is not a match on the `Old` regex specifications, then SPy will move on to the next. The next map matches on a particular asset path (`Example >> Cooling Tower 1`) and a set of subareas (`A`, `B`, or `C`) and then maps them to the same area underneath `Example >> Cooling Tower 2`.\n",
    "\n",
    "In this manner, you can use arbitrarily-complex mapping logic to accomplish the goal of re-mapping a workbook within the same Seeq server or properly mapping from one Seeq server to another."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "605e2579",
   "metadata": {},
   "source": [
    "#### Datasource Mapping in Action\n",
    "\n",
    "Let's run through an actual mapping scenario to see how it works and how to troubleshoot it when it goes wrong.\n",
    "\n",
    "First we have to grab a couple of signal IDs so that we can use them later to illustrate some functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0816b3f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Query successful</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">Datasource Name</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: right;\">Count</td><td style=\"background-color: #EEFFEE; text-align: right;\">Pages</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Example Data</td><td style=\"text-align: left; vertical-align: top;\">Area A_Optimizer</td><td style=\"vertical-align: top;\">00:00:00.02</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "area_a_temperature_id = spy.search({'Datasource Name': 'Example Data', 'Name': 'Area A_Temperature'}).iloc[0]['ID']\n",
    "area_a_optimizer_id = spy.search({'Datasource Name': 'Example Data', 'Name': 'Area A_Optimizer'}).iloc[0]['ID']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea4f5404",
   "metadata": {},
   "source": [
    "Now we will write out a datasource map file that has, as its first map, a `New` block that will map to a `Name` that does not exist. This will let us see what happens both when the mapping is successful and when there are errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b5eb1f03",
   "metadata": {},
   "outputs": [],
   "source": [
    "datasource_map = {\n",
    "    \"Datasource Class\": \"Time Series CSV Files\",\n",
    "    \"Datasource ID\": \"Example Data\",\n",
    "    \"Datasource Name\": \"Example Data\",\n",
    "    \"Item-Level Map Files\": [],\n",
    "    \"RegEx-Based Maps\": [\n",
    "        {\n",
    "            \"Old\": {\n",
    "                \"Type\": \"(?<type>.*)\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Name\": \"Area A_Optimizer\"\n",
    "            },\n",
    "            \"New\": {\n",
    "                \"Type\": \"${type}\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Name\": \"Area NonExistent_Optimizer\"\n",
    "            },\n",
    "            # In this contrived example, if we match on the \"Old\" criteria, we don't want to continue to the next regex map\n",
    "            \"On Match\": \"Stop\"\n",
    "        },\n",
    "        {\n",
    "            \"Old\": {\n",
    "                \"Type\": \"(?<type>.*)\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Data ID\": \"(?<data_id>.*)\"\n",
    "            },\n",
    "            \"New\": {\n",
    "                \"Type\": \"${type}\",\n",
    "                \"Datasource Class\": \"Time Series CSV Files\",\n",
    "                \"Datasource Name\": \"Example Data\",\n",
    "                \"Data ID\": \"${data_id}\"\n",
    "            }        \n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "with open(os.path.join(job_folder, 'Datasource Maps', 'Datasource_Map_Time Series CSV Files_Example Data_Example Data.json'), 'w') as f:\n",
    "    json.dump(datasource_map, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b1ca9c",
   "metadata": {},
   "source": [
    "Now we push to server using a label that is guaranteed to differentiate our activity from other users. As you will see, there will be errors reported in the `Results` column because one item won't be mapped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d200e249",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #FFEEEE;color:black; text-align: left;\">Errors encountered, look at Result column in returned DataFrame</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #FFEEEE;\"></td><td style=\"background-color: #FFEEEE; text-align: left;\">ID</td><td style=\"background-color: #FFEEEE; text-align: left;\">Name</td><td style=\"background-color: #FFEEEE; text-align: left;\">Type</td><td style=\"background-color: #FFEEEE; text-align: left;\">Workbook Type</td><td style=\"background-color: #FFEEEE; text-align: left;\">Count</td><td style=\"background-color: #FFEEEE; text-align: left;\">Time</td><td style=\"background-color: #FFEEEE; text-align: left;\">Errors</td><td style=\"background-color: #FFEEEE; text-align: left;\">Result</td><td style=\"background-color: #FFEEEE; text-align: left;\">Pushed Workbook ID</td><td style=\"background-color: #FFEEEE; text-align: left;\">URL</td></tr><tr style=\"background-color: #FFEEEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-AF85-7580-8F20-D91393026BA1</td><td style=\"text-align: left; vertical-align: top;\">Example Analysis</td><td style=\"text-align: left; vertical-align: top;\">Workbook</td><td style=\"text-align: left; vertical-align: top;\">Analysis</td><td style=\"text-align: right; vertical-align: top;\">52</td><td style=\"vertical-align: top;\">00:00:03.00</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">Success, but with errors:<br>StoredSignal \"Area A_Optimizer\" (0EECC4AC-CDE9-7710-8028-B7DE1EA451C8) not mapped, only override maps used<br>Using overrides from \\\\?\\C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\\Datasource Maps:<ul><li>Used \"\\\\?\\C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\\Datasource Maps\\Datasource_Map_Time Series CSV Files_Example Data_Example Data.json\"</li><li>RegEx-Based Map 0: Item not found on server. Details:</li></ul><br>    \"Type\"<br>        regex          \"(?<type>.*)\"<br>        matched on     \"StoredSignal\"<br>        searched for   \"Signal\"<br>    \"Datasource Class\"<br>        regex          \"Time Series CSV Files\"<br>        matched on     \"Time Series CSV Files\"<br>        searched for   \"Time Series CSV Files\"<br>    \"Name\"<br>        regex          \"Area A_Optimizer\"<br>        matched on     \"Area A_Optimizer\"<br>        searched for   \"Area NonExistent_Optimizer\"<br>    \"Datasource ID\"<br>        searched for   \"Example Data\"<br>    Capture groups:<br>        type           \"StoredSignal\"</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-E608-66C0-9E13-B2F80838C5AC</td><td style=\"text-align: left; vertical-align: top;\"><a target=\"_blank\" href=\"http://localhost:34216/0EECC4B1-E50F-6660-BC36-DFFCDF4B9D97/workbook/0EECC4B1-E608-66C0-9E13-B2F80838C5AC/worksheet/0EECC4B1-F709-7730-AD05-27B88E1688C6\">link</a></td></tr><tr style=\"background-color: #FFEEEE;\"><td style=\"vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-C7E3-6230-B51A-DC819224BB74</td><td style=\"text-align: left; vertical-align: top;\">Example Topic</td><td style=\"text-align: left; vertical-align: top;\">Workbook</td><td style=\"text-align: left; vertical-align: top;\">Topic</td><td style=\"text-align: right; vertical-align: top;\">5</td><td style=\"vertical-align: top;\">00:00:00.60</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B2-020D-6460-87D5-93CF2AC495AF</td><td style=\"text-align: left; vertical-align: top;\"><a target=\"_blank\" href=\"http://localhost:34216/0EECC4B1-E50F-6660-BC36-DFFCDF4B9D97/workbook/0EECC4B2-020D-6460-87D5-93CF2AC495AF/worksheet/0EECC4B2-034F-E8A0-9CAA-00910D7056DA\">link</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "push_df = spy.workbooks.job.push(job_folder,\n",
    "                                 path='SPy Documentation Examples >> Workbook Jobs',\n",
    "                                 label=f'{spy.session.user.name} Workbook Job Example',\n",
    "                                 errors='catalog')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb55986b",
   "metadata": {},
   "source": [
    "You can see the error, but it's not formatted very well. So let's use a troubleshooting tool-- the \"explain\" function on the returned DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e6231440",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StoredSignal \"Area A_Optimizer\" (0EECC4AC-CDE9-7710-8028-B7DE1EA451C8) not mapped, only override maps used\n",
      "Using overrides from \\\\?\\C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\\Datasource Maps:\n",
      "- Used \"\\\\?\\C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\\Datasource Maps\\Datasource_Map_Time Series CSV Files_Example Data_Example Data.json\"\n",
      "- RegEx-Based Map 0: Item not found on server. Details:\n",
      "    \"Type\"\n",
      "        regex          \"(?<type>.*)\"\n",
      "        matched on     \"StoredSignal\"\n",
      "        searched for   \"Signal\"\n",
      "    \"Datasource Class\"\n",
      "        regex          \"Time Series CSV Files\"\n",
      "        matched on     \"Time Series CSV Files\"\n",
      "        searched for   \"Time Series CSV Files\"\n",
      "    \"Name\"\n",
      "        regex          \"Area A_Optimizer\"\n",
      "        matched on     \"Area A_Optimizer\"\n",
      "        searched for   \"Area NonExistent_Optimizer\"\n",
      "    \"Datasource ID\"\n",
      "        searched for   \"Example Data\"\n",
      "    Capture groups:\n",
      "        type           \"StoredSignal\"\n"
     ]
    }
   ],
   "source": [
    "print(push_df.spy.item_map.explain(area_a_optimizer_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4abc5594",
   "metadata": {},
   "source": [
    "This detailed explanation is intended to give you a starting point for troubleshooting. You can see the **regex** that was specified, the property values that were **matched on**, the **Capture groups** that resulted from the RegEx specifications, and the property values that were subsequently **searched for**. Since `Area NonExistent_Optimizer` does not exist, the explanation for `RegEx-Based Map 0` says _Item not found on server_.\n",
    "\n",
    "Now let's look at the explanation for a successful map (`Area A_Temperature`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "367d61ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using overrides from \\\\?\\C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\\Datasource Maps:\n",
      "- Used \"\\\\?\\C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\\Datasource Maps\\Datasource_Map_Time Series CSV Files_Example Data_Example Data.json\"\n",
      "- RegEx-Based Map 0: Unsuccessful match. Details:\n",
      "    \"Name\"\n",
      "        regex          \"Area A_Optimizer\"\n",
      "        does not match \"Area A_Temperature\"\n",
      "- RegEx-Based Map 1: Successfully matched. Details:\n",
      "    \"Type\"\n",
      "        regex          \"(?<type>.*)\"\n",
      "        matched on     \"StoredSignal\"\n",
      "        searched for   \"Signal\"\n",
      "        and found      \"StoredSignal\"\n",
      "    \"Datasource Class\"\n",
      "        regex          \"Time Series CSV Files\"\n",
      "        matched on     \"Time Series CSV Files\"\n",
      "        searched for   \"Time Series CSV Files\"\n",
      "        and found      \"Time Series CSV Files\"\n",
      "    \"Data ID\"\n",
      "        regex          \"(?<data_id>.*)\"\n",
      "        matched on     \"[Tag] Area A_Temperature.sim.ts.csv\"\n",
      "        searched for   \"[Tag] Area A_Temperature.sim.ts.csv\"\n",
      "        and found      \"[Tag] Area A_Temperature.sim.ts.csv\"\n",
      "    \"Datasource ID\"\n",
      "        searched for   \"Example Data\"\n",
      "        and found      \"Example Data\"\n",
      "    Capture groups:\n",
      "        type           \"StoredSignal\"\n",
      "        data_id        \"[Tag] Area A_Temperature.sim.ts.csv\"\n",
      "Successful mapping:\n",
      "  Old: StoredSignal \"Area A_Temperature\" (0EECC4AC-CD98-EE00-A7AE-F057000E4F51)\n",
      "  New: StoredSignal \"Area A_Temperature\" (0EECC4AC-CD98-EE00-A7AE-F057000E4F51)\n"
     ]
    }
   ],
   "source": [
    "print(push_df.spy.item_map.explain(area_a_temperature_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "379b6459",
   "metadata": {},
   "source": [
    "### Dummy Items\n",
    "\n",
    "In cases where we couldn't map successfully, we can tell SPy to create \"dummy\" items. A dummy item is a signal, condition or scalar that has all the properties of the original item but has no data. (We'll show how to push data to dummy items later...)\n",
    "\n",
    "Note the use of `create_dummy_items=True`, and also note `resume=False` so that SPy tries to push the workbooks again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8166e94b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Push successful</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: left;\">Type</td><td style=\"background-color: #EEFFEE; text-align: left;\">Workbook Type</td><td style=\"background-color: #EEFFEE; text-align: left;\">Count</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: left;\">Errors</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td><td style=\"background-color: #EEFFEE; text-align: left;\">Pushed Workbook ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">URL</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-AF85-7580-8F20-D91393026BA1</td><td style=\"text-align: left; vertical-align: top;\">Example Analysis</td><td style=\"text-align: left; vertical-align: top;\">Workbook</td><td style=\"text-align: left; vertical-align: top;\">Analysis</td><td style=\"text-align: right; vertical-align: top;\">53</td><td style=\"vertical-align: top;\">00:00:02.98</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-E608-66C0-9E13-B2F80838C5AC</td><td style=\"text-align: left; vertical-align: top;\"><a target=\"_blank\" href=\"http://localhost:34216/0EECC4B1-E50F-6660-BC36-DFFCDF4B9D97/workbook/0EECC4B1-E608-66C0-9E13-B2F80838C5AC/worksheet/0EECC4B1-F709-7730-AD05-27B88E1688C6\">link</a></td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-C7E3-6230-B51A-DC819224BB74</td><td style=\"text-align: left; vertical-align: top;\">Example Topic</td><td style=\"text-align: left; vertical-align: top;\">Workbook</td><td style=\"text-align: left; vertical-align: top;\">Topic</td><td style=\"text-align: right; vertical-align: top;\">5</td><td style=\"vertical-align: top;\">00:00:01.14</td><td style=\"text-align: right; vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">Success</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B2-020D-6460-87D5-93CF2AC495AF</td><td style=\"text-align: left; vertical-align: top;\"><a target=\"_blank\" href=\"http://localhost:34216/0EECC4B1-E50F-6660-BC36-DFFCDF4B9D97/workbook/0EECC4B2-020D-6460-87D5-93CF2AC495AF/worksheet/0EECC4B2-034F-E8A0-9CAA-00910D7056DA\">link</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "push_df = spy.workbooks.job.push(\n",
    "    job_folder,\n",
    "    path='SPy Documentation Examples >> Workbook Jobs',\n",
    "    label=f'{spy.session.user.name} Workbook Job Example',\n",
    "    create_dummy_items=True,\n",
    "    resume=False,\n",
    "    errors='catalog')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dca52b69",
   "metadata": {},
   "source": [
    "Now there are no errors, because any item that couldn't be mapped would be replaced by a dummy item.\n",
    "\n",
    "Find the `Example Analysis` row of the output table above and click on the _link_ in the `URL` column to take a look at the resulting. You'll see that the **Details Pane** worksheet contains **Area A_Optimizer**, which is a blank \"dummy item\". \n",
    "\n",
    "We can look at what dummy items were created by inspecting `push_df.spy.item_map.dummy_items`. You can see that the `Name` is the same as the original and important properties like `Maximum Interpolation` have made their way to the dummy item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2a27b2b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Archived</th>\n",
       "      <th>Cache Enabled</th>\n",
       "      <th>Cache ID</th>\n",
       "      <th>Data ID</th>\n",
       "      <th>Enabled</th>\n",
       "      <th>Interpolation Method</th>\n",
       "      <th>Key Unit Of Measure</th>\n",
       "      <th>Maximum Interpolation</th>\n",
       "      <th>Name</th>\n",
       "      <th>Source Maximum Interpolation</th>\n",
       "      <th>...</th>\n",
       "      <th>Datasource Name</th>\n",
       "      <th>Original ID</th>\n",
       "      <th>Original Datasource Class</th>\n",
       "      <th>Original Datasource ID</th>\n",
       "      <th>Original Data ID</th>\n",
       "      <th>Scoped To</th>\n",
       "      <th>Datasource Class</th>\n",
       "      <th>Datasource ID</th>\n",
       "      <th>Formula Parameters</th>\n",
       "      <th>ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0eecc4ac-cde9-7710-9ba8-bdbd7cda94ba</td>\n",
       "      <td>[0EECC4B2-09C4-75F0-A556-91E2096CC581] {Signal...</td>\n",
       "      <td>True</td>\n",
       "      <td>Linear</td>\n",
       "      <td>ns</td>\n",
       "      <td>2min</td>\n",
       "      <td>Area A_Optimizer</td>\n",
       "      <td>2min</td>\n",
       "      <td>...</td>\n",
       "      <td>Example Data</td>\n",
       "      <td>0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td>\n",
       "      <td>Time Series CSV Files</td>\n",
       "      <td>Example Data</td>\n",
       "      <td>[Tag] Area A_Optimizer.sim.ts.csv</td>\n",
       "      <td>0EECC4B2-09C4-75F0-A556-91E2096CC581</td>\n",
       "      <td>Seeq Data Lab</td>\n",
       "      <td>Seeq Data Lab</td>\n",
       "      <td>[]</td>\n",
       "      <td>0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Archived  Cache Enabled                              Cache ID  \\\n",
       "0     False          False  0eecc4ac-cde9-7710-9ba8-bdbd7cda94ba   \n",
       "\n",
       "                                             Data ID  Enabled  \\\n",
       "0  [0EECC4B2-09C4-75F0-A556-91E2096CC581] {Signal...     True   \n",
       "\n",
       "  Interpolation Method Key Unit Of Measure Maximum Interpolation  \\\n",
       "0               Linear                  ns                  2min   \n",
       "\n",
       "               Name Source Maximum Interpolation  ... Datasource Name  \\\n",
       "0  Area A_Optimizer                         2min  ...    Example Data   \n",
       "\n",
       "                            Original ID Original Datasource Class  \\\n",
       "0  0EECC4AC-CDE9-7710-8028-B7DE1EA451C8     Time Series CSV Files   \n",
       "\n",
       "   Original Datasource ID                   Original Data ID  \\\n",
       "0            Example Data  [Tag] Area A_Optimizer.sim.ts.csv   \n",
       "\n",
       "                              Scoped To Datasource Class  Datasource ID  \\\n",
       "0  0EECC4B2-09C4-75F0-A556-91E2096CC581    Seeq Data Lab  Seeq Data Lab   \n",
       "\n",
       "  Formula Parameters                                    ID  \n",
       "0                 []  0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540  \n",
       "\n",
       "[1 rows x 25 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "push_df.spy.item_map.dummy_items"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25844f10",
   "metadata": {},
   "source": [
    "## Including Data\n",
    "\n",
    "Dummy items are helpful, but they are \"blank,\" they do not have any data associated with them. If you are transferring workbooks between servers and the destination server doesn't have access to the same datasources, it is useful to be able transfer the data itself from the source server to the dummy items on the destination server. A set of SPy functions is provided in `spy.workbooks.job.data` for this purpose.\n",
    "\n",
    "As `spy.workbooks.job.pull()` pulls workbook information, it also tracks the usage of data items on Workbench Worksheets and in Organizer Topic Documents. This information is collated and saved to disk as the _data manifest_. You can inspect the manifest like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2b0fd752",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Path</th>\n",
       "      <th>Asset</th>\n",
       "      <th>Name</th>\n",
       "      <th>Start</th>\n",
       "      <th>End</th>\n",
       "      <th>Calculation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0EECC4AC-CD98-EE00-A7AE-F057000E4F51</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Temperature</td>\n",
       "      <td>2018-11-01 12:36:56.759000+00:00</td>\n",
       "      <td>2024-02-15 23:42:28.015433+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-01T1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0EECC4AC-D00E-EC20-8933-BEFAD4423372</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Compressor Stage</td>\n",
       "      <td>2018-11-01 12:36:56.759000+00:00</td>\n",
       "      <td>2024-02-15 21:42:28.015433+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-01T1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Optimizer</td>\n",
       "      <td>2018-11-11 04:22:45.084000+00:00</td>\n",
       "      <td>2024-02-15 21:42:28.015433+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-11T0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0EECC4AC-D2B5-7780-99B9-17F6440D6464</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Compressor Power</td>\n",
       "      <td>2018-11-10 16:31:09.311000+00:00</td>\n",
       "      <td>2024-02-15 21:42:28.015433+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-10T1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0EECC4AC-D293-64A0-B188-6B0363BC3917</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Wet Bulb</td>\n",
       "      <td>2018-11-11 04:22:45.084000+00:00</td>\n",
       "      <td>2018-12-17 06:18:49.287000+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-11T0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Relative Humidity</td>\n",
       "      <td>2018-11-01 12:36:56.759000+00:00</td>\n",
       "      <td>2018-11-12 04:22:45.084000+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-01T1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0EECC4AC-D1F4-F990-B25B-871ACD3DC386</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area C_Temperature</td>\n",
       "      <td>2018-11-10 16:31:09.311000+00:00</td>\n",
       "      <td>2024-02-15 21:42:28.015433+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2018-11-10T1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0EECC4AC-CC98-E870-973E-05647554A760</td>\n",
       "      <td>Example &gt;&gt; Cooling Tower 1</td>\n",
       "      <td>Area A</td>\n",
       "      <td>Temperature</td>\n",
       "      <td>2019-09-07 13:23:27.130000+00:00</td>\n",
       "      <td>2019-10-07 23:23:27.130000+00:00</td>\n",
       "      <td>$within = condition(\\n   capsule(\"2019-09-07T1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID                        Path   Asset  \\\n",
       "0  0EECC4AC-CD98-EE00-A7AE-F057000E4F51                         NaN     NaN   \n",
       "1  0EECC4AC-D00E-EC20-8933-BEFAD4423372                         NaN     NaN   \n",
       "2  0EECC4AC-CDE9-7710-8028-B7DE1EA451C8                         NaN     NaN   \n",
       "3  0EECC4AC-D2B5-7780-99B9-17F6440D6464                         NaN     NaN   \n",
       "4  0EECC4AC-D293-64A0-B188-6B0363BC3917                         NaN     NaN   \n",
       "5  0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A                         NaN     NaN   \n",
       "6  0EECC4AC-D1F4-F990-B25B-871ACD3DC386                         NaN     NaN   \n",
       "7  0EECC4AC-CC98-E870-973E-05647554A760  Example >> Cooling Tower 1  Area A   \n",
       "\n",
       "                       Name                            Start  \\\n",
       "0        Area A_Temperature 2018-11-01 12:36:56.759000+00:00   \n",
       "1   Area A_Compressor Stage 2018-11-01 12:36:56.759000+00:00   \n",
       "2          Area A_Optimizer 2018-11-11 04:22:45.084000+00:00   \n",
       "3   Area A_Compressor Power 2018-11-10 16:31:09.311000+00:00   \n",
       "4           Area A_Wet Bulb 2018-11-11 04:22:45.084000+00:00   \n",
       "5  Area A_Relative Humidity 2018-11-01 12:36:56.759000+00:00   \n",
       "6        Area C_Temperature 2018-11-10 16:31:09.311000+00:00   \n",
       "7               Temperature 2019-09-07 13:23:27.130000+00:00   \n",
       "\n",
       "                               End  \\\n",
       "0 2024-02-15 23:42:28.015433+00:00   \n",
       "1 2024-02-15 21:42:28.015433+00:00   \n",
       "2 2024-02-15 21:42:28.015433+00:00   \n",
       "3 2024-02-15 21:42:28.015433+00:00   \n",
       "4 2018-12-17 06:18:49.287000+00:00   \n",
       "5 2018-11-12 04:22:45.084000+00:00   \n",
       "6 2024-02-15 21:42:28.015433+00:00   \n",
       "7 2019-10-07 23:23:27.130000+00:00   \n",
       "\n",
       "                                         Calculation  \n",
       "0  $within = condition(\\n   capsule(\"2018-11-01T1...  \n",
       "1  $within = condition(\\n   capsule(\"2018-11-01T1...  \n",
       "2  $within = condition(\\n   capsule(\"2018-11-11T0...  \n",
       "3  $within = condition(\\n   capsule(\"2018-11-10T1...  \n",
       "4  $within = condition(\\n   capsule(\"2018-11-11T0...  \n",
       "5  $within = condition(\\n   capsule(\"2018-11-01T1...  \n",
       "6  $within = condition(\\n   capsule(\"2018-11-10T1...  \n",
       "7  $within = condition(\\n   capsule(\"2019-09-07T1...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manifest_df = spy.workbooks.job.data.manifest(job_folder)\n",
    "\n",
    "# Simplify the DataFrame so that it fits on the screen better\n",
    "manifest_df[['ID', 'Path', 'Asset', 'Name', 'Start', 'End', 'Calculation']]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26d95a0-2969-4907-9e3a-f0d7e6ec629d",
   "metadata": {},
   "source": [
    "You can see `Start` and `End` columns that provide the overall time bounds that were detected in the workbook data references. There is also a `Calculation` column that refines this broad time period into specific \"chunks\" of data, defined as individual capsules and using the `within()` and `touches()` Seeq Formula functions to pull data only for those time periods -- not just everything between `Start` and `End`.\n",
    "\n",
    "This manifest DataFrame can be fed directly into `spy.pull()` but it is recommended that you use `spy.workbooks.job.data.pull()` like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2408c450-c016-41e3-a7d7-dc81417ee1b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Pull successful from <strong>2024-02-15 20:42:38.235146+00:00</strong> to <strong>2024-02-15 21:42:38.235146+00:00</strong></div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Path</td><td style=\"background-color: #EEFFEE; text-align: left;\">Asset</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: right;\">Count</td><td style=\"background-color: #EEFFEE; text-align: right;\">Pages</td><td style=\"background-color: #EEFFEE; text-align: left;\">Data Processed</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-CD98-EE00-A7AE-F057000E4F51</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area A_Temperature</td><td style=\"vertical-align: top;\">00:00:03.21</td><td style=\"text-align: right; vertical-align: top;\">13330</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">22 MB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">1</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-D00E-EC20-8933-BEFAD4423372</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area A_Compressor Stage</td><td style=\"vertical-align: top;\">00:00:02.50</td><td style=\"text-align: right; vertical-align: top;\">6518</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">22 MB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">2</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area A_Optimizer</td><td style=\"vertical-align: top;\">00:00:02.54</td><td style=\"text-align: right; vertical-align: top;\">756</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">22 MB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">3</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-D2B5-7780-99B9-17F6440D6464</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area A_Compressor Power</td><td style=\"vertical-align: top;\">00:00:02.81</td><td style=\"text-align: right; vertical-align: top;\">3242</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">22 MB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">4</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-D293-64A0-B188-6B0363BC3917</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area A_Wet Bulb</td><td style=\"vertical-align: top;\">00:00:00.66</td><td style=\"text-align: right; vertical-align: top;\">1446</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">415 KB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">5</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area A_Relative Humidity</td><td style=\"vertical-align: top;\">00:00:00.63</td><td style=\"text-align: right; vertical-align: top;\">5766</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">122 KB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">6</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-D1F4-F990-B25B-871ACD3DC386</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: right; vertical-align: top;\">nan</td><td style=\"text-align: left; vertical-align: top;\">Area C_Temperature</td><td style=\"vertical-align: top;\">00:00:03.16</td><td style=\"text-align: right; vertical-align: top;\">2886</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">22 MB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">7</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-CC98-E870-973E-05647554A760</td><td style=\"text-align: left; vertical-align: top;\">Example >> Cooling Tower 1</td><td style=\"text-align: left; vertical-align: top;\">Area A</td><td style=\"text-align: left; vertical-align: top;\">Temperature</td><td style=\"vertical-align: top;\">00:00:01.04</td><td style=\"text-align: right; vertical-align: top;\">21903</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"text-align: right; vertical-align: top;\">350 KB</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Result</th>\n",
       "      <th>ID</th>\n",
       "      <th>Type</th>\n",
       "      <th>Path</th>\n",
       "      <th>Asset</th>\n",
       "      <th>Name</th>\n",
       "      <th>Time</th>\n",
       "      <th>Count</th>\n",
       "      <th>Pages</th>\n",
       "      <th>Data Processed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-CD98-EE00-A7AE-F057000E4F51</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-CD98-EE00-A7AE-F057000E4F51</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Temperature</td>\n",
       "      <td>0:00:03.212532</td>\n",
       "      <td>13330</td>\n",
       "      <td>1</td>\n",
       "      <td>22 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-D00E-EC20-8933-BEFAD4423372</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-D00E-EC20-8933-BEFAD4423372</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Compressor Stage</td>\n",
       "      <td>0:00:02.499201</td>\n",
       "      <td>6518</td>\n",
       "      <td>1</td>\n",
       "      <td>22 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Optimizer</td>\n",
       "      <td>0:00:02.543195</td>\n",
       "      <td>756</td>\n",
       "      <td>1</td>\n",
       "      <td>22 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-D2B5-7780-99B9-17F6440D6464</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-D2B5-7780-99B9-17F6440D6464</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Compressor Power</td>\n",
       "      <td>0:00:02.805694</td>\n",
       "      <td>3242</td>\n",
       "      <td>1</td>\n",
       "      <td>22 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-D293-64A0-B188-6B0363BC3917</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-D293-64A0-B188-6B0363BC3917</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Wet Bulb</td>\n",
       "      <td>0:00:00.664129</td>\n",
       "      <td>1446</td>\n",
       "      <td>1</td>\n",
       "      <td>415 KB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Relative Humidity</td>\n",
       "      <td>0:00:00.631141</td>\n",
       "      <td>5766</td>\n",
       "      <td>1</td>\n",
       "      <td>122 KB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-D1F4-F990-B25B-871ACD3DC386</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-D1F4-F990-B25B-871ACD3DC386</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area C_Temperature</td>\n",
       "      <td>0:00:03.159527</td>\n",
       "      <td>2886</td>\n",
       "      <td>1</td>\n",
       "      <td>22 MB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-CC98-E870-973E-05647554A760</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4AC-CC98-E870-973E-05647554A760</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Example &gt;&gt; Cooling Tower 1</td>\n",
       "      <td>Area A</td>\n",
       "      <td>Temperature</td>\n",
       "      <td>0:00:01.044136</td>\n",
       "      <td>21903</td>\n",
       "      <td>1</td>\n",
       "      <td>350 KB</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       Result  \\\n",
       "0EECC4AC-CD98-EE00-A7AE-F057000E4F51  Success   \n",
       "0EECC4AC-D00E-EC20-8933-BEFAD4423372  Success   \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  Success   \n",
       "0EECC4AC-D2B5-7780-99B9-17F6440D6464  Success   \n",
       "0EECC4AC-D293-64A0-B188-6B0363BC3917  Success   \n",
       "0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A  Success   \n",
       "0EECC4AC-D1F4-F990-B25B-871ACD3DC386  Success   \n",
       "0EECC4AC-CC98-E870-973E-05647554A760  Success   \n",
       "\n",
       "                                                                        ID  \\\n",
       "0EECC4AC-CD98-EE00-A7AE-F057000E4F51  0EECC4AC-CD98-EE00-A7AE-F057000E4F51   \n",
       "0EECC4AC-D00E-EC20-8933-BEFAD4423372  0EECC4AC-D00E-EC20-8933-BEFAD4423372   \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  0EECC4AC-CDE9-7710-8028-B7DE1EA451C8   \n",
       "0EECC4AC-D2B5-7780-99B9-17F6440D6464  0EECC4AC-D2B5-7780-99B9-17F6440D6464   \n",
       "0EECC4AC-D293-64A0-B188-6B0363BC3917  0EECC4AC-D293-64A0-B188-6B0363BC3917   \n",
       "0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A  0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A   \n",
       "0EECC4AC-D1F4-F990-B25B-871ACD3DC386  0EECC4AC-D1F4-F990-B25B-871ACD3DC386   \n",
       "0EECC4AC-CC98-E870-973E-05647554A760  0EECC4AC-CC98-E870-973E-05647554A760   \n",
       "\n",
       "                                     Type                        Path   Asset  \\\n",
       "0EECC4AC-CD98-EE00-A7AE-F057000E4F51  NaN                         NaN     NaN   \n",
       "0EECC4AC-D00E-EC20-8933-BEFAD4423372  NaN                         NaN     NaN   \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  NaN                         NaN     NaN   \n",
       "0EECC4AC-D2B5-7780-99B9-17F6440D6464  NaN                         NaN     NaN   \n",
       "0EECC4AC-D293-64A0-B188-6B0363BC3917  NaN                         NaN     NaN   \n",
       "0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A  NaN                         NaN     NaN   \n",
       "0EECC4AC-D1F4-F990-B25B-871ACD3DC386  NaN                         NaN     NaN   \n",
       "0EECC4AC-CC98-E870-973E-05647554A760  NaN  Example >> Cooling Tower 1  Area A   \n",
       "\n",
       "                                                          Name  \\\n",
       "0EECC4AC-CD98-EE00-A7AE-F057000E4F51        Area A_Temperature   \n",
       "0EECC4AC-D00E-EC20-8933-BEFAD4423372   Area A_Compressor Stage   \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8          Area A_Optimizer   \n",
       "0EECC4AC-D2B5-7780-99B9-17F6440D6464   Area A_Compressor Power   \n",
       "0EECC4AC-D293-64A0-B188-6B0363BC3917           Area A_Wet Bulb   \n",
       "0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A  Area A_Relative Humidity   \n",
       "0EECC4AC-D1F4-F990-B25B-871ACD3DC386        Area C_Temperature   \n",
       "0EECC4AC-CC98-E870-973E-05647554A760               Temperature   \n",
       "\n",
       "                                                Time  Count Pages  \\\n",
       "0EECC4AC-CD98-EE00-A7AE-F057000E4F51  0:00:03.212532  13330     1   \n",
       "0EECC4AC-D00E-EC20-8933-BEFAD4423372  0:00:02.499201   6518     1   \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  0:00:02.543195    756     1   \n",
       "0EECC4AC-D2B5-7780-99B9-17F6440D6464  0:00:02.805694   3242     1   \n",
       "0EECC4AC-D293-64A0-B188-6B0363BC3917  0:00:00.664129   1446     1   \n",
       "0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A  0:00:00.631141   5766     1   \n",
       "0EECC4AC-D1F4-F990-B25B-871ACD3DC386  0:00:03.159527   2886     1   \n",
       "0EECC4AC-CC98-E870-973E-05647554A760  0:00:01.044136  21903     1   \n",
       "\n",
       "                                     Data Processed  \n",
       "0EECC4AC-CD98-EE00-A7AE-F057000E4F51          22 MB  \n",
       "0EECC4AC-D00E-EC20-8933-BEFAD4423372          22 MB  \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8          22 MB  \n",
       "0EECC4AC-D2B5-7780-99B9-17F6440D6464          22 MB  \n",
       "0EECC4AC-D293-64A0-B188-6B0363BC3917         415 KB  \n",
       "0EECC4AC-CE3A-6020-B5E1-30F6D0EC3E6A         122 KB  \n",
       "0EECC4AC-D1F4-F990-B25B-871ACD3DC386          22 MB  \n",
       "0EECC4AC-CC98-E870-973E-05647554A760         350 KB  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spy.workbooks.job.data.pull(job_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb209b9-0e8e-4fd2-bf0a-aa588fe1b91f",
   "metadata": {},
   "source": [
    "Data has now been added to the job folder for the time periods identified in the manifest and can be pushed to the dummy items like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "db7e3976-e6dd-43f6-bb03-3600c0e61287",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Pushed successfully to datasource <strong>Seeq Data Lab [Datasource ID: Seeq Data Lab]</strong> and scoped to workbook ID <strong>0EECC4B2-09C4-75F0-A556-91E2096CC581</strong><br>Click the following link to see what you pushed in Seeq:<br><a href=\"http://localhost:34216/0EECC4B2-0965-6280-BF34-4FFD36EC4253/workbook/0EECC4B2-09C4-75F0-A556-91E2096CC581/worksheet/0EECC4B2-0A17-6610-90FB-5CC634AA3613\" target=\"_blank\">http://localhost:34216/0EECC4B2-0965-6280-BF34-4FFD36EC4253/workbook/0EECC4B2-09C4-75F0-A556-91E2096CC581/worksheet/0EECC4B2-0A17-6610-90FB-5CC634AA3613</a></div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Type</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: right;\">Count</td><td style=\"background-color: #EEFFEE; text-align: right;\">Pages</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540</td><td style=\"text-align: left; vertical-align: top;\">StoredSignal</td><td style=\"text-align: left; vertical-align: top;\">Area A_Optimizer</td><td style=\"text-align: right; vertical-align: top;\">753</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"vertical-align: top;\">00:00:00.04</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Result</th>\n",
       "      <th>ID</th>\n",
       "      <th>Type</th>\n",
       "      <th>Path</th>\n",
       "      <th>Asset</th>\n",
       "      <th>Name</th>\n",
       "      <th>Time</th>\n",
       "      <th>Count</th>\n",
       "      <th>Pages</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540</td>\n",
       "      <td>StoredSignal</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Optimizer</td>\n",
       "      <td>0:00:00.042000</td>\n",
       "      <td>753</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       Result  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  Success   \n",
       "\n",
       "                                                                        ID  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540   \n",
       "\n",
       "                                              Type Path Asset  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  StoredSignal  NaN   NaN   \n",
       "\n",
       "                                                  Name            Time Count  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  Area A_Optimizer  0:00:00.042000   753   \n",
       "\n",
       "                                     Pages  \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8     1  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spy.workbooks.job.data.push(job_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04183b3a-6820-47c6-a8d6-9431d7d4ee9d",
   "metadata": {},
   "source": [
    "If you refresh the page of the **Details Pane** within the **Example Analysis**, you'll now see data for **Area A_Optimizer**.\n",
    "\n",
    "However, if you move the Display Range to the left, you'll see that **Area A_Optimizer** data only exists for the time period that was originally on the screen.\n",
    "\n",
    "If you want to expand how much data is pulled for a particular item, you can execute a command to alter the manifest and then pull/push again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f8f9b207-8c14-4434-a8f9-3b05ad117b5c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Pushed successfully to datasource <strong>Seeq Data Lab [Datasource ID: Seeq Data Lab]</strong> and scoped to workbook ID <strong>0EECC4B2-09C4-75F0-A556-91E2096CC581</strong><br>Click the following link to see what you pushed in Seeq:<br><a href=\"http://localhost:34216/0EECC4B2-0965-6280-BF34-4FFD36EC4253/workbook/0EECC4B2-09C4-75F0-A556-91E2096CC581/worksheet/0EECC4B2-0A17-6610-90FB-5CC634AA3613\" target=\"_blank\">http://localhost:34216/0EECC4B2-0965-6280-BF34-4FFD36EC4253/workbook/0EECC4B2-09C4-75F0-A556-91E2096CC581/worksheet/0EECC4B2-0A17-6610-90FB-5CC634AA3613</a></div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Type</td><td style=\"background-color: #EEFFEE; text-align: left;\">Name</td><td style=\"background-color: #EEFFEE; text-align: right;\">Count</td><td style=\"background-color: #EEFFEE; text-align: right;\">Pages</td><td style=\"background-color: #EEFFEE; text-align: left;\">Time</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540</td><td style=\"text-align: left; vertical-align: top;\">StoredSignal</td><td style=\"text-align: left; vertical-align: top;\">Area A_Optimizer</td><td style=\"text-align: right; vertical-align: top;\">30993</td><td style=\"text-align: right; vertical-align: top;\">1</td><td style=\"vertical-align: top;\">00:00:00.31</td><td style=\"text-align: left; vertical-align: top;\">Success</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Result</th>\n",
       "      <th>ID</th>\n",
       "      <th>Type</th>\n",
       "      <th>Path</th>\n",
       "      <th>Asset</th>\n",
       "      <th>Name</th>\n",
       "      <th>Time</th>\n",
       "      <th>Count</th>\n",
       "      <th>Pages</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</th>\n",
       "      <td>Success</td>\n",
       "      <td>0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540</td>\n",
       "      <td>StoredSignal</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Area A_Optimizer</td>\n",
       "      <td>0 days 00:00:00.313617</td>\n",
       "      <td>30993</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       Result  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  Success   \n",
       "\n",
       "                                                                        ID  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  0EECC4B2-0F1B-E8F0-9654-58C4BB0AD540   \n",
       "\n",
       "                                              Type Path Asset  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  StoredSignal  NaN   NaN   \n",
       "\n",
       "                                                  Name                   Time  \\\n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  Area A_Optimizer 0 days 00:00:00.313617   \n",
       "\n",
       "                                      Count Pages  \n",
       "0EECC4AC-CDE9-7710-8028-B7DE1EA451C8  30993     1  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Expand the time periods for Area A_Optimizer by 2 weeks on either side\n",
    "spy.workbooks.job.data.expand(job_folder, {'Name': 'Area A_Optimizer'}, by='2w')\n",
    "\n",
    "spy.workbooks.job.data.pull(job_folder, resume=False)\n",
    "spy.workbooks.job.data.push(job_folder, resume=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48d2e8de-692c-48d3-852c-7f949d79a041",
   "metadata": {},
   "source": [
    "There are a series of functions to alter the manifest, and\n",
    "\n",
    "- `spy.workbooks.job.data.expand()` - Expand the existing time periods.\n",
    "- `spy.workbooks.job.data.add()` - Add a specific time period.\n",
    "- `spy.workbooks.job.data.remove()` - Remove a specific time period.\n",
    "- `spy.workbooks.job.data.calculation()` - Apply a specific calculation, such as resample().\n",
    "\n",
    "Documentation for these functions is found under the **Detailed Help** section below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71651071",
   "metadata": {},
   "source": [
    "## Redoing Specific Workbooks/Items\n",
    "\n",
    "In the process of pushing and pulling, you will usually use the `errors='catalog'` flag, which means that errors will be enumerated but the operation will keep going if at all possible. When you resume an operation, those items that had errors will not be re-attempted, because SPy (by default) assumes that you don't care about them.\n",
    "\n",
    "But you will often care about errors, and you will figure out how to fix them (say, by altering a Datasource Map). You can force SPy to redo a push or pull operation for a particular item or set of items using the `redo` family of functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e247daf4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Successfully marked specified items to be redone. Execute spy.job.workbooks.pull() and/or spy.job.workbooks.push() again.</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">0EECC4B1-AF85-7580-8F20-D91393026BA1</td><td style=\"text-align: left; vertical-align: top;\">Pull will be redone</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spy.workbooks.job.redo(job_folder, example_analysis_workbook_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "93f9221b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Successfully marked specified items to be redone. Execute spy.job.workbooks.data.pull() and/or spy.job.workbooks.data.push() again.</div><table class=\"tex2jax_ignore\" style=\"color:black;\"><tr><td style=\"background-color: #EEFFEE;\"></td><td style=\"background-color: #EEFFEE; text-align: left;\">ID</td><td style=\"background-color: #EEFFEE; text-align: left;\">Result</td></tr><tr style=\"background-color: #EEFFEE;\"><td style=\"vertical-align: top;\">0</td><td style=\"text-align: left; vertical-align: top;\">0EECC4AC-CDE9-7710-8028-B7DE1EA451C8</td><td style=\"text-align: left; vertical-align: top;\">Data pull will be redone</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spy.workbooks.job.data.redo(job_folder, area_a_optimizer_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92260d80",
   "metadata": {},
   "source": [
    "## Zip/Unzip the Job Folder\n",
    "\n",
    "If you are intending to transfer workbook information to another Seeq server, it is convenient to package up the job folder as a zip file. There are two functions for this purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e6cade51",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Success: Zip file written to \"C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job.zip\"</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spy.workbooks.job.zip(job_folder, overwrite=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "01621de8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"background-color: #EEFFEE;color:black; text-align: left;\">Success: Zip file unzipped to \"C:\\dev\\develop\\sdk\\pypi\\seeq\\spy\\docs\\Documentation\\Output\\My First Workbooks Job\"</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spy.workbooks.job.unzip(job_folder + '.zip', overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c89284e9-9d37-41e5-80c3-e9babca60415",
   "metadata": {},
   "source": [
    "## Detailed Help\n",
    "\n",
    "All SPy functions have detailed documentation to help you use them. Just execute `help(spy.<func>)` like\n",
    "you see below.\n",
    "\n",
    "**Make sure you re-execute the cells below to see the latest documentation. It otherwise might be from an\n",
    "earlier version of SPy.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "5f4ebe44-5581-49cd-b781-a95f1fad713a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function pull in module seeq.spy.workbooks.job._pull:\n",
      "\n",
      "pull(job_folder: 'str', workbooks_df: 'Union[pd.DataFrame, str]', *, resume: 'bool' = True, include_referenced_workbooks: 'bool' = True, include_rendered_content: 'bool' = False, errors: 'Optional[str]' = None, quiet: 'Optional[bool]' = None, status: 'Optional[Status]' = None, session: 'Optional[Session]' = None) -> 'pd.DataFrame'\n",
      "    Pulls the definitions for each workbook specified by workbooks_df on to\n",
      "    disk, in a restartable \"job\"-like fashion.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to a folder on disk where the workbooks\n",
      "        definitions will be saved. If the folder does not exist, it will be\n",
      "        created. If the folder exists, the job will continue where it left off.\n",
      "    \n",
      "    workbooks_df : {str, pd.DataFrame}\n",
      "        A DataFrame containing 'ID', 'Type' and 'Workbook Type' columns that\n",
      "        can be used to identify the workbooks to pull. This is usually created\n",
      "        via a call to spy.workbooks.search(). Alternatively, you can supply a\n",
      "        workbook ID directly as a str or the URL of a Seeq Workbench worksheet\n",
      "        as a str.\n",
      "    \n",
      "    resume : bool, default True\n",
      "        True if the pull should resume from where it left off, False if it\n",
      "        should pull everything again.\n",
      "    \n",
      "    include_referenced_workbooks : bool, default True\n",
      "        If True, Analyses that are depended upon by Topics will be\n",
      "        automatically included in the resulting list even if they were not part\n",
      "        of the workbooks_df DataFrame.\n",
      "    \n",
      "    include_rendered_content : bool, default False\n",
      "        If True, any Organizer Topics pulled will include rendered content\n",
      "        images, which will cause spy.workbooks.save() to include a folder for\n",
      "        each Topic Document with its HTML and images such that it can be loaded\n",
      "        and viewed in a browser.\n",
      "    \n",
      "    errors : {'raise', 'catalog'}, default 'raise'\n",
      "        If 'raise', any errors encountered will cause an exception. If\n",
      "        'catalog', errors will be added to a 'Result' column in the status.df\n",
      "        DataFrame (errors='catalog' must be combined with\n",
      "        status=<Status object>).\n",
      "    \n",
      "    quiet : bool\n",
      "        If True, suppresses progress output. Note that when status is\n",
      "        provided, the quiet setting of the Status object that is passed\n",
      "        in takes precedence.\n",
      "    \n",
      "    status : spy.Status, optional\n",
      "        If specified, the supplied Status object will be updated as the command\n",
      "        progresses. It gets filled in with the same information you would see\n",
      "        in Jupyter in the blue/green/red table below your code while the\n",
      "        command is executed. The table itself is accessible as a DataFrame via\n",
      "        the status.df property.\n",
      "    \n",
      "    session : spy.Session, optional\n",
      "        If supplied, the Session object (and its Options) will be used to\n",
      "        store the login session state. This is useful to log in to different\n",
      "        Seeq servers at the same time or with different credentials.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(spy.workbooks.job.pull)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "72ed1c0c-46fd-4177-9262-7d748e3631c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function push in module seeq.spy.workbooks.job._push:\n",
      "\n",
      "push(job_folder, *, resume: 'bool' = True, path: 'str' = None, owner: 'str' = None, label: 'str' = None, datasource=None, use_full_path: 'bool' = False, access_control: 'str' = None, override_max_interp: 'bool' = False, scope_globals_to_workbook: 'bool' = False, create_dummy_items: 'bool' = False, errors: 'Optional[str]' = None, quiet: 'Optional[bool]' = None, status: 'Optional[Status]' = None, session: 'Optional[Session]' = None) -> 'pd.DataFrame'\n",
      "    Pushes the definitions for each workbook that was pulled by the\n",
      "    spy.workbooks.job.pull() function, in a restartable \"job\"-like fashion.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    resume : bool, default True\n",
      "        True if the push should resume from where it left off, False if it\n",
      "        should push everything again.\n",
      "    \n",
      "    path : str, default None\n",
      "        A '>>'-delimited folder path to create to contain the workbooks. Note\n",
      "        that a further subfolder hierarchy will be created to preserve the\n",
      "        relative paths that the folders were in when they were searched for\n",
      "        and pulled. If you specify None, then the workbook will stay where\n",
      "        it is (if it has already been pushed once).\n",
      "    \n",
      "        If you specify spy.workbooks.MY_FOLDER, it will be moved to the user's\n",
      "        home folder.\n",
      "    \n",
      "        If you specify a folder ID directly, it will be pushed to that folder.\n",
      "    \n",
      "        If you specify spy.workbooks.ORIGINAL_FOLDER, it will be pushed to the\n",
      "        folder it was in when it was originally pulled. The folder hierarchy\n",
      "        will be recreated on the target server if it doesn't already exist.\n",
      "        (You must be an admin to use this to ensure you have permissions to\n",
      "        put things where they need to go.)\n",
      "    \n",
      "    owner : str, default None\n",
      "        Determines the ownership of pushed workbooks and folders.\n",
      "    \n",
      "        By default, the current owner will be preserved. If the content doesn't\n",
      "        exist yet, the logged-in user will be the owner.\n",
      "    \n",
      "        All other options require that the logged-in user is an admin:\n",
      "    \n",
      "        If spy.workbooks.ORIGINAL_OWNER, ownership is assigned according to the\n",
      "        original owner of the pulled content.\n",
      "    \n",
      "        If spy.workbooks.FORCE_ME_AS_OWNER, existing content will be\n",
      "        reassigned to the logged-in user.\n",
      "    \n",
      "        If a username or a user's Seeq ID is supplied, that user will be\n",
      "        assigned as owner.\n",
      "    \n",
      "        You may need to supply an appropriate datasource map if the usernames\n",
      "        are different between the original and the target servers.\n",
      "    \n",
      "    label : str\n",
      "        A user-defined label that differentiates this push operation from\n",
      "        others. By default, the label will be the logged-in user's username\n",
      "        OR the username from the 'owner' argument so that push activity will\n",
      "        generally be isolated by user. But you can override this with a label\n",
      "        of your choosing.\n",
      "    \n",
      "    datasource : str, optional, default 'Seeq Data Lab'\n",
      "        The name of the datasource within which to contain all the pushed items.\n",
      "        Items inherit access control permissions from their datasource unless it\n",
      "        has been overridden at a lower level. If you specify a datasource using\n",
      "        this argument, you can later manage access control (using spy.acl functions)\n",
      "        at the datasource level for all the items you have pushed.\n",
      "    \n",
      "        If you instead want access control for your items to be inherited from the\n",
      "        workbook they are scoped to, specify `spy.INHERIT_FROM_WORKBOOK`.\n",
      "    \n",
      "    use_full_path : bool, default False\n",
      "        If True, the original full path for an item is reconstructed, as\n",
      "        opposed to the path that is relative to the Path property supplied to\n",
      "        the spy.workbooks.search() call that originally helped create these\n",
      "        workbook definitions. Note that this full path will still be inside\n",
      "        the folder specified by the 'path' argument, if supplied.\n",
      "    \n",
      "    access_control : str, default None\n",
      "        Specifies how Access Control Lists should be treated, via the\n",
      "        following keywords: add/replace,loose/strict\n",
      "    \n",
      "        - If None, then no access control entries are pushed.\n",
      "        - If 'add', then existing access control entries will not be disturbed\n",
      "          but new entries will be added.\n",
      "        - If 'replace', then existing access control entries will be removed\n",
      "          and replaced with the entries from workbook definitions.\n",
      "        - If 'loose', then any unmapped users/groups from the workbook\n",
      "          definitions will be silently ignored.\n",
      "        - If 'strict', then any unmapped users/groups will result in errors.\n",
      "    \n",
      "        Example: access_control='replace,loose'\n",
      "    \n",
      "    override_max_interp : bool, default False\n",
      "        If True, then the Maximum Interpolation overrides from the source\n",
      "        system will be written to the destination system.\n",
      "    \n",
      "    scope_globals_to_workbook : bool, default False\n",
      "        If include_inventory=True and scope_globals_to_workbook=True, then\n",
      "        all globally-scoped (aka \"Available outside this analysis\") items\n",
      "        will be scoped to the workbook. False if you want to actually push\n",
      "        these items to the global scope.\n",
      "    \n",
      "    create_dummy_items : bool, default False\n",
      "        If true, then \"dummy\" items will be created for any stored items\n",
      "        that are not successfully mapped to the target system. This should\n",
      "        be specified as True if you intend to use the spy.workbooks.job.data\n",
      "        module. The dummy items will be created in the target system's\n",
      "        \"Seeq Data Lab\" datasource and will have the same name as the original\n",
      "        item, and no data. They will be scoped to a workbook under a\n",
      "        \"SPy Workbook Jobs\" folder and named after the job folder.\n",
      "    \n",
      "    errors : {'raise', 'catalog'}, default 'raise'\n",
      "        If 'raise', any errors encountered will cause an exception. If\n",
      "        'catalog', errors will be added to a 'Result' column in the status.df\n",
      "        DataFrame (errors='catalog' must be combined with\n",
      "        status=<Status object>).\n",
      "    \n",
      "    quiet : bool\n",
      "        If True, suppresses progress output. Note that when status is\n",
      "        provided, the quiet setting of the Status object that is passed\n",
      "        in takes precedence.\n",
      "    \n",
      "    status : spy.Status, optional\n",
      "        If specified, the supplied Status object will be updated as the command\n",
      "        progresses. It gets filled in with the same information you would see\n",
      "        in Jupyter in the blue/green/red table below your code while the\n",
      "        command is executed. The table itself is accessible as a DataFrame via\n",
      "        the status.df property.\n",
      "    \n",
      "    session : spy.Session, optional\n",
      "        If supplied, the Session object (and its Options) will be used to\n",
      "        store the login session state. This is useful to log in to different\n",
      "        Seeq servers at the same time or with different credentials.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(spy.workbooks.job.push)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1b2b7df8-61f9-427c-9cad-f210056e201d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function pull in module seeq.spy.workbooks.job.data._pull:\n",
      "\n",
      "pull(job_folder, *, resume: 'bool' = True, errors: 'Optional[str]' = None, quiet: 'Optional[bool]' = None, status: 'Optional[Status]' = None, session: 'Optional[Session]' = None) -> 'pd.DataFrame'\n",
      "    Pulls all the data that is used by the workbooks according to the Data\n",
      "    Usages sections of the data_usage.json file in the job folder.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    resume : bool, default True\n",
      "        True if the pull should resume from where it left off, False if it\n",
      "        should pull everything again.\n",
      "    \n",
      "    errors : {'raise', 'catalog'}, default 'raise'\n",
      "        If 'raise', any errors encountered will cause an exception. If\n",
      "        'catalog', errors will be added to a 'Result' column in the status.df\n",
      "        DataFrame (errors='catalog' must be combined with\n",
      "        status=<Status object>).\n",
      "    \n",
      "    quiet : bool\n",
      "        If True, suppresses progress output. Note that when status is\n",
      "        provided, the quiet setting of the Status object that is passed\n",
      "        in takes precedence.\n",
      "    \n",
      "    status : spy.Status, optional\n",
      "        If specified, the supplied Status object will be updated as the command\n",
      "        progresses. It gets filled in with the same information you would see\n",
      "        in Jupyter in the blue/green/red table below your code while the\n",
      "        command is executed. The table itself is accessible as a DataFrame via\n",
      "        the status.df property.\n",
      "    \n",
      "    session : spy.Session, optional\n",
      "        If supplied, the Session object (and its Options) will be used to\n",
      "        store the login session state. This is useful to log in to different\n",
      "        Seeq servers at the same time or with different credentials.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(spy.workbooks.job.data.pull)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6fc23cd9-894d-4ca5-aa02-3b1a5f9473c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function manifest in module seeq.spy.workbooks.job.data._pull:\n",
      "\n",
      "manifest(job_folder, *, reset=False)\n",
      "    Generates and returns a DataFrame with the list of items and data to be\n",
      "    pulled. The manifest is initially generated by spy.workbooks.job.pull(),\n",
      "    and is constructed by examining all of the Analyses and Topics that \"touch\"\n",
      "    a signal or condition and noting the display ranges at play.\n",
      "    \n",
      "    You can modify the manifest using its sibling functions: expand(), add()\n",
      "    or remove().\n",
      "    \n",
      "    The DataFrame returned is in a format suitable for spy.pull(), but in\n",
      "    general it is expected that you will just use\n",
      "    spy.workbooks.job.data.pull(), which has all the resume-ability of the\n",
      "    spy.workbooks.job family of functions.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    reset: {bool}, default False\n",
      "        If True, the manifest will be reset to the original state (i.e. all\n",
      "        modifications made by expand(), add(), and remove() will be\n",
      "        undone).\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    pandas.DataFrame\n",
      "        A DataFrame with the list of items that would be pulled by\n",
      "        spy.workbooks.data.pull(). Note the presence of \"Start\", \"End\" and\n",
      "        \"Calculation\" columns:\n",
      "    \n",
      "        Start        The earliest timestamp that will be requested for the item\n",
      "        End          The latest timestamp that will be requested for the item\n",
      "        Calculation  A within() or touches() calculation that will be used to\n",
      "                     more precisely request the non-contiguous time periods\n",
      "                     represented by the manifest.\n",
      "\n",
      "Help on function expand in module seeq.spy.workbooks.job.data._pull:\n",
      "\n",
      "expand(job_folder: 'str', items: 'Union[str, list, dict]' = None, *, by: 'Union[str, timedelta, pd.Timedelta]' = None, start_by: 'Union[str, timedelta, pd.Timedelta]' = None, end_by: 'Union[str, timedelta, pd.Timedelta]' = None)\n",
      "    Expands the start and end of the time periods to be pulled (by\n",
      "    spy.workbooks.job.data.pull) by the specified amount.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    items : {str, list, dict}, optional\n",
      "        The items to affect in the data manifest.\n",
      "    \n",
      "        If not specified, all items are affected.\n",
      "    \n",
      "        If a string, it is interpreted as an item ID.\n",
      "    \n",
      "        If a dict, it is interpreted as a query to match\n",
      "        against the properties of each item. The keys of the dict are\n",
      "        property names and the values are the matching criteria, and\n",
      "        can include wildcards (*, ?) and regular expressions (aka Regex).\n",
      "        Regex expressions must be enclosed in forward slashes (/).\n",
      "    \n",
      "        If a list, it is interpreted as a list of item IDs and/or\n",
      "        dictionaries.\n",
      "    \n",
      "    by : {str, timedelta, pd.Timedelta}, optional\n",
      "        A str, timedelta or pd.Timedelta object specifying the amount of time\n",
      "        to expand both the start and end times of the periods to be pulled.\n",
      "    \n",
      "    start_by : {str, timedelta, pd.Timedelta}, optional\n",
      "        A str, timedelta or pd.Timedelta object specifying the amount of time\n",
      "        to expand the start times of the periods to be pulled.\n",
      "    \n",
      "    end_by : {str, timedelta, pd.Timedelta}, optional\n",
      "        A str, timedelta or pd.Timedelta object specifying the amount of time\n",
      "        to expand the start times of the periods to be pulled.\n",
      "\n",
      "Help on function add in module seeq.spy.workbooks.job.data._pull:\n",
      "\n",
      "add(job_folder: 'str', items: 'Union[str, list, dict]' = None, *, start: 'Union[str, pd.Timestamp]', end: 'Union[str, pd.Timestamp]' = None)\n",
      "    Adds a time period (start and end) to the list of the time periods to be\n",
      "    pulled (by spy.workbooks.job.data.pull) for a particular item or set of\n",
      "    items.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    items : {str, list, dict}, optional\n",
      "        The items to affect in the data manifest.\n",
      "    \n",
      "        If not specified, all items are affected.\n",
      "    \n",
      "        If a string, it is interpreted as an item ID.\n",
      "    \n",
      "        If a dict, it is interpreted as a query to match\n",
      "        against the properties of each item. The keys of the dict are\n",
      "        property names and the values are the matching criteria, and\n",
      "        can include wildcards (*, ?) and regular expressions (aka Regex).\n",
      "        Regex expressions must be enclosed in forward slashes (/).\n",
      "    \n",
      "        If a list, it is interpreted as a list of item IDs and/or\n",
      "        dictionaries.\n",
      "    \n",
      "    start : {str, pd.Timestamp}\n",
      "        The starting time for which to pull data. This argument must be a\n",
      "        string that pandas.to_datetime() can parse, or a pandas.Timestamp.\n",
      "    \n",
      "    end : {str, pd.Timestamp}, optional\n",
      "        The end time for which to pull data. This argument must be a string\n",
      "        that pandas.to_datetime() can parse, or a pandas.Timestamp.\n",
      "        If not provided, 'end' will default to now.\n",
      "\n",
      "Help on function remove in module seeq.spy.workbooks.job.data._pull:\n",
      "\n",
      "remove(job_folder: 'str', items: 'Union[str, list, dict]' = None, *, start: 'Union[str, pd.Timestamp]' = None, end: 'Union[str, pd.Timestamp]' = None)\n",
      "    Removes a time period (start and end) to the set of the time periods to be\n",
      "    pulled (by spy.workbooks.job.data.pull) for a particular item or set of\n",
      "    items.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    items : {str, list, dict}, optional\n",
      "        The items to affect in the data manifest.\n",
      "    \n",
      "        If not specified, all items are affected.\n",
      "    \n",
      "        If a string, it is interpreted as an item ID.\n",
      "    \n",
      "        If a dict, it is interpreted as a query to match\n",
      "        against the properties of each item. The keys of the dict are\n",
      "        property names and the values are the matching criteria, and\n",
      "        can include wildcards (*, ?) and regular expressions (aka Regex).\n",
      "        Regex expressions must be enclosed in forward slashes (/).\n",
      "    \n",
      "        If a list, it is interpreted as a list of item IDs and/or\n",
      "        dictionaries.\n",
      "    \n",
      "    start : {str, pd.Timestamp}, optional\n",
      "        The starting time for the time period to remove. This argument must\n",
      "        be a string that pandas.to_datetime() can parse, or a pandas.Timestamp.\n",
      "        If not provided, 'start' will default to \"the beginning of time,\"\n",
      "        meaning that all time prior to 'end' will be removed. (If neither\n",
      "        'start' nor 'end' are specified, no data for the item will be pulled.)\n",
      "    \n",
      "    end : {str, pd.Timestamp}, optional\n",
      "        The end time for the time period to remove. This argument must be a\n",
      "        string that pandas.to_datetime() can parse, or a pandas.Timestamp.\n",
      "        If not provided, 'end' will default to \"the end of time,\" meaning that\n",
      "        all time after 'start' will be removed. (If neither 'start' nor 'end'\n",
      "        are specified, no data for the item will be pulled.)\n",
      "    \n",
      "    session : spy.Session, optional\n",
      "        If supplied, the Session object (and its Options) will be used to\n",
      "        store the login session state. This is useful to log in to different\n",
      "        Seeq servers at the same time or with different credentials.\n",
      "\n",
      "Help on function calculation in module seeq.spy.workbooks.job.data._pull:\n",
      "\n",
      "calculation(job_folder: 'str', items: 'Union[str, list, dict]' = None, *, formula: 'str' = None)\n",
      "    Apply a specific formula to the items when pulling. For example,\n",
      "    you may wish to specify formula=\"resample($signal, 1h)\" to\n",
      "    reduce the density of a signal. This calculation will be applied\n",
      "    for all time periods that are pulled for the item.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull().\n",
      "    \n",
      "    items : {str, list, dict}, optional\n",
      "        The items to affect in the data manifest.\n",
      "    \n",
      "        If not specified, all items are affected.\n",
      "    \n",
      "        If a string, it is interpreted as an item ID.\n",
      "    \n",
      "        If a dict, it is interpreted as a query to match\n",
      "        against the properties of each item. The keys of the dict are\n",
      "        property names and the values are the matching criteria, and\n",
      "        can include wildcards (*, ?) and regular expressions (aka Regex).\n",
      "        Regex expressions must be enclosed in forward slashes (/).\n",
      "    \n",
      "        If a list, it is interpreted as a list of item IDs and/or\n",
      "        dictionaries.\n",
      "    \n",
      "    formula : {str}\n",
      "        The calculation to apply to the set of items. For signals, the\n",
      "        formula must contain a reference to $signal, for conditions it\n",
      "        must contain a reference to $condition.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(spy.workbooks.job.data.manifest)\n",
    "help(spy.workbooks.job.data.expand)\n",
    "help(spy.workbooks.job.data.add)\n",
    "help(spy.workbooks.job.data.remove)\n",
    "help(spy.workbooks.job.data.calculation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "493884fb-147e-4ed5-8692-ea63f195ba09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function push in module seeq.spy.workbooks.job.data._push:\n",
      "\n",
      "push(job_folder, *, resume: 'bool' = True, replace: 'Optional[dict]' = None, datasource: 'str' = None, errors: 'Optional[str]' = None, quiet: 'Optional[bool]' = None, status: 'Optional[Status]' = None, session: 'Optional[Session]' = None) -> 'pd.DataFrame'\n",
      "    Pulls all the data that is used by the workbooks according to the Data\n",
      "    Usages sections of the data_usage.json file in the job folder.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder created by\n",
      "        spy.workbooks.job.pull() and populated with data by\n",
      "        spy.workbooks.job.pull_data().\n",
      "    \n",
      "    resume : bool, default True\n",
      "        True if the pull should resume from where it left off, False if it\n",
      "        should pull everything again.\n",
      "    \n",
      "    replace : dict, default None\n",
      "        A dict with the keys 'Start' and 'End'. If provided, any existing samples\n",
      "        or capsules with the start date in the provided time period will be\n",
      "        replaced. The start of the time period is inclusive and the end of the\n",
      "        time period is exclusive. If replace is provided but data is not\n",
      "        specified, all samples/capsules within the provided time period will be\n",
      "        removed.\n",
      "    \n",
      "    datasource : str, optional, default 'Seeq Data Lab'\n",
      "        The name of the datasource within which to contain all the pushed items.\n",
      "        Items inherit access control permissions from their datasource unless it\n",
      "        has been overridden at a lower level. If you specify a datasource using\n",
      "        this argument, you can later manage access control (using spy.acl functions)\n",
      "        at the datasource level for all the items you have pushed.\n",
      "    \n",
      "        If you instead want access control for your items to be inherited from the\n",
      "        workbook they are scoped to, specify `spy.INHERIT_FROM_WORKBOOK`.\n",
      "    \n",
      "    errors : {'raise', 'catalog'}, default 'raise'\n",
      "        If 'raise', any errors encountered will cause an exception. If\n",
      "        'catalog', errors will be added to a 'Result' column in the status.df\n",
      "        DataFrame (errors='catalog' must be combined with\n",
      "        status=<Status object>).\n",
      "    \n",
      "    quiet : bool\n",
      "        If True, suppresses progress output. Note that when status is\n",
      "        provided, the quiet setting of the Status object that is passed\n",
      "        in takes precedence.\n",
      "    \n",
      "    status : spy.Status, optional\n",
      "        If specified, the supplied Status object will be updated as the command\n",
      "        progresses. It gets filled in with the same information you would see\n",
      "        in Jupyter in the blue/green/red table below your code while the\n",
      "        command is executed. The table itself is accessible as a DataFrame via\n",
      "        the status.df property.\n",
      "    \n",
      "    session : spy.Session, optional\n",
      "        If supplied, the Session object (and its Options) will be used to\n",
      "        store the login session state. This is useful to log in to different\n",
      "        Seeq servers at the same time or with different credentials.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(spy.workbooks.job.data.push)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "d83e8b49",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function redo in module seeq.spy.workbooks.job._redo:\n",
      "\n",
      "redo(job_folder: 'str', workbooks_df: 'Union[pd.DataFrame, str, list]', action: 'Optional[str]' = None, *, quiet: 'Optional[bool]' = None, status: 'Optional[Status]' = None)\n",
      "    Creates a zip file of the job folder for easy sharing.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder to be zipped.\n",
      "    \n",
      "    workbooks_df : {pd.DataFrame, str, list}\n",
      "        A DataFrame containing an 'ID' column that can be used to identify the\n",
      "        workbooks to affect. These IDs are based on the source system (not the\n",
      "        destination system). Alternatively, you can supply a workbook ID\n",
      "        directly as a str or list of strs.\n",
      "    \n",
      "    action : str\n",
      "        If supplied, limits the redo to the specified actions. You can specify\n",
      "        'pull' or 'push'. If not supplied, both pull and push are affected.\n",
      "        Note that 'pull' automatically includes 'push'.\n",
      "    \n",
      "    quiet : bool\n",
      "        If True, suppresses progress output. Note that when status is\n",
      "        provided, the quiet setting of the Status object that is passed\n",
      "        in takes precedence.\n",
      "    \n",
      "    status : spy.Status, optional\n",
      "        If specified, the supplied Status object will be updated as the command\n",
      "        progresses. It gets filled in with the same information you would see\n",
      "        in Jupyter in the blue/green/red table below your code while the\n",
      "        command is executed.\n",
      "\n",
      "Help on function redo in module seeq.spy.workbooks.job.data._redo:\n",
      "\n",
      "redo(job_folder: 'str', items_df: 'Union[pd.DataFrame, str, list]', action: 'Optional[str]' = None, *, quiet: 'Optional[bool]' = None, status: 'Optional[Status]' = None)\n",
      "    Creates a zip file of the job folder for easy sharing.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    job_folder : {str}\n",
      "        A full or partial path to the job folder to be zipped.\n",
      "    \n",
      "    items_df : {pd.DataFrame, str, list}\n",
      "        A DataFrame containing an 'ID' column that can be used to identify\n",
      "        the data items to affect. These IDs are based on the source system\n",
      "        (not the destination system). Alternatively, you can supply an item\n",
      "        ID directly as a str or list of strs.\n",
      "    \n",
      "    action : str\n",
      "        If supplied, limits the redo to the specified actions. You can specify\n",
      "        'pull' or 'push'. If not supplied, both pull and push are affected.\n",
      "        Note that 'pull' automatically includes 'push'.\n",
      "    \n",
      "    quiet : bool\n",
      "        If True, suppresses progress output. Note that when status is\n",
      "        provided, the quiet setting of the Status object that is passed\n",
      "        in takes precedence.\n",
      "    \n",
      "    status : spy.Status, optional\n",
      "        If specified, the supplied Status object will be updated as the command\n",
      "        progresses. It gets filled in with the same information you would see\n",
      "        in Jupyter in the blue/green/red table below your code while the\n",
      "        command is executed.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(spy.workbooks.job.redo)\n",
    "help(spy.workbooks.job.data.redo)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.11",
   "language": "python",
   "name": "python311"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
