{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LLM for alle - Introduksjonskurs til språkmodeller med Python og Azure OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduksjon\n",
    "\n",
    "Under et prosjekt vi gjorde da vi satt på benk i høst fikk vi hands-on erfaring med å utvikle og bruke språkmodeller (LLMer) i praksis. Vi så hvordan slike modeller kan hjelpe med å spare tid, redusere kostnader og automatisere manuelle oppgaver på en måte som enkelt kan gjenbrukes i andre prosjekter.\n",
    "\n",
    "I dette kurset ser vi på et konkret og gjenkjennbart case: analyse av resultatene fra en medarbeiderundersøkelse. Dette er en oppgave som kan være vanskelig å strukturere og som kan være ekstremt tidkrevende å gjøre for hånd når det er store volum av tilbakemeldinger. Vi skal lære dere å koble dere til en LLM via API, bruke LangChain Expression Language (LCEL) til å bygge AI-kjeder og hente ut strukturert innsikt med Pydantic, alt i én oversiktlig notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Oppgave\n",
    "Du jobber i et IT-selskap og har fått i oppgave å analysere svarene fra en intern medarbeiderundersøkelse. Undersøkelsen er anonym og du har fått tilsendt en CSV-fil med 50 tilbakemeldinger, én per ansatt. Målet er å finne ut hva folk er fornøyde eller misfornøyde med, og særlig se nærmere på temaene Nettverk, Opplæring og IT-support, som ledelsen har gitt beskjed om at har vært sentrale temaer i undersøkelser de har sendt ut tidligere. Tilbakemeldinger som ikke passer i disse kategoriene skal også få sin plass. Til slutt skal du lage en oppsummering som kan sendes til ledelsen.\n",
    "\n",
    "For å jobbe effektivt bruker du en språkmodell til å hjelpe deg med både kategorisering og oppsummering. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasett"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vi har fått rådatasettet levert som en csv-fil hvor vi i utgangspunktet kun er interessert i de 2 kolonnene \"ID\" og \"Feedback\". \"ID\" (int) er en unik id for hver ansatt og \"Feedback\" (str) inneholder tilbakemeldingene. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Feedback</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>The network infrastructure is robust and usually efficient, though intermittent glitches sometimes hamper productivity and clarity.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Our current infrastructure aligns with business needs, yet occasional rigidity in legacy systems hinders innovative approaches.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>I appreciate the proactive approach to security, though the multi-step authentication process sometimes seems overly complicated.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Our tools maintain high quality with intuitive design and regular updates, even though rare performance lags can hinder productivity.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>The IT-support team is consistently prompt and courteous, though occasional miscommunications lead to fleeting moments of uncertainty.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID  \\\n",
       "0   1   \n",
       "1   2   \n",
       "2   3   \n",
       "3   4   \n",
       "4   5   \n",
       "\n",
       "                                                                                                                                 Feedback  \n",
       "0     The network infrastructure is robust and usually efficient, though intermittent glitches sometimes hamper productivity and clarity.  \n",
       "1         Our current infrastructure aligns with business needs, yet occasional rigidity in legacy systems hinders innovative approaches.  \n",
       "2       I appreciate the proactive approach to security, though the multi-step authentication process sometimes seems overly complicated.  \n",
       "3   Our tools maintain high quality with intuitive design and regular updates, even though rare performance lags can hinder productivity.  \n",
       "4  The IT-support team is consistently prompt and courteous, though occasional miscommunications lead to fleeting moments of uncertainty.  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the raw data containing the feedback. There is one row per employee.\n",
    "\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_colwidth', None) # Ensure no truncated output of dataframe\n",
    "\n",
    "path_to_survey = \"../files/IT_survey.csv\"\n",
    "df = pd.read_csv(\n",
    "    path_to_survey,\n",
    "    usecols = [\"ID\", \"Feedback\"],       # Extract relevant columns\n",
    "    dtype={\"ID\": int, \"Feedback\": str}  # Specify expected types\n",
    ")\n",
    "\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Før vi går løs på oppgaven skal du bli bedre kjent med verktøyene og metodene vi skal bruke."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Koble til Azure OpenAI via LangChain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### En vanlig LLM-spørring\n",
    "Enkle LLM-spørringer er bygget opp av noen sentrale deler:\n",
    "\n",
    "1. Tilkobling til en ressurs, som feks. Azure OpenAI\n",
    "\n",
    "2. En prompt, som vil si en tekstbasert forespørsel/instruks\n",
    "\n",
    "3. Sending av prompt til språkmodellen for å hente en respons\n",
    "\n",
    "\n",
    "Som en del av tilkoblingen er det vanlig å oppgi en temperaturparameter. Denne parametreren angir nivået av presisjon du ønsker å få i responsen fra språkmodellen, og kan måles numerisk fra 0 til 1.\n",
    "Hvis denne parameteren settes nærme 0 tillater du liten grad av variasjon og kreativitet i responsen, og du vil få tryggere og mer forutsigbare svar. Hvis den derimot settes nærme 1 tillater du større grad av kreativitet og detaljer, men vil følgelig også få en mer uforutsigbar respons. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect through the API\n",
    "from langchain_openai import AzureChatOpenAI\n",
    "from dotenv import find_dotenv, load_dotenv\n",
    "import os\n",
    "from IPython.display import Markdown # Pretty output\n",
    "\n",
    "\n",
    "# Get environment variables\n",
    "load_dotenv(find_dotenv(), override=True)\n",
    "\n",
    "\n",
    "llm = AzureChatOpenAI(\n",
    "    azure_deployment=os.getenv(\"AZURE_OPENAI_DEPLOYMENT_GPT_4O_MINI\"),\n",
    "    model=os.getenv(\"OPENAI_MODEL_GPT_4O-MINI\", default=\"gpt-4o-mini\"),\n",
    "    temperature=0,\n",
    ")\n",
    "\n",
    "reasoning_llm = AzureChatOpenAI(\n",
    "    azure_deployment=os.getenv(\"AZURE_OPENAI_DEPLOYMENT_O4_MINI\"),\n",
    "    model=os.getenv(\"OPENAI_MODEL_O4-MINI\", default=\"o4-mini\"),\n",
    "    reasoning_effort=\"medium\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test modellen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='Hei! Hvordan kan jeg hjelpe deg i dag?', additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 11, 'prompt_tokens': 9, 'total_tokens': 20, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_7a53abb7a2', 'prompt_filter_results': [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}], 'finish_reason': 'stop', 'logprobs': None, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}, id='run-2f347b32-98e4-4333-ad44-62fef6193a90-0', usage_metadata={'input_tokens': 9, 'output_tokens': 11, 'total_tokens': 20, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate the prompt\n",
    "prompt = 'Hei!'\n",
    "\n",
    "# Send the prompt and recieve a response\n",
    "response = llm.invoke(prompt)\n",
    "\n",
    "# Show the response from the model\n",
    "response\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hei! Hvordan kan jeg hjelpe deg i dag?'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show only the content of the response\n",
    "response.content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lek deg rundt med temperatur-parameteren"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Once, a cat named Whiskers decided he wanted to impress his humans by catching a mouse. He stealthily crept into the kitchen, only to find the family hamster, Mr. Nibbles, staring back at him. Mistaking the hamster for a mouse, Whiskers pounced, only to have Mr. Nibbles leap onto his head and start spinning like a tiny furry tornado. In a panic, Whiskers darted around the house, creating chaos, while the humans laughed and snapped pictures. From that day on, Whiskers became known as the cat who was defeated by a hamster, forever earning the nickname \"Cat-tastrophe.\"\n"
     ]
    }
   ],
   "source": [
    "# Test out the temperature-parameter\n",
    "\n",
    "# Define the prompt\n",
    "prompt = 'Tell me a funny story about a cat. Write max. 5 sentences.'\n",
    "\n",
    "# Set temperature (0 = most predictable, 1 = most creative and random)\n",
    "temperature = 0.9  # Try to switch this out with 0.1 and 0.5 to see the difference. \n",
    "\n",
    "# Send the prompt and receive a response\n",
    "response = llm.invoke(prompt, temperature=temperature)\n",
    "\n",
    "# Print the response from the model\n",
    "print(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hvordan lage en enkel AI-kjede med LangChain Expression Language (LCEL)\n",
    " \n",
    "LCEL er en metode for å bygge og kjøre såkalte *kjeder* i LangChain på. Kjeder, eller chains, brukes for å koble sammen ulike AI-komponenter. F.eks. kan språkmodeller, datakilder og logikk kobles sammen til én sammenhengende prosess, dvs. en kjede. LCEL gir et standardisert språk for å definere disse kjedene, og er brukervennlig fordi man slipper å lage alt manuelt med kode. Med andre ord får du en \"oppskrift\" på hvordan AI-komponentene dine skal jobbe sammen.\n",
    " \n",
    "**Fordelene med LCEL**\n",
    "1. Støtter parallell og asynkron kjøring - Ulike deler av kjeden kan kjøre samtidig, og systemet kan behandle flere forespørsler på en gang. Dermed kan oppgaver behandles raskere.\n",
    "2. Strømming av resultater - Man kan begynne å se svar mens AI-en fremdeles jobber. (Passer spesielt godt for chatbaserte løsninger)\n",
    "3. Enkel feilsøking med LangSmith - Når kjedene blir komplekse er det viktig å kunne se hva som har blitt gjort underveis. LCEL logger automatisk alt til LangSmith, som gjør det enklere å feilsøke.\n",
    "4. Standardisert - Alle kjeder i LCEL bruker samme grensesnitt, som gjør dem enkle å kombinere og gjenbruke på tvers av prosjekter.\n",
    " \n",
    "LCEL bruker en pipe-operator (|) til å koble sammen ulike trinn i kjeden. Den tar ut data fra én komponent og sender den direkte som input til neste komponent. LCEL bruker også PromptTemplate, som kan tenkes på som en mal for teksten du sender til språkmodellen. PromptTemplate gjør det enkelt å lage dynamiske meldinger ved at man kan sette inn variabler i teksten, litt som en oppskrift der du fyller inn det som mangler før det sendes til AI-modellen. Fordelen med PromptTemplate er at man kan lage én mal, og bruke den med ulike data. Det hjelper deg også med å skille selve teksten fra logikken, og kan gjøre prosessen sikrere ved at man unngår feil som kan oppstå ved manuell string-manipulasjon. Vi skal nå se på noen eksempler med LCEL som bruker pipe-operator og PromptTemplate.\n",
    " \n",
    "Kilde: xxx\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PromptTemplate(input_variables=['role'], input_types={}, partial_variables={}, template='\\n    Hi! Please talk like a {role}.\\n    ')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.prompts import PromptTemplate\n",
    "prompt = PromptTemplate.from_template(\n",
    "    \"\"\"\n",
    "    Hi! Please talk like a {role}.\n",
    "    \"\"\"\n",
    ")\n",
    "prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content=\"Ahoy, matey! What be ye wantin' to chat about on this fine day upon the high seas? Be it treasure maps, fearsome sea monsters, or the finest grog in the land? Speak up, or I’ll be sendin’ ye to Davy Jones’ locker! Arrr! 🏴\\u200d☠️\", additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 72, 'prompt_tokens': 18, 'total_tokens': 90, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_7a53abb7a2', 'prompt_filter_results': [{'prompt_index': 0, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}], 'finish_reason': 'stop', 'logprobs': None, 'content_filter_results': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}, id='run-9adc02c4-bc35-413b-a8d0-42b68fab1b29-0', usage_metadata={'input_tokens': 18, 'output_tokens': 72, 'total_tokens': 90, 'input_token_details': {'audio': 0, 'cache_read': 0}, 'output_token_details': {'audio': 0, 'reasoning': 0}})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain = prompt | llm\n",
    "\n",
    "chain.invoke({\"role\": \"pirate\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Som dere ser returnerer modellen mye mer enn kun svaret på prompten vår, nemlig en hel haug med meta-data. Ofte er man kun interessert i tekst-responsen fra modellen og da kan man bruke StrOutputParser for å redusere den unødvendige støyen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Ahoy, matey! What be ye wantin' to chat about on this fine day upon the high seas? Arrr!\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "chain_str = chain | StrOutputParser()\n",
    "\n",
    "chain_str.invoke({\"role\": \"pirate\"})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vi vil også introdusere et annet nyttig verktøy, nemlig batching. Så langt har vi brukt 'invoke' for å sende prompten til modellen. Den tar imot en input og returnerer en output. Batch forventer en liste med input og returner en liste med output. Dette er en mye mer effektiv metode når du vat at du øsnker å gjøre flere kall på modellen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Ahoy, matey! What be ye wantin' to chat about on this fine day upon the high seas? Be it treasure maps, fearsome sea monsters, or the finest grog in the land? Speak up, or I’ll be sendin’ ye to Davy Jones’ locker! Arrr! 🏴\\u200d☠️\",\n",
       " \"Well howdy there, partner! What brings ya 'round these parts? If yer lookin' to chew the fat or spin a yarn, I’m all ears. Just remember, out here in the wild west, we value a good ol' fashioned chat as much as a cool drink on a hot day. So saddle up and let’s rustle up some conversation! Yeehaw!\",\n",
       " 'Greetings, silent shadow. The night whispers secrets, and the wind carries the tales of the unseen. What knowledge do you seek, traveler of the hidden path? Speak swiftly, for the moon watches over us. 🥷🌙']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain_str.batch(\n",
    "    [\n",
    "        {\"role\": \"pirate\"},\n",
    "        {\"role\": \"cowboy\"},\n",
    "        {\"role\": \"ninja\"},\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prøv selv - Bygg din egen mal\n",
    "Finn på en prompt med variabler som f.eks\n",
    "- Finner antall inbyggere i land X i år Y\n",
    " - Finner forventet levealder på hunderase X\n",
    " - Gir modellen en rolle X (assistent, detektiv, lege etc) og ber den forklare deg konsept Y\n",
    " - Gjør noe helt sykt, her setter kun fantasien din grenser!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sure! What would you like to discuss or explore today? You can ask me anything or give me a specific topic to work with!'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# PromptTemplate as an email-template. \n",
    "\n",
    "prompt = PromptTemplate.from_template(\n",
    "    \"\"\"\n",
    "    Your prompt :)\n",
    "    \"\"\"\n",
    ")  \n",
    "\n",
    "chain = prompt | llm\n",
    "chain_str = chain | StrOutputParser()\n",
    "\n",
    "chain_str.invoke({\"variabel_1\":...}) # Fill out the blanks!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kategorisering av hver enkelt tilbakemelding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bruk LLMen til å kategorisere tilbakemeldingene\n",
    "I første omgang av kategoriseringen er vi interesserte i å se hvor mange av tilbakemeldingene som passer innenfor de kategoriene ledelsen foreslo, nemlig Network, Training og IT-support. Tilbakemeldinger modellen mener at ikke passer i noen av disse forhåndsbestemte kategoriene vil vi samle opp i en 'Other' kategori."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Category: IT-support'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorize_prompt = PromptTemplate.from_template(\n",
    "\"\"\"\n",
    "Categorize the following feedback into one of the following categories:\n",
    "- Network\n",
    "- Training\n",
    "- IT-support\n",
    "- Other\n",
    "\n",
    "Feedback:\n",
    "<feedback>\n",
    "{feedback}\n",
    "</feedback>\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "categorize_chain = categorize_prompt | llm | StrOutputParser()\n",
    "\n",
    "categorize_chain.invoke({\"feedback\": \"I am very happy with the IT support I received last week.\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datavalidering og strukturering med Pydantic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Når vi jobber med store språkmodeller (LLMs), er det viktig å kunne styre og validere output. Vi ønsker ikke bare et sva, men et svar på riktig format.\n",
    "Derfor bruker vi verktøy som Pydantic, som hjelper oss med:\n",
    "- Å strukturere data på en tydelig og eksplisitt måte\n",
    "- Å validere at data har riktig type og verdi\n",
    "- Å tvinge output fra LLM til å følge en bestemt mal (for eksempel en JSON-struktur)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hva er Pydantic?\n",
    "Pydantic er et Python-bibliotek for datavalidering basert på Python type hints. Hovedklassen er BaseModel, som vi arver fra når vi definerer vår egen datastruktur.\n",
    "Pydantic gir oss:\n",
    "1. Datavalidering\n",
    "    - Du kan begrense hva som er lovlige verdier. F.eks: Godkjente land = [\"Norge\", \"Sverige\", \"Finland\"] → Australia blir avvist.\n",
    "2. Typekonvertering\n",
    "    - Hvis du sier at du forventer en int og sender inn \"1\" (str), vil den konverteres automatisk.\n",
    "3. Strukturell kontroll\n",
    "    - Du kan bruke Pydantic-modellen til å validere at output fra LLM matcher en bestemt struktur, som JSON."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eksempel: Enkel modell med én kategori "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel, Field\n",
    "\n",
    "class Categorize(BaseModel):\n",
    "    \"Categorization of a single feedback entry from an IT survey.\"\n",
    "    category : str = Field(description=\"The best fitting category. Only one.\") \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Her har vi definert en modell med ett felt: category.\n",
    "Modellen krever at category er en streng og vil derfor avvise andre typer. Dette blir for eksempel ulovlig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValidationError",
     "evalue": "1 validation error for Categorize\ncategory\n  Input should be a valid string [type=string_type, input_value=1, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.10/v/string_type",
     "output_type": "error",
     "traceback": [
      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
      "\u001b[31mValidationError\u001b[39m                           Traceback (most recent call last)",
      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[14]\u001b[39m\u001b[32m, line 1\u001b[39m\n\u001b[32m----> \u001b[39m\u001b[32m1\u001b[39m \u001b[43mCategorize\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcategory\u001b[49m\u001b[43m=\u001b[49m\u001b[32;43m1\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "\u001b[36mFile \u001b[39m\u001b[32m~/GenAI_arshjul/LLM101/.venv/lib/python3.13/site-packages/pydantic/main.py:214\u001b[39m, in \u001b[36mBaseModel.__init__\u001b[39m\u001b[34m(self, **data)\u001b[39m\n\u001b[32m    212\u001b[39m \u001b[38;5;66;03m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[32m    213\u001b[39m __tracebackhide__ = \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[32m--> \u001b[39m\u001b[32m214\u001b[39m validated_self = \u001b[38;5;28;43mself\u001b[39;49m\u001b[43m.\u001b[49m\u001b[43m__pydantic_validator__\u001b[49m\u001b[43m.\u001b[49m\u001b[43mvalidate_python\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdata\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mself_instance\u001b[49m\u001b[43m=\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[32m    215\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m validated_self:\n\u001b[32m    216\u001b[39m     warnings.warn(\n\u001b[32m    217\u001b[39m         \u001b[33m'\u001b[39m\u001b[33mA custom validator is returning a value other than `self`.\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[33m'\u001b[39m\n\u001b[32m    218\u001b[39m         \u001b[33m\"\u001b[39m\u001b[33mReturning anything other than `self` from a top level model validator isn\u001b[39m\u001b[33m'\u001b[39m\u001b[33mt supported when validating via `__init__`.\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[33m\"\u001b[39m\n\u001b[32m    219\u001b[39m         \u001b[33m'\u001b[39m\u001b[33mSee the `model_validator` docs (https://docs.pydantic.dev/latest/concepts/validators/#model-validators) for more details.\u001b[39m\u001b[33m'\u001b[39m,\n\u001b[32m    220\u001b[39m         stacklevel=\u001b[32m2\u001b[39m,\n\u001b[32m    221\u001b[39m     )\n",
      "\u001b[31mValidationError\u001b[39m: 1 validation error for Categorize\ncategory\n  Input should be a valid string [type=string_type, input_value=1, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.10/v/string_type"
     ]
    }
   ],
   "source": [
    "Categorize(category=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mens dette er helt ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Categorize(category='Network')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Categorize(category=\"Network\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```Field()``` kan brukes til å spesifisere beskrivelse, standardverdi, valideringsregler og mer. Vi bruker det her for å gi modellen litt mer kontekst.\n",
    "De spesielt interesserte kan lese mer om funksjonaliteten [her](https://docs.pydantic.dev/latest/concepts/fields/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bruke modellen sammen med LLM\n",
    "Vi bruker en wrapperen ```with_structured_output(...)``` for å fortelle LLMen at den må svare med output matcher en Pydantic-modellen.\n",
    "### Eksempel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Categorize(category='IT-support')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorize_chain_structured_output = categorize_prompt | llm.with_structured_output(\n",
    "    Categorize,\n",
    "    method=\"json_schema\", # Demands JSON-schema for output\n",
    "    strict=True           # The model must adhere to the specified schema, no extra fields or missing fields are allowed. All types must be an exact match. \n",
    ")\n",
    "\n",
    "categorize_chain_structured_output.invoke(\n",
    "    {\"feedback\": \"I am very happy with the IT support I received last week.\"}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Begrense mulige verdier med Literal\n",
    "Noen ganger ønsker vi å begrense hvilke verdier som er gyldige, slik som å si at category bare kan være én av fire forhåndsdefinerte valg.\n",
    "Dette gjør vi med ```Literal```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Literal\n",
    "\n",
    "CATEGORIES = Literal[\n",
    "    \"Network\",\n",
    "    \"Training\",\n",
    "    \"IT-support\",\n",
    "    'Other'\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nå kan vi bruke ```CATEGORIES``` som en type i Pydantic-modellen vår. Da vil modellen validere at det bare kommer svar som matcher én av disse fire.\n",
    "### Eksempel: Strukturert output med begrensede kategorier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Literal\n",
    "\n",
    "categorize_prompt = PromptTemplate.from_template(               #Notice that we no longer pass the list of categories with the prompt.\n",
    "\"\"\"\n",
    "Categorize the following feedback into the provided categories. \n",
    "\n",
    "Feedback:\n",
    "<feedback>\n",
    "{feedback}\n",
    "</feedback>\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "\n",
    "\n",
    "class CategorizeFromOptions(BaseModel):\n",
    "    \"Categorization of a single feedback entry from an IT survey.\"\n",
    "    category: CATEGORIES = Field(\n",
    "        description=\"Chosen category for the feedback. Choose 'Other' if the other categories provided are not a good fit.\"  \n",
    "    )\n",
    "\n",
    "\n",
    "categorize_chain_structured_output = categorize_prompt | llm.with_structured_output(\n",
    "    CategorizeFromOptions,\n",
    "    method=\"json_schema\",\n",
    "    strict=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CategorizeFromOptions(category='IT-support')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = categorize_chain_structured_output.invoke(\n",
    "    {\"feedback\": \"I am very happy with the IT support I received last week.\"}\n",
    ")\n",
    "\n",
    "result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nå har vi satt opp en modell hvor vi er sikre på:\n",
    "- At outputen er i riktig format\n",
    "- At category bare kan være en av de forhåndsdefinerte verdiene\n",
    "\n",
    "Dette er utrolig nyttig hvis du skal bruke output videre i en ny prompt, i en database, eller som input til et system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chain-of-thought\n",
    " \n",
    "Chain of Thought (CoT) er en teknikk innen prompt engineering som hjelper språkmodeller med å løse oppgaver som krever flere tankesteg. I stedet for å hoppe rett til svaret, blir modellen ledet gjennom en logisk og trinnvis prosess som kan gi mer presise og gjennomtenkte svar [1].\n",
    " \n",
    "Du kan altså be modellen om å \"tenke høyt\" under oppgaven og forklare stegene sine før den leverer et endelig svar. Dette ber du om i prompten som sendes inn.\n",
    " \n",
    "Eksempel på en prompt **uten** CoT:\n",
    " \n",
    "    Prompt: \"Hvor mange armer har Eline og Kaspara?\"\n",
    " \n",
    "    Svar: \"4\"\n",
    " \n",
    "Eksempel på en prompt **med** CoT:\n",
    " \n",
    "    Prompt: \"Hvor mange armer har ELine og Kaspara? Tenk trinn for trinn.\"\n",
    " \n",
    "    Svar: \"En person har to armer. To personer betyr 2x2 = 4 armer. Svaret er 4.\"\n",
    " \n",
    "Det kan være fordelaktig å bruke CoT når man jobber med komplekse oppgaver, da nøyaktigheten på outputet fra modellen øker når den \"får lov\" til å jobbe seg gjennom problemet.\n",
    " \n",
    "I tillegg kan du se hvordan modellen tenker, som gjør det lettere for deg å evaluere svaret. Det blir også lettere å se hvor det gikk galt hvis modellen svarer feil.\n",
    " \n",
    "##### *Kilder*\n",
    "[1] [IBM](https://www.ibm.com/think/topics/chain-of-thoughts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Eksempel: CoT vs. non-CoT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CategorizeCot(BaseModel):\n",
    "    \"Categorization of a single feedback entry from an IT survey.\"\n",
    "    chain_of_thought: str = Field(\n",
    "        description=\"Use this space to think through the categorization.\"  # We have defined a variable that explicitly asks the model to think in chain-of-thought\n",
    "    )\n",
    "    category: CATEGORIES = Field(\n",
    "        description=\"Chosen category for the feedback. Choose 'Other' if the other categories provided are not a good fit.\"  \n",
    "    )\n",
    "\n",
    "\n",
    "categorize_chain_cot = categorize_prompt | llm.with_structured_output(\n",
    "    CategorizeCot,\n",
    "    method=\"json_schema\",\n",
    "    strict=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The feedback specifically mentions issues with internet speed, which falls under the category of Network. The humorous tone suggests frustration with the network performance, indicating that the user is experiencing connectivity problems rather than issues with IT support or training.\n",
      "Network\n"
     ]
    }
   ],
   "source": [
    "result = categorize_chain_cot.invoke(\"The internet-speed is so slow I could chase after the IT-survice guy with a baseball-bat!\")\n",
    "print(result.chain_of_thought) \n",
    "print(result.category)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Resonneringsmodell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resonneringsmodeller, som Azure Open AI sin O4-mini-modell, er språkmodeller som er spesielt trent på å tenke før de svarer. Altså *resonnerer* de seg frem til et svar, i motsetning til en typisk LLM som gir raske svar.\n",
    "Resonneringsmodeller bruker chain of thought til å bryte ned oppgaven i mindre deler, for å så jobbe seg gjennom problemstillingen stegvis [3]. Dette er fordelaktig å bruke til oppgaver som krever kompleks problemløsning, logisk tenkning som koding eller matematikk eller til oppgaver med flere steg. Det vil også være fordelaktig å bruke i situasjoner der nøyaktighet og forklarbarhet er viktig [2].\n",
    " \n",
    " \n",
    "##### Kilder\n",
    "[OpenAI](https://platform.openai.com/docs/guides/reasoning?api-mode=chat)\n",
    "\n",
    "[AIavisen](https://aiavisen.no/resonneringsmodeller/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eksempel på ressonering vs. ikke-ressonering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Because the ladder is attached to the boat, it rises with the tide.  No new rungs go underwater, so the number of submerged rungs stays exactly the same as at the start.  In particular, if none were underwater initially, none will be after 3 h.\n",
      "If the tide rises 1 meter every hour, after 3 hours, the tide will have risen:\n",
      "\n",
      "\\[\n",
      "1 \\text{ meter/hour} \\times 3 \\text{ hours} = 3 \\text{ meters}\n",
      "\\]\n",
      "\n",
      "Since the rungs of the ladder are 30 cm apart, we need to convert the rise in tide from meters to centimeters:\n",
      "\n",
      "\\[\n",
      "3 \\text{ meters} = 3 \\times 100 \\text{ cm} = 300 \\text{ cm}\n",
      "\\]\n",
      "\n",
      "Now, to find out how many rungs will be underwater, we divide the total rise in tide by the distance between the rungs:\n",
      "\n",
      "\\[\n",
      "\\frac{300 \\text{ cm}}{30 \\text{ cm/rung}} = 10 \\text{ rungs}\n",
      "\\]\n",
      "\n",
      "Therefore, after 3 hours, **10 rungs** will be underwater.\n"
     ]
    }
   ],
   "source": [
    "# Example\n",
    "\n",
    "prompt = PromptTemplate.from_template(\"\"\"\n",
    "A rope ladder hangs over the side of a boat. The rungs are 30 cm apart. The tide rises 1 meter every hour. After 3 hours, how many rungs will be underwater?\n",
    "\"\"\")\n",
    "\n",
    "reasoning_chain = prompt | reasoning_llm | StrOutputParser()\n",
    "\n",
    "print(reasoning_chain.invoke({}))\n",
    "\n",
    "none_reasoning_chain = prompt | llm | StrOutputParser()\n",
    "\n",
    "print(none_reasoning_chain.invoke({}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aha! Her ser vi et eksempel på at resonneringsmodellen tenker riktig fordi den bryter ned problemet, mens det vanlige llm-kallet uten resonnering går fem på lureoppgaven og svarer feil."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Nå lar vi modellen gå løs på hele datasettet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nå skal vi bruke verktøyene vi har lært til å utføre den første oppgaven, nemlig å kategorisere tilbakemeldingene på tvers av datasettet. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_batch = categorize_chain_structured_output.batch(  # Pass all feedback to the model through batching\n",
    "    [\n",
    "    {\"feedback\": feedback} for feedback in df[\"Feedback\"]\n",
    "    ]\n",
    ")\n",
    "\n",
    "df[\"AI Classification\"] = [result.category for result in result_batch]  # Add column containing the categories to the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Feedback</th>\n",
       "      <th>AI Classification</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>The network infrastructure is robust and usually efficient, though intermittent glitches sometimes hamper productivity and clarity.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Our current infrastructure aligns with business needs, yet occasional rigidity in legacy systems hinders innovative approaches.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>I appreciate the proactive approach to security, though the multi-step authentication process sometimes seems overly complicated.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Our tools maintain high quality with intuitive design and regular updates, even though rare performance lags can hinder productivity.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>The IT-support team is consistently prompt and courteous, though occasional miscommunications lead to fleeting moments of uncertainty.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>Our advanced software tools offer a mix of efficiency and creativity, but the occasional downtime introduces a fleeting sense of dismay.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>I feel more secure with the recent network defense upgrades, yet the complexity of the security software occasionally overwhelms my routine.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>The new cybersecurity measures give me confidence in our data protection, though frequently changing protocols sometimes cause confusion.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>Our enhanced firewall settings significantly lower threats, though frequent reminders to update can feel intrusive.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>The assistance from IT-support is generally effective, though sporadic technical mishaps occasionally undermine steady progress.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>Network performance is generally satisfactory, but unexpected downtime during peak hours creates anxiety about meeting deadlines.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>The training materials are succinct and creatively presented, but rapid changes in content sometimes create a disconnect.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>The integration of technology with our business goals is promising, though outdated processes occasionally clash with modern expectations.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>The curriculum is robust and adapts to emerging trends, though it sometimes lacks tailored details specific to my role.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>The hands-on exercises are engaging and boost confidence, though I occasionally struggle to keep up with the fast pace.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>Although the sessions are structured to be interactive, unexpected glitches disrupt the flow and leave me with some unanswered questions.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>I recognize the efforts to merge IT and business strategies, though sporadic mismatches sometimes lead to project delays.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>The digital tools consistently support day-to-day work, yet occasional interface inconsistencies open up space for minor critiques.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>I appreciate the blend of innovation and stability in our tools, though sporadic integration issues sometimes test my patience.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>The network supports daily operations well, yet intermittent signal drops leave me with a mix of irritation and hope for prompt improvements.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>While the training sessions aim to cover a wide range of topics, their fast pacing occasionally results in uneven clarity and minor confusion.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>While the network operates effectively most times, sporadic slow transmissions during heavy usage create brief instability.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>While IT-support usually offers strong resolution strategies, intermittent communication gaps occasionally disrupt workflow.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>I value our firm stance on cybersecurity, even as a continuous stream of alerts occasionally turns reassurance into slight irritation.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>The security team usually implements robust measures successfully, yet sporadic warnings create moments of heightened alertness.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>The recent training sessions on new software updates provided clear guidance, though sometimes their rapid pace left me wishing for more practical examples.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>The suite of tools is modern and user-friendly, though a few lagging behind expectations stir brief episodes of frustration.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>The stringent security protocols inspire trust, even if the regular system prompts interrupt workflow more than I'd like.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>Robust security protocols foster a safe system environment, but occasional delays in updates sometimes spark minor apprehension.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>Live workshops are full of energy and support, though scheduling conflicts now and then lead to mixed messages.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>31</td>\n",
       "      <td>Timely security updates fortify our systems, even though brief periods of isolation during updates can be disconcerting.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>I appreciate the steady connectivity, although occasional latency disrupts workflow and adds a layer of controlled frustration.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>Our systems generally support business functions effectively, yet sporadic delays in adapting to new trends sometimes create operational bottlenecks.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>The interactive training modules are well-designed but occasionally overwhelm with too many complex details, balancing excitement with mild frustration.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>35</td>\n",
       "      <td>Customer service from IT-support is engaging and resourceful, but the occasional oversight leaves tasks unfinished and briefly worrisome.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>I appreciate how the sessions cover both the basics and advanced features, yet the limited time for Q&amp;A sometimes leaves lingering doubts.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>37</td>\n",
       "      <td>Iâ€™m impressed by the robustness of our software tools, yet occasional glitches during critical operations disrupt my workflow.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>38</td>\n",
       "      <td>The IT solutions seem aligned with our evolving business needs, though occasional mismatches in technology and strategy leave me seeking clearer direction.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>39</td>\n",
       "      <td>I appreciate IT-supportâ€™s readiness to tackle challenges, even though periodic delays in follow-up communications cloud the experience.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>40</td>\n",
       "      <td>The balance between technical capability and business strategy is well-considered, though occasional oversights in market adaptability create subtle frustrations.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>41</td>\n",
       "      <td>The office network's reliability is commendable, yet sporadic slowdowns introduce brief moments of uncertainty during critical tasks.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>42</td>\n",
       "      <td>The comprehensive responses from IT-support are quite reassuring, though backlog delays sometimes counteract the positive support.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>43</td>\n",
       "      <td>I value IT-supportâ€™s clear follow-ups, yet delays in resolving complex issues sometimes leave a lingering sense of apprehension.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>44</td>\n",
       "      <td>The high quality of our system tools fosters productivity, though minor and persistent connectivity challenges occasionally dampen the overall experience.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>45</td>\n",
       "      <td>IT-support often exceeds expectations with timely responses and practical solutions, though recurring issues still stir mild frustration.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>46</td>\n",
       "      <td>I find the training generally beneficial, yet at times the depth of content exceeds my comfort level and leaves parts underexplored.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>47</td>\n",
       "      <td>The quality of our primary tools is commendable, though sporadic bugs and outdated interfaces sometimes temper initial enthusiasm.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>48</td>\n",
       "      <td>Connectivity is strong and dependable\\t however, periodic drops sow seeds of concern regarding data consistency.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>49</td>\n",
       "      <td>Enhanced monitoring tools fortify our security practices, though occasional false alarms trigger unnecessary concerns.</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>50</td>\n",
       "      <td>I appreciate the efforts to tailor IT solutions to our business model, even if delays in responsiveness sometimes generate uncertainty.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ID  \\\n",
       "0    1   \n",
       "1    2   \n",
       "2    3   \n",
       "3    4   \n",
       "4    5   \n",
       "5    6   \n",
       "6    7   \n",
       "7    8   \n",
       "8    9   \n",
       "9   10   \n",
       "10  11   \n",
       "11  12   \n",
       "12  13   \n",
       "13  14   \n",
       "14  15   \n",
       "15  16   \n",
       "16  17   \n",
       "17  18   \n",
       "18  19   \n",
       "19  20   \n",
       "20  21   \n",
       "21  22   \n",
       "22  23   \n",
       "23  24   \n",
       "24  25   \n",
       "25  26   \n",
       "26  27   \n",
       "27  28   \n",
       "28  29   \n",
       "29  30   \n",
       "30  31   \n",
       "31  32   \n",
       "32  33   \n",
       "33  34   \n",
       "34  35   \n",
       "35  36   \n",
       "36  37   \n",
       "37  38   \n",
       "38  39   \n",
       "39  40   \n",
       "40  41   \n",
       "41  42   \n",
       "42  43   \n",
       "43  44   \n",
       "44  45   \n",
       "45  46   \n",
       "46  47   \n",
       "47  48   \n",
       "48  49   \n",
       "49  50   \n",
       "\n",
       "                                                                                                                                                              Feedback  \\\n",
       "0                                  The network infrastructure is robust and usually efficient, though intermittent glitches sometimes hamper productivity and clarity.   \n",
       "1                                      Our current infrastructure aligns with business needs, yet occasional rigidity in legacy systems hinders innovative approaches.   \n",
       "2                                    I appreciate the proactive approach to security, though the multi-step authentication process sometimes seems overly complicated.   \n",
       "3                                Our tools maintain high quality with intuitive design and regular updates, even though rare performance lags can hinder productivity.   \n",
       "4                               The IT-support team is consistently prompt and courteous, though occasional miscommunications lead to fleeting moments of uncertainty.   \n",
       "5                             Our advanced software tools offer a mix of efficiency and creativity, but the occasional downtime introduces a fleeting sense of dismay.   \n",
       "6                         I feel more secure with the recent network defense upgrades, yet the complexity of the security software occasionally overwhelms my routine.   \n",
       "7                            The new cybersecurity measures give me confidence in our data protection, though frequently changing protocols sometimes cause confusion.   \n",
       "8                                                  Our enhanced firewall settings significantly lower threats, though frequent reminders to update can feel intrusive.   \n",
       "9                                     The assistance from IT-support is generally effective, though sporadic technical mishaps occasionally undermine steady progress.   \n",
       "10                                   Network performance is generally satisfactory, but unexpected downtime during peak hours creates anxiety about meeting deadlines.   \n",
       "11                                           The training materials are succinct and creatively presented, but rapid changes in content sometimes create a disconnect.   \n",
       "12                          The integration of technology with our business goals is promising, though outdated processes occasionally clash with modern expectations.   \n",
       "13                                             The curriculum is robust and adapts to emerging trends, though it sometimes lacks tailored details specific to my role.   \n",
       "14                                             The hands-on exercises are engaging and boost confidence, though I occasionally struggle to keep up with the fast pace.   \n",
       "15                           Although the sessions are structured to be interactive, unexpected glitches disrupt the flow and leave me with some unanswered questions.   \n",
       "16                                           I recognize the efforts to merge IT and business strategies, though sporadic mismatches sometimes lead to project delays.   \n",
       "17                                 The digital tools consistently support day-to-day work, yet occasional interface inconsistencies open up space for minor critiques.   \n",
       "18                                     I appreciate the blend of innovation and stability in our tools, though sporadic integration issues sometimes test my patience.   \n",
       "19                       The network supports daily operations well, yet intermittent signal drops leave me with a mix of irritation and hope for prompt improvements.   \n",
       "20                      While the training sessions aim to cover a wide range of topics, their fast pacing occasionally results in uneven clarity and minor confusion.   \n",
       "21                                         While the network operates effectively most times, sporadic slow transmissions during heavy usage create brief instability.   \n",
       "22                                        While IT-support usually offers strong resolution strategies, intermittent communication gaps occasionally disrupt workflow.   \n",
       "23                              I value our firm stance on cybersecurity, even as a continuous stream of alerts occasionally turns reassurance into slight irritation.   \n",
       "24                                    The security team usually implements robust measures successfully, yet sporadic warnings create moments of heightened alertness.   \n",
       "25        The recent training sessions on new software updates provided clear guidance, though sometimes their rapid pace left me wishing for more practical examples.   \n",
       "26                                        The suite of tools is modern and user-friendly, though a few lagging behind expectations stir brief episodes of frustration.   \n",
       "27                                           The stringent security protocols inspire trust, even if the regular system prompts interrupt workflow more than I'd like.   \n",
       "28                                    Robust security protocols foster a safe system environment, but occasional delays in updates sometimes spark minor apprehension.   \n",
       "29                                                     Live workshops are full of energy and support, though scheduling conflicts now and then lead to mixed messages.   \n",
       "30                                            Timely security updates fortify our systems, even though brief periods of isolation during updates can be disconcerting.   \n",
       "31                                     I appreciate the steady connectivity, although occasional latency disrupts workflow and adds a layer of controlled frustration.   \n",
       "32               Our systems generally support business functions effectively, yet sporadic delays in adapting to new trends sometimes create operational bottlenecks.   \n",
       "33            The interactive training modules are well-designed but occasionally overwhelm with too many complex details, balancing excitement with mild frustration.   \n",
       "34                           Customer service from IT-support is engaging and resourceful, but the occasional oversight leaves tasks unfinished and briefly worrisome.   \n",
       "35                          I appreciate how the sessions cover both the basics and advanced features, yet the limited time for Q&A sometimes leaves lingering doubts.   \n",
       "36                                    Iâ€™m impressed by the robustness of our software tools, yet occasional glitches during critical operations disrupt my workflow.   \n",
       "37         The IT solutions seem aligned with our evolving business needs, though occasional mismatches in technology and strategy leave me seeking clearer direction.   \n",
       "38                           I appreciate IT-supportâ€™s readiness to tackle challenges, even though periodic delays in follow-up communications cloud the experience.   \n",
       "39  The balance between technical capability and business strategy is well-considered, though occasional oversights in market adaptability create subtle frustrations.   \n",
       "40                               The office network's reliability is commendable, yet sporadic slowdowns introduce brief moments of uncertainty during critical tasks.   \n",
       "41                                  The comprehensive responses from IT-support are quite reassuring, though backlog delays sometimes counteract the positive support.   \n",
       "42                                  I value IT-supportâ€™s clear follow-ups, yet delays in resolving complex issues sometimes leave a lingering sense of apprehension.   \n",
       "43          The high quality of our system tools fosters productivity, though minor and persistent connectivity challenges occasionally dampen the overall experience.   \n",
       "44                           IT-support often exceeds expectations with timely responses and practical solutions, though recurring issues still stir mild frustration.   \n",
       "45                                I find the training generally beneficial, yet at times the depth of content exceeds my comfort level and leaves parts underexplored.   \n",
       "46                                  The quality of our primary tools is commendable, though sporadic bugs and outdated interfaces sometimes temper initial enthusiasm.   \n",
       "47                                                    Connectivity is strong and dependable\\t however, periodic drops sow seeds of concern regarding data consistency.   \n",
       "48                                              Enhanced monitoring tools fortify our security practices, though occasional false alarms trigger unnecessary concerns.   \n",
       "49                             I appreciate the efforts to tailor IT solutions to our business model, even if delays in responsiveness sometimes generate uncertainty.   \n",
       "\n",
       "   AI Classification  \n",
       "0            Network  \n",
       "1              Other  \n",
       "2              Other  \n",
       "3              Other  \n",
       "4         IT-support  \n",
       "5            Network  \n",
       "6            Network  \n",
       "7              Other  \n",
       "8              Other  \n",
       "9         IT-support  \n",
       "10           Network  \n",
       "11          Training  \n",
       "12             Other  \n",
       "13          Training  \n",
       "14          Training  \n",
       "15        IT-support  \n",
       "16             Other  \n",
       "17             Other  \n",
       "18             Other  \n",
       "19           Network  \n",
       "20          Training  \n",
       "21           Network  \n",
       "22        IT-support  \n",
       "23             Other  \n",
       "24             Other  \n",
       "25          Training  \n",
       "26             Other  \n",
       "27             Other  \n",
       "28             Other  \n",
       "29          Training  \n",
       "30             Other  \n",
       "31           Network  \n",
       "32             Other  \n",
       "33          Training  \n",
       "34        IT-support  \n",
       "35          Training  \n",
       "36             Other  \n",
       "37             Other  \n",
       "38        IT-support  \n",
       "39             Other  \n",
       "40           Network  \n",
       "41        IT-support  \n",
       "42        IT-support  \n",
       "43           Network  \n",
       "44        IT-support  \n",
       "45          Training  \n",
       "46             Other  \n",
       "47           Network  \n",
       "48             Other  \n",
       "49        IT-support  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nå vil vi se nærmere på radene som falt under 'Other' og ikke ble kategorisert i første omgang. Til dette vil vi bruke en resonneringsmodell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Categorize_others(BaseModel):\n",
    "    \"Categorization of a single feedback entry from an IT survey.\"\n",
    "    category_take1 : str = Field(\n",
    "        description=\"The best fitting category. Only one.\"\n",
    "    )\n",
    "    category_take2 : str = Field(\n",
    "        description = \"Do another take of the categorization. If the category you chose is very similar to any of the predetermined categories 'Network','Training', 'IT-support' then rather use one of those.\"\n",
    "    )\n",
    " \n",
    " \n",
    "categorize_prompt_other = PromptTemplate.from_template(\n",
    "\"\"\"\n",
    "Categorize the following feedback from an IT-survey into the category that best describes the feedback.\n",
    " \n",
    "Feedback:\n",
    "<feedback>\n",
    "{feedback}\n",
    "</feedback>\n",
    "\"\"\"\n",
    ")\n",
    " \n",
    "categorize_chain_structured_output_others = categorize_prompt_other | reasoning_llm.with_structured_output(\n",
    "    Categorize_others,\n",
    "    method=\"json_schema\",\n",
    "    strict=True,\n",
    ")\n",
    " \n",
    "# Retrieve indexes for all rows where AI classification = \"Other\"\n",
    "other_indices = df[df['AI Classification'] == \"Other\"].index\n",
    " \n",
    " \n",
    "result_batch = categorize_chain_structured_output_others.batch(\n",
    "    [\n",
    "    {\"feedback\": feedback} for feedback in df.loc[other_indices, \"Feedback\"]\n",
    "    ]\n",
    ")\n",
    " \n",
    "df.loc[other_indices, \"AI Classification\"] = [result.category_take2 for result in result_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Feedback</th>\n",
       "      <th>AI Classification</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>The network infrastructure is robust and usually efficient, though intermittent glitches sometimes hamper productivity and clarity.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Our current infrastructure aligns with business needs, yet occasional rigidity in legacy systems hinders innovative approaches.</td>\n",
       "      <td>Legacy Systems</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>I appreciate the proactive approach to security, though the multi-step authentication process sometimes seems overly complicated.</td>\n",
       "      <td>Security</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Our tools maintain high quality with intuitive design and regular updates, even though rare performance lags can hinder productivity.</td>\n",
       "      <td>Software Updates &amp; Maintenance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>The IT-support team is consistently prompt and courteous, though occasional miscommunications lead to fleeting moments of uncertainty.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>Our advanced software tools offer a mix of efficiency and creativity, but the occasional downtime introduces a fleeting sense of dismay.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>7</td>\n",
       "      <td>I feel more secure with the recent network defense upgrades, yet the complexity of the security software occasionally overwhelms my routine.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>The new cybersecurity measures give me confidence in our data protection, though frequently changing protocols sometimes cause confusion.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>9</td>\n",
       "      <td>Our enhanced firewall settings significantly lower threats, though frequent reminders to update can feel intrusive.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>The assistance from IT-support is generally effective, though sporadic technical mishaps occasionally undermine steady progress.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11</td>\n",
       "      <td>Network performance is generally satisfactory, but unexpected downtime during peak hours creates anxiety about meeting deadlines.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>12</td>\n",
       "      <td>The training materials are succinct and creatively presented, but rapid changes in content sometimes create a disconnect.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>13</td>\n",
       "      <td>The integration of technology with our business goals is promising, though outdated processes occasionally clash with modern expectations.</td>\n",
       "      <td>Change Management</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>14</td>\n",
       "      <td>The curriculum is robust and adapts to emerging trends, though it sometimes lacks tailored details specific to my role.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>15</td>\n",
       "      <td>The hands-on exercises are engaging and boost confidence, though I occasionally struggle to keep up with the fast pace.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>16</td>\n",
       "      <td>Although the sessions are structured to be interactive, unexpected glitches disrupt the flow and leave me with some unanswered questions.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>17</td>\n",
       "      <td>I recognize the efforts to merge IT and business strategies, though sporadic mismatches sometimes lead to project delays.</td>\n",
       "      <td>Project Management</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>18</td>\n",
       "      <td>The digital tools consistently support day-to-day work, yet occasional interface inconsistencies open up space for minor critiques.</td>\n",
       "      <td>Usability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>19</td>\n",
       "      <td>I appreciate the blend of innovation and stability in our tools, though sporadic integration issues sometimes test my patience.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>20</td>\n",
       "      <td>The network supports daily operations well, yet intermittent signal drops leave me with a mix of irritation and hope for prompt improvements.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>21</td>\n",
       "      <td>While the training sessions aim to cover a wide range of topics, their fast pacing occasionally results in uneven clarity and minor confusion.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>22</td>\n",
       "      <td>While the network operates effectively most times, sporadic slow transmissions during heavy usage create brief instability.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>23</td>\n",
       "      <td>While IT-support usually offers strong resolution strategies, intermittent communication gaps occasionally disrupt workflow.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>24</td>\n",
       "      <td>I value our firm stance on cybersecurity, even as a continuous stream of alerts occasionally turns reassurance into slight irritation.</td>\n",
       "      <td>Security Notifications</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>25</td>\n",
       "      <td>The security team usually implements robust measures successfully, yet sporadic warnings create moments of heightened alertness.</td>\n",
       "      <td>Security</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>26</td>\n",
       "      <td>The recent training sessions on new software updates provided clear guidance, though sometimes their rapid pace left me wishing for more practical examples.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>27</td>\n",
       "      <td>The suite of tools is modern and user-friendly, though a few lagging behind expectations stir brief episodes of frustration.</td>\n",
       "      <td>Application performance and usability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28</td>\n",
       "      <td>The stringent security protocols inspire trust, even if the regular system prompts interrupt workflow more than I'd like.</td>\n",
       "      <td>Usability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>29</td>\n",
       "      <td>Robust security protocols foster a safe system environment, but occasional delays in updates sometimes spark minor apprehension.</td>\n",
       "      <td>Updates</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>30</td>\n",
       "      <td>Live workshops are full of energy and support, though scheduling conflicts now and then lead to mixed messages.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>31</td>\n",
       "      <td>Timely security updates fortify our systems, even though brief periods of isolation during updates can be disconcerting.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>I appreciate the steady connectivity, although occasional latency disrupts workflow and adds a layer of controlled frustration.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>Our systems generally support business functions effectively, yet sporadic delays in adapting to new trends sometimes create operational bottlenecks.</td>\n",
       "      <td>System Updates</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>The interactive training modules are well-designed but occasionally overwhelm with too many complex details, balancing excitement with mild frustration.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>35</td>\n",
       "      <td>Customer service from IT-support is engaging and resourceful, but the occasional oversight leaves tasks unfinished and briefly worrisome.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>36</td>\n",
       "      <td>I appreciate how the sessions cover both the basics and advanced features, yet the limited time for Q&amp;A sometimes leaves lingering doubts.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>37</td>\n",
       "      <td>Iâ€™m impressed by the robustness of our software tools, yet occasional glitches during critical operations disrupt my workflow.</td>\n",
       "      <td>System Stability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>38</td>\n",
       "      <td>The IT solutions seem aligned with our evolving business needs, though occasional mismatches in technology and strategy leave me seeking clearer direction.</td>\n",
       "      <td>IT Strategy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>39</td>\n",
       "      <td>I appreciate IT-supportâ€™s readiness to tackle challenges, even though periodic delays in follow-up communications cloud the experience.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>40</td>\n",
       "      <td>The balance between technical capability and business strategy is well-considered, though occasional oversights in market adaptability create subtle frustrations.</td>\n",
       "      <td>Strategic IT Alignment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>41</td>\n",
       "      <td>The office network's reliability is commendable, yet sporadic slowdowns introduce brief moments of uncertainty during critical tasks.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>42</td>\n",
       "      <td>The comprehensive responses from IT-support are quite reassuring, though backlog delays sometimes counteract the positive support.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>43</td>\n",
       "      <td>I value IT-supportâ€™s clear follow-ups, yet delays in resolving complex issues sometimes leave a lingering sense of apprehension.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>44</td>\n",
       "      <td>The high quality of our system tools fosters productivity, though minor and persistent connectivity challenges occasionally dampen the overall experience.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>45</td>\n",
       "      <td>IT-support often exceeds expectations with timely responses and practical solutions, though recurring issues still stir mild frustration.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>46</td>\n",
       "      <td>I find the training generally beneficial, yet at times the depth of content exceeds my comfort level and leaves parts underexplored.</td>\n",
       "      <td>Training</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>47</td>\n",
       "      <td>The quality of our primary tools is commendable, though sporadic bugs and outdated interfaces sometimes temper initial enthusiasm.</td>\n",
       "      <td>Software usability</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>48</td>\n",
       "      <td>Connectivity is strong and dependable\\t however, periodic drops sow seeds of concern regarding data consistency.</td>\n",
       "      <td>Network</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>49</td>\n",
       "      <td>Enhanced monitoring tools fortify our security practices, though occasional false alarms trigger unnecessary concerns.</td>\n",
       "      <td>Security</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>50</td>\n",
       "      <td>I appreciate the efforts to tailor IT solutions to our business model, even if delays in responsiveness sometimes generate uncertainty.</td>\n",
       "      <td>IT-support</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ID  \\\n",
       "0    1   \n",
       "1    2   \n",
       "2    3   \n",
       "3    4   \n",
       "4    5   \n",
       "5    6   \n",
       "6    7   \n",
       "7    8   \n",
       "8    9   \n",
       "9   10   \n",
       "10  11   \n",
       "11  12   \n",
       "12  13   \n",
       "13  14   \n",
       "14  15   \n",
       "15  16   \n",
       "16  17   \n",
       "17  18   \n",
       "18  19   \n",
       "19  20   \n",
       "20  21   \n",
       "21  22   \n",
       "22  23   \n",
       "23  24   \n",
       "24  25   \n",
       "25  26   \n",
       "26  27   \n",
       "27  28   \n",
       "28  29   \n",
       "29  30   \n",
       "30  31   \n",
       "31  32   \n",
       "32  33   \n",
       "33  34   \n",
       "34  35   \n",
       "35  36   \n",
       "36  37   \n",
       "37  38   \n",
       "38  39   \n",
       "39  40   \n",
       "40  41   \n",
       "41  42   \n",
       "42  43   \n",
       "43  44   \n",
       "44  45   \n",
       "45  46   \n",
       "46  47   \n",
       "47  48   \n",
       "48  49   \n",
       "49  50   \n",
       "\n",
       "                                                                                                                                                              Feedback  \\\n",
       "0                                  The network infrastructure is robust and usually efficient, though intermittent glitches sometimes hamper productivity and clarity.   \n",
       "1                                      Our current infrastructure aligns with business needs, yet occasional rigidity in legacy systems hinders innovative approaches.   \n",
       "2                                    I appreciate the proactive approach to security, though the multi-step authentication process sometimes seems overly complicated.   \n",
       "3                                Our tools maintain high quality with intuitive design and regular updates, even though rare performance lags can hinder productivity.   \n",
       "4                               The IT-support team is consistently prompt and courteous, though occasional miscommunications lead to fleeting moments of uncertainty.   \n",
       "5                             Our advanced software tools offer a mix of efficiency and creativity, but the occasional downtime introduces a fleeting sense of dismay.   \n",
       "6                         I feel more secure with the recent network defense upgrades, yet the complexity of the security software occasionally overwhelms my routine.   \n",
       "7                            The new cybersecurity measures give me confidence in our data protection, though frequently changing protocols sometimes cause confusion.   \n",
       "8                                                  Our enhanced firewall settings significantly lower threats, though frequent reminders to update can feel intrusive.   \n",
       "9                                     The assistance from IT-support is generally effective, though sporadic technical mishaps occasionally undermine steady progress.   \n",
       "10                                   Network performance is generally satisfactory, but unexpected downtime during peak hours creates anxiety about meeting deadlines.   \n",
       "11                                           The training materials are succinct and creatively presented, but rapid changes in content sometimes create a disconnect.   \n",
       "12                          The integration of technology with our business goals is promising, though outdated processes occasionally clash with modern expectations.   \n",
       "13                                             The curriculum is robust and adapts to emerging trends, though it sometimes lacks tailored details specific to my role.   \n",
       "14                                             The hands-on exercises are engaging and boost confidence, though I occasionally struggle to keep up with the fast pace.   \n",
       "15                           Although the sessions are structured to be interactive, unexpected glitches disrupt the flow and leave me with some unanswered questions.   \n",
       "16                                           I recognize the efforts to merge IT and business strategies, though sporadic mismatches sometimes lead to project delays.   \n",
       "17                                 The digital tools consistently support day-to-day work, yet occasional interface inconsistencies open up space for minor critiques.   \n",
       "18                                     I appreciate the blend of innovation and stability in our tools, though sporadic integration issues sometimes test my patience.   \n",
       "19                       The network supports daily operations well, yet intermittent signal drops leave me with a mix of irritation and hope for prompt improvements.   \n",
       "20                      While the training sessions aim to cover a wide range of topics, their fast pacing occasionally results in uneven clarity and minor confusion.   \n",
       "21                                         While the network operates effectively most times, sporadic slow transmissions during heavy usage create brief instability.   \n",
       "22                                        While IT-support usually offers strong resolution strategies, intermittent communication gaps occasionally disrupt workflow.   \n",
       "23                              I value our firm stance on cybersecurity, even as a continuous stream of alerts occasionally turns reassurance into slight irritation.   \n",
       "24                                    The security team usually implements robust measures successfully, yet sporadic warnings create moments of heightened alertness.   \n",
       "25        The recent training sessions on new software updates provided clear guidance, though sometimes their rapid pace left me wishing for more practical examples.   \n",
       "26                                        The suite of tools is modern and user-friendly, though a few lagging behind expectations stir brief episodes of frustration.   \n",
       "27                                           The stringent security protocols inspire trust, even if the regular system prompts interrupt workflow more than I'd like.   \n",
       "28                                    Robust security protocols foster a safe system environment, but occasional delays in updates sometimes spark minor apprehension.   \n",
       "29                                                     Live workshops are full of energy and support, though scheduling conflicts now and then lead to mixed messages.   \n",
       "30                                            Timely security updates fortify our systems, even though brief periods of isolation during updates can be disconcerting.   \n",
       "31                                     I appreciate the steady connectivity, although occasional latency disrupts workflow and adds a layer of controlled frustration.   \n",
       "32               Our systems generally support business functions effectively, yet sporadic delays in adapting to new trends sometimes create operational bottlenecks.   \n",
       "33            The interactive training modules are well-designed but occasionally overwhelm with too many complex details, balancing excitement with mild frustration.   \n",
       "34                           Customer service from IT-support is engaging and resourceful, but the occasional oversight leaves tasks unfinished and briefly worrisome.   \n",
       "35                          I appreciate how the sessions cover both the basics and advanced features, yet the limited time for Q&A sometimes leaves lingering doubts.   \n",
       "36                                    Iâ€™m impressed by the robustness of our software tools, yet occasional glitches during critical operations disrupt my workflow.   \n",
       "37         The IT solutions seem aligned with our evolving business needs, though occasional mismatches in technology and strategy leave me seeking clearer direction.   \n",
       "38                           I appreciate IT-supportâ€™s readiness to tackle challenges, even though periodic delays in follow-up communications cloud the experience.   \n",
       "39  The balance between technical capability and business strategy is well-considered, though occasional oversights in market adaptability create subtle frustrations.   \n",
       "40                               The office network's reliability is commendable, yet sporadic slowdowns introduce brief moments of uncertainty during critical tasks.   \n",
       "41                                  The comprehensive responses from IT-support are quite reassuring, though backlog delays sometimes counteract the positive support.   \n",
       "42                                  I value IT-supportâ€™s clear follow-ups, yet delays in resolving complex issues sometimes leave a lingering sense of apprehension.   \n",
       "43          The high quality of our system tools fosters productivity, though minor and persistent connectivity challenges occasionally dampen the overall experience.   \n",
       "44                           IT-support often exceeds expectations with timely responses and practical solutions, though recurring issues still stir mild frustration.   \n",
       "45                                I find the training generally beneficial, yet at times the depth of content exceeds my comfort level and leaves parts underexplored.   \n",
       "46                                  The quality of our primary tools is commendable, though sporadic bugs and outdated interfaces sometimes temper initial enthusiasm.   \n",
       "47                                                    Connectivity is strong and dependable\\t however, periodic drops sow seeds of concern regarding data consistency.   \n",
       "48                                              Enhanced monitoring tools fortify our security practices, though occasional false alarms trigger unnecessary concerns.   \n",
       "49                             I appreciate the efforts to tailor IT solutions to our business model, even if delays in responsiveness sometimes generate uncertainty.   \n",
       "\n",
       "                        AI Classification  \n",
       "0                                 Network  \n",
       "1                          Legacy Systems  \n",
       "2                                Security  \n",
       "3          Software Updates & Maintenance  \n",
       "4                              IT-support  \n",
       "5                                 Network  \n",
       "6                                 Network  \n",
       "7                                Training  \n",
       "8                                 Network  \n",
       "9                              IT-support  \n",
       "10                                Network  \n",
       "11                               Training  \n",
       "12                      Change Management  \n",
       "13                               Training  \n",
       "14                               Training  \n",
       "15                             IT-support  \n",
       "16                     Project Management  \n",
       "17                              Usability  \n",
       "18                             IT-support  \n",
       "19                                Network  \n",
       "20                               Training  \n",
       "21                                Network  \n",
       "22                             IT-support  \n",
       "23                 Security Notifications  \n",
       "24                               Security  \n",
       "25                               Training  \n",
       "26  Application performance and usability  \n",
       "27                              Usability  \n",
       "28                                Updates  \n",
       "29                               Training  \n",
       "30                             IT-support  \n",
       "31                                Network  \n",
       "32                         System Updates  \n",
       "33                               Training  \n",
       "34                             IT-support  \n",
       "35                               Training  \n",
       "36                       System Stability  \n",
       "37                            IT Strategy  \n",
       "38                             IT-support  \n",
       "39                 Strategic IT Alignment  \n",
       "40                                Network  \n",
       "41                             IT-support  \n",
       "42                             IT-support  \n",
       "43                                Network  \n",
       "44                             IT-support  \n",
       "45                               Training  \n",
       "46                     Software usability  \n",
       "47                                Network  \n",
       "48                               Security  \n",
       "49                             IT-support  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df\n",
    "\n",
    "# Spør Arne\n",
    "# Per nå bruker den kategoriene vi fikk fra LLMen. Når den kategorieserer Other får vi mange nye og litt varierende kategorier. Det er også mange av de. Skulle vi bare brukt den som eksempel også bruke den ekte kategoriseringe her?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oi! Her fikk vi mange andre kategorier. Du kan sammenligne AI sin kategorisering med fasiten i kolonnen \"Category\" - reflekter litt rundt hvorvidt AI er for spesifik. Her kan f.eks. prompten skrives om for å få mer / mindre spesifikke kategorier. Kunne vi definert en til kategori som \"eksempel\" for modellen for å unngå så mange kategorier med små variasjoner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nå har vi definert passende kategorier for alle tilbakemeldingene. \n",
    "Videre vil vi få modellen til å oppsummere per kategori, slik at vi sitter igjen med en overordnet oversikt. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Din tur!\n",
    "\n",
    "Nå som kategoriseringen er gjennomført, og vi har forsikret oss om at alle radene har fått en passende kategori, kan vi gå videre til neste oppgave. Her er målet å samle informasjonen fra kategoriseringen til en overordnet oversikt. Lag en LCEL-kjede som tar resultatet fra forrige oppgave og oppsummerer per kategori ved hjelp av en LLM. Inkluder structured output. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tip 1: Create a good prompt! Use this to reflect on what the end goal is.\n",
    "prompt = PromptTemplate.from_template(\"\"\"\n",
    "Well, how should I phrase this? Shouldn't there be some data included here as well?\n",
    "\"\"\")\n",
    "\n",
    "# Tip 2: Can you create a class that inherits from BaseModel to make this easier?\n",
    "class SummarizeFeedback(BaseModel):\n",
    "    \"Description...\"\n",
    "    summary: str = ...\n",
    "\n",
    "# Tip 3: Time to create the chain\n",
    "summary_chain_structured_output = ...\n",
    "\n",
    "# Tip 4: Call the model with the dataset from the survey\n",
    "summary = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Fasit\n",
    "# 1. Prompt Template\n",
    "summary_prompt = PromptTemplate.from_template(\"\"\"\n",
    "You are a domain expert in internal IT operations and organizational analysis. You will be provided with a dataset containing qualitative feedback from employees in an IT company. \n",
    "Each row in the dataset represents a feedback entry and is associated with a specific category.\n",
    "\n",
    "For each category, carefully:\n",
    "1. Read and interpret the feedback entries assigned to that category.\n",
    "2. Identify core themes, recurring patterns, and contrasting opinions within that category.\n",
    "3. Evaluate the feedback logically: What are the likely underlying causes of recurring issues or praises? Are there signs of systemic problems, isolated incidents, or misaligned expectations?\n",
    "4. Summarize each category in 3 to 6 bullet points, highlighting key sentiments (positive and negative), representative concerns or compliments, and any significant outliers\n",
    "\n",
    "Present your findings in a clean, professional way with one section per category. \n",
    "\n",
    "This is the employee feedback data: {survey_results}\n",
    "\"\"\")\n",
    "\n",
    "# 2. Class for structured output\n",
    "class SummarizeFeedback(BaseModel):\n",
    "    \"Summary of the different categorizes recognized in the feedback from an IT-survey.\"\n",
    "    summary : str = Field(\n",
    "        description=\"For each category: Category name and 3-6 bullet points summarizing the category.\"\n",
    "    )\n",
    "\n",
    "# 3. Summary-chain\n",
    "summary_chain_structured_output = summary_prompt | llm.with_structured_output(\n",
    "    SummarizeFeedback,\n",
    "    method=\"json_schema\",\n",
    "    strict=True,\n",
    ")\n",
    "\n",
    "# 4. Kall modellen med det kategoriserte datasettet fra undersøkelsen \n",
    "summary = summary_chain_structured_output.invoke({\"survey_results\": df})\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "### Network\n",
       "- Generally robust and efficient, but intermittent glitches can hamper productivity.\n",
       "- Occasional slowdowns during peak usage create anxiety about meeting deadlines.\n",
       "- Connectivity is strong, yet periodic drops raise concerns about data consistency.\n",
       "\n",
       "### Legacy Systems\n",
       "- Current infrastructure aligns with business needs, but rigidity in legacy systems hinders innovation.\n",
       "- Outdated processes clash with modern expectations, causing frustration.\n",
       "\n",
       "### Security\n",
       "- Proactive security measures instill confidence, but complexity can overwhelm users.\n",
       "- Frequent alerts and reminders can feel intrusive, leading to slight irritation.\n",
       "- Robust protocols foster a safe environment, though sporadic warnings heighten alertness.\n",
       "\n",
       "### Software Updates & Maintenance\n",
       "- High-quality tools with intuitive design, but occasional performance lags disrupt productivity.\n",
       "- Rapid changes in software updates can create confusion and disconnect.\n",
       "\n",
       "### IT-support\n",
       "- Generally prompt and courteous, but miscommunications occasionally disrupt workflow.\n",
       "- Effective assistance, though backlog delays can counteract positive experiences.\n",
       "- Clear follow-ups are appreciated, but complex issues may take longer to resolve.\n",
       "\n",
       "### Training\n",
       "- Training sessions are engaging and well-structured, but fast pacing can lead to confusion.\n",
       "- Hands-on exercises boost confidence, yet some struggle to keep up with the pace.\n",
       "- Content depth sometimes exceeds comfort levels, leaving parts underexplored.\n",
       "\n",
       "### Change Management\n",
       "- Integration of technology with business goals is promising, but outdated processes can create bottlenecks.\n",
       "- Sporadic mismatches in technology and strategy lead to project delays.\n",
       "\n",
       "### Usability\n",
       "- Digital tools support daily work, but interface inconsistencies can lead to minor critiques.\n",
       "- User-friendly tools are appreciated, but some lag behind expectations, causing frustration.\n",
       "\n",
       "### System Updates\n",
       "- Timely updates fortify systems, but brief isolation periods during updates can be disconcerting.\n",
       "- Delays in adapting to new trends create operational bottlenecks.\n",
       "\n",
       "### Strategic IT Alignment\n",
       "- IT solutions align with evolving business needs, but occasional oversights in market adaptability create frustrations."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# A nice way to show the output from the model\n",
    "from IPython.display import Markdown\n",
    "\n",
    "display(Markdown(summary.summary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rapport til ledelsen\n",
    "\n",
    "I siste oppgave skal vi bruke oppsummeringen vi nettopp lagde til å generere en rapport som kan sendes til ledelsen. Rapporten skal gi mottakerne en oversikt over de viktigste punktene i medarbeideundersøkelsen og komme med noen forslag til endringer som kan skape forbedringer til neste år. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Oppgave \n",
    "report_prompt = PromptTemplate.from_template(\"\"\"\n",
    "Bla bla bla\n",
    "\"\"\")\n",
    "\n",
    "# 2. Class for structured output\n",
    "class ReportForLeadership(BaseModel):\n",
    "    \"Raport for the leadership of an IT-company on results of an internal IT-survey.\"\n",
    "    snappy_title : str = Field(\n",
    "        description=\"A fitting title for the report. Must begin with '# ' to ensure easy markdown formatting.\"\n",
    "    )\n",
    "    key_takeaway:str = ...\n",
    "\n",
    "# 3. Report-chain\n",
    "report_chain_structured_output = ...\n",
    "\n",
    "# 4. Kall modellen med oppsummeringen av kategoriene\n",
    "report = ...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fasit\n",
    "report_prompt = PromptTemplate.from_template(\"\"\"\n",
    "You are an expert HR and technical operations analyst. I will provide you with a dataset of employee feedback collected from an IT company.\n",
    "\n",
    "Your task is to deeply analyze this feedback and generate a concise executive-level summary report in markdown format that includes:\n",
    "\n",
    "1. Key Takeaways\n",
    "Provide a short summary of the overall feedback in 3-5 bullet points. Focus only on the main issues or areas of satisfaction.\n",
    "Include both positive and negative themes, but prioritize the most important and impactful points.\n",
    "Limit each point to 1-2 sentences.\n",
    "Before finalizing each point, take a moment to reflect on why each issue might be present (e.g., systemic problems, temporary issues, resource constraints, etc.)\n",
    "\n",
    "2. Suggested Improvements\n",
    "Based on the overall feedback, propose 2-3 high-level, actionable measures that the company could take to address the most pressing issues and enhance overall performance or satisfaction.\n",
    "Each suggestion should be brief, directly tied to the feedback, and strategic in nature.\n",
    "Think about short-term vs long-term solutions and consider the feasibility of each suggestion.\n",
    "\n",
    "3. Output\n",
    "Present your findings in a structured way with clear section headings, bullet points for easy scanning, and a consise, direct and professional tone suitable for leadership review.\n",
    "\n",
    "This is the employee feedback data: {summary_text}\n",
    "\"\"\")\n",
    "\n",
    "# 2. Class for structured output\n",
    "class ReportForLeadership(BaseModel):\n",
    "    \"Raport for the leadership of an IT-company on results of an internal IT-survey.\"\n",
    "    snappy_title : str = Field(\n",
    "        description=\"A fitting title for the report. Must begin with '# ' to ensure easy markdown formatting.\"\n",
    "    )\n",
    "    intro : str = Field(\n",
    "        description=\"1 sentence describing thepurpose of the report.\" \n",
    "    )\n",
    "    key_takeaways : str = Field(\n",
    "        description=\"3-5 bulletpoints describing the key-takeaways. Limit each point to 1-2 sentences.\" \n",
    "    )\n",
    "    suggested_improvements : str = Field(\n",
    "        description=\"2-3 actionable measures for the company. Keep it brief.\"\n",
    "    )\n",
    "    outro: str = Field(\n",
    "        description=\"1 sentence ending for the report. Be creative.\" \n",
    "    )\n",
    "\n",
    "# 3. Report-chain\n",
    "report_chain_structured_output = report_prompt | llm.with_structured_output(\n",
    "    ReportForLeadership,\n",
    "    method=\"json_schema\",\n",
    "    strict=True,\n",
    ")\n",
    "\n",
    "# 4. Kall modellen med oppsummeringen av kategoriene\n",
    "report = report_chain_structured_output.invoke({\"summary_text\": summary.summary})\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Executive Summary of Employee Feedback on IT Operations\n",
       "\n",
       "This report summarizes the key findings from the recent employee feedback survey regarding IT operations.\n",
       "\n",
       "## Key-takeaways\n",
       "\n",
       "- **Network Reliability**: While the network is generally robust, intermittent glitches and slowdowns during peak usage create anxiety and hinder productivity, indicating potential resource constraints during high-demand periods.\n",
       "- **Legacy Systems**: The rigidity of legacy systems is stifling innovation, as outdated processes clash with modern expectations, suggesting a need for a strategic overhaul of existing infrastructure.\n",
       "- **Security Measures**: Proactive security measures are appreciated, but their complexity can overwhelm users, highlighting a need for better user education and streamlined communication.\n",
       "- **IT Support Efficiency**: Although IT support is generally prompt, miscommunications and backlog delays can disrupt workflow, indicating a need for improved internal processes and resource allocation.\n",
       "- **Training Effectiveness**: Training sessions are engaging but often too fast-paced, leaving some employees struggling to keep up, which points to a need for more tailored training approaches.\n",
       "\n",
       "## Suggested improvements\n",
       "\n",
       "- **Upgrade Legacy Systems**: Initiate a phased upgrade of legacy systems to enhance flexibility and innovation, ensuring alignment with modern business needs and reducing frustration among employees.\n",
       "- **Enhance IT Support Communication**: Implement a more structured communication protocol within IT support to minimize miscommunications and streamline the resolution process, potentially reducing backlog delays.\n",
       "- **Revise Training Programs**: Adjust the pacing and depth of training sessions to accommodate varying employee skill levels, ensuring that all staff can fully engage with and benefit from the training provided.\n",
       "\n",
       "By addressing these key areas, we can enhance employee satisfaction and operational efficiency, paving the way for a more innovative and responsive IT environment."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(\n",
    "    \"\\n\\n\".join([report.snappy_title,\n",
    "                 report.intro,\n",
    "                 '## Key-takeaways',report.key_takeaways,\n",
    "                 '## Suggested improvements',report.suggested_improvements,\n",
    "                 report.outro])\n",
    "                 \n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fantastisk! \n",
    "Du har nå fullført kurset *LLM for alle*, og jobbet deg gjennom hvordan du kan bruke metoder og verktøy som LCEL og structured output til å...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ekstraoppgave: Hvordan kunne vi gjort dette bedre?\n",
    "\n",
    "Anta at du har fått levert resultatene fra denne spørreundersøkelsen i fanget av en stressa mellomleder som ber deg levere en rapport han kan presentere ledelsen. \n",
    "Gitt verktøyene du har fått en innføring i gjennom dette kursene (og kanskje andre erfaringer?), hvordan ville du løst oppgaven?\n",
    "\n",
    "Ser du for eksempel noe som kunne vært forbedret i\n",
    "- Rekkefølgen på måten vi leter etter kategorier?\n",
    "- Legger vi får mye/lite vekt på inputen vi fikk om hva ledelsen \"tror\" kategoriene kommer til å være?\n",
    "- Promptingen?\n",
    "- Variablene eller type hintingen i pydantic-klassene?\n",
    "\n",
    "Ville du kanskje gjort det helt annerledes? \n",
    "Now's your chance to try!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prøv deg frem :))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hva er risikoene ved bruk av LLM til denne typen analyse?\n",
    "Ved bruk av store språkmodeller (LLM) til analyse av åpne tekstsvar i spørreundersøkelser er risikoen generelt lavere enn i mer kritiske anvendelser som faktagenerering eller beslutningsstøtte. Likevel finnes det enkelte utfordringer som bør tas på alvor for å sikre kvalitet og transparens i analysen.\n",
    " \n",
    "Skjevheter og bias\n",
    "LLM-er kan tolke og vekte visse formuleringer eller temaer ulikt basert på mønstre fra treningsdataene. Dette kan føre til at noen svar over- eller underrepresenteres i visse kategorier avhengig av hvordan modellen \"forstår\" teksten. For eksempel kan emosjonelt ladede eller språklig avanserte svar få uforholdsmessig stor plass, mens mer nøkterne eller utypiske svar blir feilklassifisert. Særlig ved tolkning av følsomme temaer er det viktig å være oppmerksom på disse tendensene.\n",
    " \n",
    "Hallusinasjon – mindre relevant, men ikke fraværende\n",
    "I denne typen oppgave, hvor modellen hovedsakelig skal kategorisere eller oppsummere eksisterende tekstsvar er risikoen for såkalte hallusinasjoner (altså at modellen «finner opp» informasjon) mindre, men ikke helt ubetydelig. Det kan forekomme at modellen overfortolker et kort eller tvetydig svar og tillegger det en mening som ikke eksplisitt finnes i teksten. Dette kan føre til at enkelte svar havner i feil kategori eller påvirker tolkningen av sentiment eller tematikk.\n",
    " \n",
    "Behov for kvalitetssikring\n",
    "Selv med høy presisjon og effektivitet bør man ikke ta modellens resultater for gitt. Det anbefales å gjennomføre stikkprøver, sammenligne med manuell koding og evaluere nøyaktigheten på klassifiseringene. Det kan være veldig nyttig å kombinere automatisk kategorisering med menneskelig vurdering både i startfasen og underveis."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
