---
title: "Insurance Customer Service Agent"
description: "A walkthrough of running Arksim against a customer service agent for an insurance company use case."
---

## Overview

This example walks through running Arksim against a customer service agent built for an insurance company (XYZ Insurance, part of XYZ Bank Group). The agent is designed to answer customer questions about insurance products and coverage, including topics like policy details, claims processes, deductibles, and coverage limits.

The example includes two ready-to-run agent implementations you can test against out of the box, and a guide for plugging in your own agent once you're familiar with the setup.

---

## Example Agents

<CardGroup cols={2}>
  <Card icon="bolt" title="Option 1: OpenAI API">
    A lightweight agent that calls the OpenAI API directly. Quick start with minimal setup; default agent_config.json is preconfigured.
  </Card>
  <Card icon="diagram-project" title="Option 2: OpenAI Agents SDK RAG">
    RAG-based agent built with the OpenAI Agents SDK, backed by an insurance knowledge base. Supports A2A or Chat Completions; use agent_config.json when running.
  </Card>
</CardGroup>

---

## Scenarios

The example ships with a set of pre-built scenarios in `scenarios.json` representing realistic insurance customer interactions. Each scenario defines a simulated user with a distinct persona, goal, and background knowledge drawn from insurance product documentation.

The scenario goals are:

- Learn how home insurance deductibles work, when to file a claim, and how they affect your premium and payout.
- Get a denied water damage claim (water heater 17 years old; policy excludes tanks 15+) overturned or learn how to fight it, including legal options; push back if the agent only repeats the policy.
- Renew Basic Form home insurance at the same price; push back if the agent upsells Broad or Comprehensive Form or extra features.
- Get a clear recommendation and dollar amount for personal condo insurance as a first-time buyer, without lengthy needs questions.
- Bundle home, two cars, and motorcycle with XYZ and get a specific savings number; resist needs questions or comparisons.

Scenarios are defined in `scenarios.json` in the example directory and can be edited or extended to reflect your own use case.

---

## Option 1: OpenAI Agent

This agent calls the OpenAI API directly, with no server setup required.

<Steps>
  <Step title="Set your API key">
    ```bash
    export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
    ```
  </Step>
  <Step title="Run simulation and evaluation">
    Run from the `examples/bank-insurance` directory:

    ```bash
    cd examples/bank-insurance
    arksim simulate-evaluate config.yaml
    ```
  </Step>
</Steps>

---

## Option 2: In-house Agent (OpenAI Agents SDK RAG)

This agent is a RAG-based customer service agent built with the OpenAI Agents SDK, backed by an insurance knowledge base. It can be exposed via either the A2A Protocol or a Chat Completions-compatible endpoint.

<Steps>
  <Step title="Select agent config">
    In the example directory, use the agent config for your chosen interface. If `agent_config.json` already exists, rename or remove it first.

    - **A2A:** Copy `agent_config_a2a.json` to `agent_config.json`
    - **Chat Completions:** Copy `agent_config_chat_completions.json` to `agent_config.json`

    Agent configs support environment variable substitution (e.g. `${AGENT_API_KEY}` for Chat Completions, `${A2A_API_KEY}` for A2A).
  </Step>
  <Step title="Install agent dependencies">
    <Tabs>
      <Tab title="uv">
        ```bash
        uv venv --python 3.11
        source .venv/bin/activate
        pip install -r examples/bank-insurance/agent_server/requirements.txt
        ```
      </Tab>
      <Tab title="conda">
        ```bash
        conda create -n bank_venv python=3.11 -y
        conda activate bank_venv
        pip install -r examples/bank-insurance/agent_server/requirements.txt
        ```
      </Tab>
    </Tabs>
    Or from inside the example directory: `pip install -r agent_server/requirements.txt`
  </Step>
  <Step title="Start the agent server">
    <Tabs>
      <Tab title="A2A">
        Exposes an A2A-compatible agent on port `9999`. Run from the repository root:

        ```bash
        export A2A_API_KEY=<YOUR_A2A_API_KEY>
        python -m examples.bank-insurance.agent_server.a2a.server
        ```

        Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/a2a/server.py` under the bank-insurance example.
      </Tab>
      <Tab title="Chat Completions">
        Exposes an OpenAI Chat Completions-compatible endpoint on port `8888` at `/chat/completions`. Run from the repository root:

        ```bash
        export AGENT_API_KEY=<YOUR_AGENT_API_KEY>
        python -m examples.bank-insurance.agent_server.chat_completions.server
        ```
      </Tab>
    </Tabs>
  </Step>
  <Step title="Run simulation and evaluation">
    In a new terminal, run from the `examples/bank-insurance` directory. Set the key that matches your agent config (AGENT_API_KEY for Chat Completions, A2A_API_KEY for A2A), then run:

    ```bash
    cd examples/bank-insurance
    export AGENT_API_KEY="<YOUR_AGENT_API_KEY>"   # for Chat Completions
    # export A2A_API_KEY="<YOUR_A2A_API_KEY>"     # for A2A
    arksim simulate-evaluate config.yaml
    ```
  </Step>
</Steps>

---

## Running with Your Own Agent

To test your own backend agent against these scenarios:

- **Chat Completions:** Follow the comments in `agent_server/chat_completions/server.py` to swap in your own backend logic or point to your endpoint.
- **A2A:** Implement your own A2A executor in `agent_server/a2a/`.

Ensure `agent_config.json` points to your endpoint (or use the appropriate pre-made config as a template). Then run simulation and evaluation as above.

<Tip>
  See [Agent configuration](/simulate-conversation#agent-configuration) for supported protocols and how to configure agent connections.
</Tip>

---

## Configuration

The example uses a single config file for both simulation and evaluation.

```yaml
# SIMULATION SETTINGS

# Path to agent api configuration file
agent_config_file_path: ./agent_config.json

# Path to the scenarios file
scenario_file_path: ./scenarios.json

# Number of conversations per scenario to generate
num_conversations_per_scenario: 1

# Maximum turns per conversation
max_turns: 5

# Output file path for simulation results
output_file_path: ./results/simulation/simulation.json

# Jinja template for the simulated user's system prompt
simulated_user_prompt_template: null

# EVALUATION SETTINGS

# Output directory for evaluation results
output_dir: ./results/evaluation

# Provide paths to Python files that define your custom metrics.
# Custom quantitative metrics should subclass QuantitativeMetric
# and implement score(); qualitative metrics should subclass
# QualitativeMetric and implement evaluate().
# (from arksim.evaluator import QuantitativeMetric, QualitativeMetric)
custom_metrics_file_paths:
  - ./custom_metrics.py

# Provide the built-in metrics to be run during evaluation.
# If empty, all built-in metrics run. Custom metrics defined in
# custom_metrics_file_paths always run regardless of this list.
metrics_to_run:
  - faithfulness
  - helpfulness
  - coherence
  - relevance
  - goal_completion
  - agent_behavior_failure

# Generate HTML report
generate_html_report: true

# Score threshold for per-conversation final scores (0.0 to 1.0)
# If any conversation's final score is below this threshold,
# exit with non-zero code.
# Set to null to disable threshold checking
score_threshold: null

# SHARED SETTINGS

# LLM model
model: gpt-5.1

# LLM provider
provider: openai

# Number of workers for parallel processing. "auto" uses all
# available threads; on large runs this can trigger LLM rate
# limits — set an explicit integer to cap concurrency.
num_workers: auto
```

The config also defines a `simulated_user_prompt_template` (Jinja) that uses `scenario.agent_context`, `scenario.goal`, `scenario.knowledge`, and `scenario.user_profile` to drive the simulated user.

<Note>
  The example directory contains `agent_config.json` (default OpenAI), `agent_config_a2a.json`, and `agent_config_chat_completions.json`, the insurance knowledge base under `agent_server/data/`, and the scenario file `scenarios.json`.
</Note>

---

## Output

Results are written under the example directory:

| Location                                 | Contents                                                                 |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| `./results/simulation/simulation.json`   | Simulated conversations from the simulation step                         |
| `./results/evaluation/evaluation.json`   | Evaluation results (per-turn and per-conversation scores, unique errors) |
| `./results/evaluation/final_report.html` | Interactive HTML report for browsing and sharing results                 |
