---
title: "E-Commerce Customer Service Agent"
description: "A walkthrough of running Arksim against a shopping assistant agent for an e-commerce use case."
---

## Overview

This example walks through running Arksim against a shopping assistant agent built for an e-commerce use case. The agent is designed to help customers navigate product discovery, orders, returns, and general shopping queries.

The example includes two ready-to-run agent setups you can test against out of the box, and a guide for plugging in your own agent once you're familiar with the setup.

---

## Example Agents

<CardGroup cols={2}>
  <Card icon="bolt" title="Option 1: OpenAI API">
    A lightweight agent that calls the OpenAI API directly. Quick start with minimal setup; default agent_config.json is preconfigured.
  </Card>
  <Card icon="diagram-project" title="Option 2: LangGraph RAG">
    RAG-based agent with OpenAI Agents SDK, backed by an e-commerce knowledge base; supports A2A or Chat Completions.
  </Card>
</CardGroup>

---

## Scenarios

The example ships with a set of pre-built scenarios representing realistic e-commerce customer interactions. Each scenario defines a simulated user with a distinct persona, goal, and background knowledge drawn from product and policy documentation.

Sample goals include:

- Asking about product availability, specifications, and delivery timelines
- Checking order status and tracking a shipment
- Initiating or following up on a return or refund request
- Comparing products and asking for recommendations

Scenarios are located at `examples/e-commerce/results/scenario/` and can be swapped out or extended to reflect your own use case.

---

## Option 1: OpenAI Agent

This agent calls the OpenAI API directly, with no server setup required.

<Steps>
  <Step title="Set your API key">
    ```bash
    export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"
    ```
  </Step>
  <Step title="Run simulation and evaluation">
    Run from the `examples/e-commerce` directory:

    ```bash
    cd examples/e-commerce
    arksim simulate-evaluate config.yaml
    ```
  </Step>
</Steps>

---

## Example Agent 2: RAG Agent

This agent is a RAG-based shopping assistant built with OpenAI Agents SDK, backed by an e-commerce knowledge base. It can be exposed via either the A2A Protocol or a Chat Completions-compatible endpoint.

<Steps>
  <Step title="Select agent config">
    In the example directory, use the agent config for your chosen interface. If `agent_config.json` already exists, rename or remove it first.

    - **A2A:** Copy `agent_config_a2a.json` to `agent_config.json`
    - **Chat Completions:** Copy `agent_config_chat_completions.json` to `agent_config.json`

    Agent configs support environment variable substitution (e.g. `${AGENT_API_KEY}` for Chat Completions, `${A2A_API_KEY}` for A2A).
  </Step>
  <Step title="Install agent dependencies">
    <Tabs>
      <Tab title="uv">
        ```bash
        uv venv --python 3.11
        source .venv/bin/activate
        pip install -r examples/e-commerce/agent_server/requirements.txt
        ```
      </Tab>
      <Tab title="conda">
        ```bash
        conda create -n ecommerce_venv python=3.11 -y
        conda activate ecommerce_venv
        pip install -r examples/e-commerce/agent_server/requirements.txt
        ```
      </Tab>
    </Tabs>
    Or from inside the example directory: `pip install -r agent_server/requirements.txt`
  </Step>
  <Step title="Start the agent server">
    <Tabs>
      <Tab title="A2A">
        Exposes an A2A-compatible agent on port `9999`. Run from the repository root:

        ```bash
        export A2A_API_KEY=<YOUR_A2A_API_KEY>
        python -m examples.e_commerce.agent_server.a2a.server
        ```

        Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/a2a/server.py` under the e-commerce example.
      </Tab>
      <Tab title="Chat Completions">
        Exposes an OpenAI Chat Completions-compatible endpoint on port `8080` at `/chat/completions`. Run from the repository root:

        ```bash
        export AGENT_API_KEY=<YOUR_AGENT_API_KEY>
        python -m examples.e_commerce.agent_server.chat_completions.server
        ```
      </Tab>
    </Tabs>
  </Step>
  <Step title="Run simulation and evaluation">
    In a **new terminal**, run from the `examples/e-commerce` directory. Set the key that matches your agent config (AGENT_API_KEY for Chat Completions, A2A_API_KEY for A2A), then run:

    ```bash
    cd examples/e-commerce
    export AGENT_API_KEY="<YOUR_AGENT_API_KEY>"   # for Chat Completions
    # export A2A_API_KEY="<YOUR_A2A_API_KEY>"     # for A2A
    arksim simulate-evaluate config.yaml
    ```
  </Step>
</Steps>

---

## Running with Your Own Agent

To test your own backend agent against these scenarios, follow the `#TODO` comments in `chat_completion_wrapper.py` to call your backend instead of the sample agent.

Once your server is running on a Chat Completions–compatible endpoint, point `agent_config.json` at it and run simulation and evaluation as normal.

<Tip>
  See [Agent configuration](/simulate-conversation#agent-configuration) for supported protocols and how to configure agent connections.
</Tip>

---

## Configuration

The example uses a single config file for both simulation and evaluation.

```yaml
# SIMULATION SETTINGS

# Path to agent api configuration file
agent_config_file_path: ./agent_config.json

# Path to the scenarios file
scenario_file_path: ./scenarios.json

# Number of conversations per scenario to generate
num_conversations_per_scenario: 1

# Maximum turns per conversation
max_turns: 5

# Output file path for simulation results
output_file_path: ./results/simulation/simulation.json

# Jinja template for the simulated user's system prompt
simulated_user_prompt_template: null

# EVALUATION SETTINGS

# Output directory for evaluation results
output_dir: ./results/evaluation

# Provide paths to Python files that define your custom metrics.
# Custom quantitative metrics should subclass QuantitativeMetric
# and implement score(); qualitative metrics should subclass
# QualitativeMetric and implement evaluate().
# (from arksim.evaluator import QuantitativeMetric, QualitativeMetric)
custom_metrics_file_paths:
  - ./custom_metrics.py

# Provide the built-in metrics to be run during evaluation.
# If empty, all built-in metrics run. Custom metrics defined in
# custom_metrics_file_paths always run regardless of this list.
metrics_to_run:
  - faithfulness
  - helpfulness
  - coherence
  - verbosity
  - relevance
  - goal_completion
  - agent_behavior_failure

# Generate HTML report
generate_html_report: true

# Score threshold for per-conversation final scores (0.0 to 1.0)
# If any conversation's final score is below this threshold,
# exit with non-zero code.
# Set to null to disable threshold checking
score_threshold: null

# SHARED SETTINGS

# LLM model
model: gpt-5.1

# LLM provider
provider: openai

# Number of workers for parallel processing. "auto" uses all
# available threads; on large runs this can trigger LLM rate
# limits — set an explicit integer to cap concurrency.
num_workers: auto
```

<Note>
  `agent_setup_dir` contains the `agent_config.json` and `knowledge.json` used to drive both simulation and evaluation. For evaluation, point `input_dir` at the folder containing `conversations.json` produced by the simulation step.
</Note>

---

## Output

**Simulation** writes to `examples/e-commerce/results/conversation/`:

- `conversations.json`: full transcripts of every simulated conversation
- `target_goals.json`: the resolved goals used to drive each simulated user

**Evaluation** writes to `examples/e-commerce/results/evaluation/`:

| File                                     | Contents                                                              |
| ---------------------------------------- | --------------------------------------------------------------------- |
| `agent_performance_per_turn.csv`         | Per-turn metric scores, reasoning, and failure labels                 |
| `agent_performance_per_conversation.csv` | Rolled-up scores and status per conversation                          |
| `unique_errors.csv`                      | Deduplicated failure patterns with descriptions and occurrence traces |
| `final_report.html`                      | Interactive report for browsing and sharing results                   |
