# Simulator Report

Metrics
forum
Total Conversations
-
sync
Avg Turns per Conversation
-
Performance Metrics

### science How Scores Are Computed expand_more

#### Metric Definitions

Each turn is evaluated on five quantitative metrics using LLM-as-a-judge:

- Helpfulness — how effectively the response addresses the user's request
- Coherence — logical flow and consistency of the response
- Verbosity — response length appropriateness (inverted: higher = more concise)
- Relevance — how on-topic the response is relative to the user's query
- Faithfulness — accuracy relative to the provided knowledge base (only flags contradictions, not missing information)

#### Score Scale

Range | Label | Meaning
--- | --- | ---
1.0 – 2.0 | Poor | Response fails to meet basic expectations
2.0 – 3.0 | Needs Improvement | Partially addresses the task but has significant gaps
3.0 – 4.0 | Good | Adequately addresses the task with minor issues
4.0 – 5.0 | Excellent | Fully addresses the task with high quality

#### Behaviour Failure Detection

If any turn-level metric scores below `3.0`, a qualitative behaviour failure evaluation is triggered. The threshold of 3.0 ("Good") was chosen because scores below this indicate the response has significant gaps worth diagnosing. Failure categories include:

- Failure to ask for clarification — agent proceeds with an ambiguous request instead of asking
- Repetition — redundant restatement within or across turns
- Disobey user request — agent ignores or misinterprets a clear request
- Lack of specific information — correct but generic/incomplete answer
- False information — hallucinated or factually incorrect content

Each score includes an LLM-generated reasoning explaining why the score was given. For turns with behaviour failures, the detailed failure reason is shown in the conversation modal under "Issue Details".

#### Conversation Score Formula

Each conversation receives a Final Score (0–1) computed as:

`final_score = turn_success_ratio × 0.75 + goal_completion × 0.25`

- Turn Success Ratio (weight: 75%) = `(total_turns − behaviour_failures) / total_turns`. Measures the fraction of turns without agent behavior failures. Weighted higher because consistent turn-level quality is the primary indicator of a reliable agent.
- Goal Completion (weight: 25%) = `1` if the user's goal was successfully addressed, `0` otherwise. Evaluated by LLM at conversation end. Weighted lower because an agent can complete the goal while still exhibiting problematic behaviour in individual turns.

Conversation status is determined by the final score:

- `1.0` = Done — perfect score, no failures
- `≥ 0.6` = Partial Failure — goal may be met but with some turn-level issues
- `< 0.6` = Failed — significant quality issues

#### Unique Errors

After all turns are evaluated, the agent behavior failure reasons are collected and deduplicated across conversations using an LLM. This produces a list of unique errors, each with a root cause description and the specific conversation turns where it occurred. For automated fix suggestions, see arklex.ai.

For more on agent simulation evaluation, see the SAGE paper.
Qualitative Metrics
Unique Errors expand_more
Identified unique errors by analyzing and duplicating recurring failure patterns.
forumConversations

chat id | scenario id | goal completion score info goal completion score indicates whether the agent has completed the goal of the user (ranges between 0-1). | final score info a weighted sum of turn_success_ratio and goal_completion_score (goal_completion_weight = 0.25, turn_success_ratio_weight = 0.75). ranges between 0-1. | status info the status of the conversation based on final_score: done: final_score == 1.0 - perfect performance with no agent behavior failures and goal completed. partial failure: final_score > 0.6 - some agent behavior failures or goal not completed, but performance is acceptable. failed: final_score ≤ 0.6 - too many agent behavior failures or goal not completed, indicating poor performance.
--- | --- | --- | --- | ---

description Evaluation Prompts Appendix expand_more

## Conversation Details