Tracing - rotalabs-audit
Description: Reasoning chain capture and decision transparency for AI systems

Skip to content

# Tracing Module

The `rotalabs_audit.tracing` module provides tools for tracing and analyzing AI decision-making, including decision capture, path analysis, and failure point detection.

## Decision Tracing

Tools for capturing and managing decision traces from AI interactions.

### DecisionTracer

Trace and capture decision points in AI reasoning.

#### `` `DecisionTracer`

Trace and capture decision points in AI reasoning.

Provides tools for creating decision traces from AI interactions, managing active trace sessions, and extracting decision-related information from reasoning text.

Attributes:

Name | Type | Description
--- | --- | ---
`parser` |  | ReasoningChainParser for parsing reasoning text.
`_active_traces` | `Dict[str, _ActiveTrace]` | Dictionary of currently active trace sessions.

Example

> > > tracer = DecisionTracer() trace = tracer.trace_decision( ... prompt="What approach should I use?", ... response="I recommend approach A because...", ... decision="Use approach A", ... ) print(f"Alternatives: {trace.alternatives_considered}")

##### `` `__init__(parser=None)`

Initialize the decision tracer.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`parser` | `Optional[ReasoningChainParser]` | Optional ReasoningChainParser instance. If not provided, a new parser will be created. | `None`

##### `` `trace_decision(prompt, response, decision, model=None, context=None)`

Create a decision trace from an AI interaction.

Parses the response for reasoning, extracts alternatives considered, identifies the rationale, and creates a structured DecisionTrace.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`prompt` | `str` | The prompt/question that led to this decision. | required
`response` | `str` | The full AI response containing the decision. | required
`decision` | `str` | The decision that was made (explicit statement). | required
`model` | `Optional[str]` | Optional model identifier. | `None`
`context` | `Optional[Dict[str, Any]]` | Optional additional context dictionary. | `None`

Returns:

Type | Description
--- | ---
`DecisionTrace` | DecisionTrace capturing the decision details.

Example

> > > trace = tracer.trace_decision( ... prompt="Should we deploy now or wait?", ... response="After considering the risks... I recommend waiting.", ... decision="Wait for deployment", ... model="gpt-4", ... )

##### `` `start_trace(goal, context=None)`

Start tracing a decision path.

Begins a new trace session for capturing a sequence of decisions related to achieving a specific goal.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`goal` | `str` | The goal or objective for this decision path. | required
`context` | `Optional[Dict[str, Any]]` | Optional context that applies to all decisions in the path. | `None`

Returns:

Name | Type | Description
--- | --- | ---
`trace_id` | `str` | Unique identifier for this trace session.

Example

> > > trace_id = tracer.start_trace("Complete the data analysis")

> > > ###### ... make decisions ...

> > > path = tracer.end_trace(trace_id, success=True)

##### `` `add_decision(trace_id, decision)`

Add a decision to an active trace.

Appends a decision trace to an ongoing trace session.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`trace_id` | `str` | The ID of the active trace session. | required
`decision` | `DecisionTrace` | The DecisionTrace to add. | required

Raises:

Type | Description
--- | ---
`KeyError` | If trace_id does not correspond to an active trace.

Example

> > > trace_id = tracer.start_trace("Complete task") decision = tracer.trace_decision(...) tracer.add_decision(trace_id, decision)

##### `` `end_trace(trace_id, success)`

End tracing and return the complete decision path.

Finalizes a trace session, marking it as successful or not, and returns the complete DecisionPath.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`trace_id` | `str` | The ID of the active trace session. | required
`success` | `bool` | Whether the decision path achieved its goal. | required

Returns:

Type | Description
--- | ---
`DecisionPath` | The complete DecisionPath with all decisions.

Raises:

Type | Description
--- | ---
`KeyError` | If trace_id does not correspond to an active trace.

Example

> > > path = tracer.end_trace(trace_id, success=True) print(f"Made {path.length} decisions, success: {path.success}")

##### `` `get_active_trace_ids()`

List all active trace IDs.

Returns:

Type | Description
--- | ---
`List[str]` | List of active trace session IDs.

##### `` `cancel_trace(trace_id)`

Cancel an active trace without creating a DecisionPath.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`trace_id` | `str` | The ID of the trace to cancel. | required

Returns:

Type | Description
--- | ---
`bool` | True if the trace was cancelled, False if not found.

##### `` `extract_alternatives(text)`

Extract alternatives considered from reasoning text.

Looks for phrases indicating alternative options that were considered during decision-making.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The reasoning text to analyze. | required

Returns:

Type | Description
--- | ---
`List[str]` | List of alternatives mentioned in the text.

Example

> > > alternatives = tracer.extract_alternatives( ... "We could use option A, alternatively option B, or option C" ... ) print(alternatives) # ["option B", "option C"]

##### `` `extract_rationale(chain)`

Extract the main rationale from a reasoning chain.

Identifies the primary justification or reasoning that supports the decision made.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`chain` | `ReasoningChain` | The reasoning chain to analyze. | required

Returns:

Type | Description
--- | ---
`str` | The extracted rationale as a string.

Example

> > > chain = parser.parse("I recommend A because it's faster and safer...") rationale = tracer.extract_rationale(chain)

##### `` `assess_reversibility(decision, context)`

Assess if a decision is reversible.

Analyzes the decision text and context to determine if the decision can be easily undone or changed.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`decision` | `str` | The decision statement. | required
`context` | `Dict[str, Any]` | Context dictionary that may contain reversibility hints. | required

Returns:

Type | Description
--- | ---
`bool` | True if the decision appears reversible, False otherwise.

Example

> > > is_reversible = tracer.assess_reversibility( ... decision="Delete the database", ... context={"has_backup": True}, ... )

##### `` `quick_trace(prompt, response, decision, model=None)`

Create a quick decision trace with minimal parsing.

A faster alternative to trace_decision that performs less analysis but creates a valid trace more quickly.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`prompt` | `str` | The prompt that led to this decision. | required
`response` | `str` | The AI response. | required
`decision` | `str` | The decision made. | required
`model` | `Optional[str]` | Optional model identifier. | `None`

Returns:

Type | Description
--- | ---
`DecisionTrace` | A DecisionTrace with basic information.

### ReasoningChainParser

Parser for chain-of-thought reasoning text used in decision tracing.

#### `` `ReasoningChainParser`

Parser for chain-of-thought reasoning text.

Extracts structured reasoning chains from raw text, classifying each step by type and estimating confidence levels.

Example

> > > parser = ReasoningChainParser() chain = parser.parse("First, I'll analyze... Then, considering...") print(f"Found {chain.step_count} reasoning steps")

##### `` `__init__()`

Initialize the parser with compiled regex patterns.

##### `` `parse(text, model=None)`

Parse chain-of-thought text into a structured ReasoningChain.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The chain-of-thought text to parse. | required
`model` | `Optional[str]` | Optional model identifier that generated this text. | `None`

Returns:

Type | Description
--- | ---
`ReasoningChain` | ReasoningChain with parsed steps, confidence, and depth.

Raises:

Type | Description
--- | ---
`ValueError` | If text is empty or invalid.

Example

> > > parser = ReasoningChainParser() chain = parser.parse("1. First, let me analyze the problem...") print(chain.steps[0].reasoning_type)

## Decision Path Analysis

Tools for analyzing complete decision paths and identifying patterns.

### DecisionPathAnalyzer

Analyze sequences of decisions in a decision path.

#### `` `DecisionPathAnalyzer`

Analyze sequences of decisions in a decision path.

Provides tools for analyzing complete decision paths, identifying critical decisions, finding failure points, and generating summaries.

Example

> > > analyzer = DecisionPathAnalyzer() analysis = analyzer.analyze_path(path) print(f"Critical decisions: {len(analysis['critical_decisions'])}")

##### `` `__init__()`

Initialize the decision path analyzer.

##### `` `analyze_path(path)`

Analyze a complete decision path.

Performs comprehensive analysis of a decision path including statistics, critical decisions, and quality metrics.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to analyze. | required

Returns:

Type | Description
--- | ---
`Dict[str, Any]` | Dictionary containing analysis results: - decision_count: Number of decisions in the path - avg_confidence: Average confidence across decisions - min_confidence: Minimum confidence in any decision - max_confidence: Maximum confidence in any decision - critical_decisions: List of critical decision IDs - reversible_count: Number of reversible decisions - irreversible_count: Number of irreversible decisions - success: Whether the path was successful - failure_point_id: ID of the failure point (if applicable) - total_alternatives: Total alternatives considered across all decisions

Example

> > > analysis = analyzer.analyze_path(path) print(f"Success rate: {analysis['success']}") print(f"Avg confidence: {analysis['avg_confidence']:.2f}")

##### `` `find_critical_decisions(path)`

Find decisions that were critical to the outcome.

Identifies decisions that significantly impacted the path's success or failure based on various criteria.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to analyze. | required

Returns:

Type | Description
--- | ---
`List[DecisionTrace]` | List of critical DecisionTrace objects.

Criteria for critical decisions

- Irreversible decisions (cannot be undone)
- Low confidence decisions (uncertainty)
- Decisions with multiple alternatives
- First and last decisions in the path

Example

> > > critical = analyzer.find_critical_decisions(path) for decision in critical: ... print(f"Critical: {decision.decision[:50]}...")

##### `` `find_failure_point(path)`

If path failed, find where it went wrong.

Analyzes a failed decision path to identify the decision most likely responsible for the failure.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to analyze. | required

Returns:

Type | Description
--- | ---
`Optional[DecisionTrace]` | The DecisionTrace that likely caused failure, or None if
`Optional[DecisionTrace]` | the path succeeded or has no decisions.

Heuristics for finding failure point

- Check if path already has a failure_point set
- Last irreversible decision before failure
- Decision with lowest confidence
- Decision with most alternatives (indecision)

Example

> > > if not path.success: ... failure = analyzer.find_failure_point(path) ... if failure: ... print(f"Failed at: {failure.decision}")

##### `` `calculate_path_confidence(path)`

Calculate overall confidence in the decision path.

Computes a weighted confidence score based on individual decision confidences, with emphasis on critical decisions.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to analyze. | required

Returns:

Type | Description
--- | ---
`float` | Overall confidence score between 0.0 and 1.0.

The calculation

- Base: weighted average of all decision confidences
- Penalty for irreversible low-confidence decisions
- Penalty for having many alternatives (indecision)

Example

> > > confidence = analyzer.calculate_path_confidence(path) if confidence < 0.5: ... print("Warning: Low confidence path")

##### `` `summarize_path(path)`

Generate a human-readable summary of the decision path.

Creates a text summary describing the path's goal, key decisions, outcome, and any notable observations.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to summarize. | required

Returns:

Type | Description
--- | ---
`str` | Human-readable summary string.

Example

> > > summary = analyzer.summarize_path(path) print(summary)

##### `` `compare_paths(path1, path2)`

Compare two decision paths.

Analyzes differences between two decision paths including confidence, decision count, and outcome.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path1` | `DecisionPath` | First decision path. | required
`path2` | `DecisionPath` | Second decision path. | required

Returns:

Type | Description
--- | ---
`Dict[str, Any]` | Dictionary with comparison results.

Example

> > > comparison = analyzer.compare_paths(path_a, path_b) print(f"Better path: {comparison['better_path']}")

##### `` `find_divergence_point(path1, path2)`

Find where two decision paths diverge.

Compares decisions in order to find the first point where the paths made different decisions.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path1` | `DecisionPath` | First decision path. | required
`path2` | `DecisionPath` | Second decision path. | required

Returns:

Type | Description
--- | ---
`Optional[int]` | Index of the first divergent decision, or None if paths
`Optional[int]` | don't diverge or one is a prefix of the other.

Example

> > > divergence = analyzer.find_divergence_point(path_a, path_b) if divergence is not None: ... print(f"Paths diverged at decision {divergence}")

##### `` `get_confidence_trend(path)`

Get the confidence trend across decisions.

Returns the sequence of confidence values to identify patterns like declining confidence.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to analyze. | required

Returns:

Type | Description
--- | ---
`List[float]` | List of confidence values in decision order.

Example

> > > trend = analyzer.get_confidence_trend(path) if trend[-1] < trend[0]: ... print("Confidence declined over the path")

##### `` `detect_confidence_decline(path, threshold=0.2)`

Detect if confidence declined significantly over the path.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`path` | `DecisionPath` | The decision path to analyze. | required
`threshold` | `float` | Minimum decline to be considered significant. | `0.2`

Returns:

Type | Description
--- | ---
`bool` | True if confidence declined by more than the threshold.

Example

> > > if analyzer.detect_confidence_decline(path): ... print("Warning: Confidence declined over the decision path")