Chains - rotalabs-audit
Description: Reasoning chain capture and decision transparency for AI systems

Skip to content

# Chains Module

The `rotalabs_audit.chains` module provides extended reasoning chain parsing and pattern analysis capabilities. It includes comprehensive pattern libraries for detecting various types of reasoning, confidence estimation from linguistic markers, and utilities for analyzing reasoning distributions.

This module complements the core parser with additional capabilities:

- Extensive Pattern Library: Comprehensive regex patterns for detecting evaluation awareness, goal reasoning, meta-cognition, and more.
- Confidence Estimation: Linguistic analysis of confidence markers (hedging, certainty expressions) to estimate confidence levels.
- Distribution Analysis: Tools for analyzing confidence distributions across reasoning chains.
- Format Detection: Automatic detection of reasoning format (numbered, bulleted, prose, etc.).

## Parser Classes

### ExtendedReasoningParser

Enhanced reasoning chain parser with rich pattern matching capabilities.

#### `` `ReasoningChainParser`

Parse natural language reasoning into structured chains.

This class provides the main interface for converting free-form reasoning text into structured ReasoningChain objects with classified steps, confidence scores, and supporting evidence.

The parser supports multiple input formats: - Numbered lists (1., 2., 3.) - Lettered lists (a., b., c.) - Bullet points (-, *, +) - Arrow sequences (=>, ->) - Sequential words (first, second, then) - Continuous prose (split by sentences)

Attributes:

Name | Type | Description
--- | --- | ---
`config` |  | Parser configuration settings.

Example

> > > parser = ReasoningChainParser() chain = parser.parse(''' ... I think we should approach this step by step. ... 1. First, consider the constraints ... 2. Then, evaluate possible solutions ... 3. Finally, select the best option ... ''') print(chain.summary())

> > > ##### With custom configuration

> > > config = ParserConfig(min_step_length=20, confidence_threshold=0.3) parser = ReasoningChainParser(config=config) chain = parser.parse(text, model="claude-3-opus")

##### `` `__init__(config=None)`

Initialize the reasoning chain parser.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`config` | `Optional[ParserConfig]` | Optional parser configuration. Uses defaults if not provided. | `None`

Example

> > > parser = ReasoningChainParser() parser = ReasoningChainParser(config=ParserConfig(min_step_length=5))

##### `` `parse(text, model=None)`

Parse reasoning text into a structured chain.

This is the main entry point for parsing. It: 1. Detects the format of the input text 2. Splits the text into individual steps 3. Classifies each step's reasoning type 4. Estimates confidence for each step 5. Aggregates results into a ReasoningChain

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The reasoning text to parse. | required
`model` | `Optional[str]` | Optional identifier of the AI model that generated the text. | `None`

Returns:

Type | Description
--- | ---
`ReasoningChain` | A ReasoningChain containing parsed and classified steps.

Example

> > > parser = ReasoningChainParser() chain = parser.parse(''' ... Let me think through this: ... 1. The problem asks for X ... 2. I believe the answer involves Y ... 3. Therefore, the solution is Z ... ''', model="gpt-4") print(f"Found {len(chain)} steps") Found 3 steps for step in chain: ... print(f"Step {step.index}: {step.reasoning_type.value}")

##### `` `parse_step(text, index)`

Parse a single reasoning step.

This method processes a single piece of text, classifying its reasoning type and estimating confidence.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The text content of this step. | required
`index` | `int` | The position of this step in the chain (0-indexed). | required

Returns:

Type | Description
--- | ---
`ReasoningStep` | A ReasoningStep with classification and confidence.

Example

> > > parser = ReasoningChainParser() step = parser.parse_step("I think the answer is probably 42", 0) print(f"Type: {step.reasoning_type}, Confidence: {step.confidence:.2f}") Type: ReasoningType.META_REASONING, Confidence: 0.35

##### `` `classify_reasoning_type(text)`

Classify the reasoning type with evidence.

This method matches the text against all reasoning patterns and returns the best-matching type along with evidence of which patterns matched.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The text to classify. | required

Returns:

Type | Description
--- | ---
`ReasoningType` | A tuple of (ReasoningType, evidence_dict) where evidence_dict
`Dict[str, List[str]]` | maps pattern categories to lists of matched strings.

Example

> > > parser = ReasoningChainParser() rtype, evidence = parser.classify_reasoning_type( ... "I believe this is correct because of the evidence" ... ) print(f"Type: {rtype}") Type: ReasoningType.META_REASONING print(f"Evidence: {evidence}") Evidence: {'meta_reasoning': ['i believe'], 'causal_reasoning': ['because']}

##### `` `split_into_steps(text)`

Split text into reasoning steps.

This method detects the format of the text and uses the appropriate splitting strategy. It handles: - Numbered lists (1., 2., 3.) - Lettered lists (a., b., c.) - Bullet points (-, *, +) - Arrow sequences - Sentence-based splitting for prose

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The text to split into steps. | required

Returns:

Type | Description
--- | ---
`List[str]` | A list of strings, each representing one reasoning step.

Example

> > > parser = ReasoningChainParser() steps = parser.split_into_steps(''' ... 1. First step ... 2. Second step ... 3. Third step ... ''') print(steps) ['First step', 'Second step', 'Third step']

> > > steps = parser.split_into_steps("First, do X. Then, do Y. Finally, Z.") print(len(steps)) 3

### ExtendedReasoningChain

A complete chain of reasoning steps with aggregate statistics.

#### `` `ReasoningChain` `dataclass`

A complete chain of reasoning steps.

This class represents the full parsed output of reasoning text, including all steps, aggregate statistics, and metadata about the source and parsing process.

Attributes:

Name | Type | Description
--- | --- | ---
`id` | `str` | Unique identifier for this chain.
`steps` | `List[ReasoningStep]` | List of reasoning steps in order.
`source_text` | `str` | Original text that was parsed.
`model` | `Optional[str]` | AI model that generated the reasoning (if known).
`detected_format` | `StepFormat` | Format detected in the source text.
`aggregate_confidence` | `float` | Combined confidence across all steps.
`primary_types` | `List[ReasoningType]` | Most common reasoning types in the chain.
`metadata` | `Dict[str, Any]` | Additional custom metadata.
`parsed_at` | `datetime` | When this chain was parsed.

Example

> > > chain = ReasoningChain( ... steps=[step1, step2, step3], ... source_text="1. First... 2. Then... 3. Finally...", ... model="gpt-4", ... detected_format=StepFormat.NUMBERED, ... ) print(f"Steps: {len(chain)}, Confidence: {chain.aggregate_confidence:.2f}")

##### `` `__len__()`

Return the number of steps in the chain.

##### `` `__iter__()`

Iterate over steps in the chain.

##### `` `__getitem__(index)`

Get a step by index.

##### `` `get_steps_by_type(reasoning_type)`

Get all steps of a specific reasoning type.

##### `` `get_low_confidence_steps(threshold=0.4)`

Get steps below a confidence threshold.

##### `` `to_dict()`

Convert chain to dictionary representation.

##### `` `summary()`

Generate a human-readable summary of the chain.

### ExtendedReasoningStep

A single step in a reasoning chain with classification and metadata.

#### `` `ReasoningStep` `dataclass`

A single step in a reasoning chain.

This class represents one discrete unit of reasoning, including its content, classification, confidence score, and supporting evidence.

Attributes:

Name | Type | Description
--- | --- | ---
`id` | `str` | Unique identifier for this step.
`index` | `int` | Position in the reasoning chain (0-indexed).
`content` | `str` | The text content of this step.
`reasoning_type` | `ReasoningType` | Primary classification of reasoning type.
`secondary_types` | `List[ReasoningType]` | Additional reasoning types detected.
`confidence` | `float` | Confidence score (0.0-1.0).
`confidence_level` | `ConfidenceLevel` | Categorical confidence level.
`evidence` | `Dict[str, List[str]]` | Pattern matches supporting the classification.
`metadata` | `Dict[str, Any]` | Additional custom metadata.
`timestamp` | `datetime` | When this step was parsed.

Example

> > > step = ReasoningStep( ... index=0, ... content="I think the answer is 42", ... reasoning_type=ReasoningType.META_REASONING, ... confidence=0.75, ... )

##### `` `to_dict()`

Convert step to dictionary representation.

### ExtendedParserConfig

Configuration for the extended reasoning chain parser.

#### `` `ParserConfig` `dataclass`

Configuration for the reasoning chain parser.

This class allows customization of parsing behavior, including how steps are split, minimum step length, and confidence thresholds.

Attributes:

Name | Type | Description
--- | --- | ---
`min_step_length` | `int` | Minimum characters for a valid step (default: 10).
`max_step_length` | `int` | Maximum characters per step before truncation (default: 2000).
`split_on_sentences` | `bool` | Whether to split prose into sentences (default: True).
`confidence_threshold` | `float` | Minimum confidence to include a step (default: 0.0).
`include_evidence` | `bool` | Whether to include pattern match evidence (default: True).
`normalize_whitespace` | `bool` | Whether to normalize whitespace in steps (default: True).
`preserve_empty_steps` | `bool` | Whether to keep empty steps (default: False).

Example

> > > config = ParserConfig( ... min_step_length=20, ... confidence_threshold=0.3, ... include_evidence=False, ... ) parser = ReasoningChainParser(config=config)

## Enumerations

### ExtendedReasoningType

Categories of reasoning detected in model outputs.

#### `` `ReasoningType`

Bases: `str`, `Enum`

Categories of reasoning detected in model outputs.

These types help classify the nature of reasoning being performed, which is useful for auditing AI behavior and detecting potential issues like evaluation gaming or misaligned goals.

Attributes:

Name | Type | Description
--- | --- | ---
`EVALUATION_AWARE` |  | Model shows awareness of being tested/evaluated.
`GOAL_REASONING` |  | Model expresses goals or objectives.
`DECISION_MAKING` |  | Model makes choices or selections.
`META_REASONING` |  | Model reasons about its own reasoning.
`UNCERTAINTY` |  | Model expresses doubt or hedging.
`INCENTIVE_REASONING` |  | Model reasons about rewards/penalties.
`CAUSAL_REASONING` |  | Model uses cause-effect logic.
`HYPOTHETICAL` |  | Model explores hypothetical scenarios.
`GENERAL` |  | No specific reasoning type detected.

Example

> > > rtype = ReasoningType.META_REASONING print(f"Type: {rtype.value}") Type: meta_reasoning

### ExtendedConfidenceLevel

Categorical confidence levels for reasoning steps.

#### `` `ConfidenceLevel`

Bases: `str`, `Enum`

Categorical confidence levels for reasoning steps.

These levels provide a human-readable interpretation of numeric confidence scores, useful for filtering and reporting.

Attributes:

Name | Type | Description
--- | --- | ---
`VERY_LOW` |  | Score < 0.2, highly uncertain language.
`LOW` |  | Score 0.2-0.4, tentative or hedged statements.
`MODERATE` |  | Score 0.4-0.6, balanced or neutral confidence.
`HIGH` |  | Score 0.6-0.8, assertive but not absolute.
`VERY_HIGH` |  | Score >= 0.8, highly confident assertions.

Example

> > > level = ConfidenceLevel.HIGH print(f"Confidence: {level.value}") Confidence: high

### StepFormat

Detected format of reasoning step markers.

#### `` `StepFormat`

Bases: `str`, `Enum`

Detected format of reasoning step markers.

Attributes:

Name | Type | Description
--- | --- | ---
`NUMBERED` |  | Steps marked with numbers (1., 2., 3.)
`LETTERED` |  | Steps marked with letters (a., b., c.)
`BULLET` |  | Steps marked with bullets (-, *, +)
`ARROW` |  | Steps marked with arrows (=>, ->)
`SEQUENTIAL_WORDS` |  | Steps using words (first, second, then)
`PROSE` |  | Continuous prose without explicit markers

## Confidence Functions

Functions for estimating and aggregating confidence from linguistic markers.

### estimate_confidence

Estimate confidence level from linguistic markers in text.

#### `` `estimate_confidence(text)`

Estimate confidence level from linguistic markers in text.

This function analyzes text for high-confidence and low-confidence indicators, returning a normalized score between 0.0 and 1.0.

The scoring algorithm: 1. Count matches for high-confidence patterns (adds to score) 2. Count matches for low-confidence patterns (subtracts from score) 3. Normalize based on total matches 4. Return 0.5 (moderate) if no indicators found

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`text` | `str` | The text to analyze for confidence indicators. | required

Returns:

Type | Description
--- | ---
`float` | A float between 0.0 (very uncertain) and 1.0 (very certain).
`float` | Returns 0.5 if no confidence indicators are found.

Example

> > > estimate_confidence("I am definitely sure about this") 0.85 estimate_confidence("Maybe this could be right") 0.2 estimate_confidence("The answer is 42") 0.5 estimate_confidence("I am certain, but there might be exceptions") 0.6

### get_confidence_level

Convert a numeric confidence score to a categorical level.

#### `` `get_confidence_level(score)`

Convert a numeric confidence score to a categorical level.

This function maps continuous confidence scores to discrete levels for easier interpretation and filtering.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`score` | `float` | A confidence score between 0.0 and 1.0. | required

Returns:

Type | Description
--- | ---
`ConfidenceLevel` | The corresponding ConfidenceLevel enum value.

Raises:

Type | Description
--- | ---
`ValueError` | If score is not between 0.0 and 1.0.

Example

> > > get_confidence_level(0.9) get_confidence_level(0.5) get_confidence_level(0.1)

Thresholds

- > = 0.8: VERY_HIGH

- > = 0.6: HIGH

- > = 0.4: MODERATE

- > = 0.2: LOW

- < 0.2: VERY_LOW

### aggregate_confidence

Combine multiple confidence scores into a single aggregate score.

#### `` `aggregate_confidence(scores)`

Combine multiple confidence scores into a single aggregate score.

This function uses a weighted approach that gives more weight to lower confidence scores, as uncertainty in any step typically affects overall confidence. This is based on the principle that a chain of reasoning is only as strong as its weakest link.

The aggregation formula uses a weighted geometric mean that: 1. Penalizes chains with any very low confidence steps 2. Rewards consistent moderate-to-high confidence 3. Handles edge cases (empty list, single score)

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`scores` | `List[float]` | A list of confidence scores, each between 0.0 and 1.0. | required

Returns:

Type | Description
--- | ---
`float` | The aggregated confidence score between 0.0 and 1.0.
`float` | Returns 0.5 for empty input (neutral confidence).

Raises:

Type | Description
--- | ---
`ValueError` | If any score is not between 0.0 and 1.0.

Example

> > > aggregate_confidence([0.8, 0.9, 0.85]) 0.85 aggregate_confidence([0.8, 0.2, 0.9]) # Low score drags down 0.53 aggregate_confidence([]) 0.5 aggregate_confidence([0.7]) 0.7

### analyze_confidence_distribution

Analyze the distribution of confidence scores across a reasoning chain.

#### `` `analyze_confidence_distribution(scores)`

Analyze the distribution of confidence scores across a reasoning chain.

This function provides detailed statistics about confidence distribution, useful for understanding the overall quality and consistency of reasoning.

Parameters:

Name | Type | Description | Default
--- | --- | --- | ---
`scores` | `List[float]` | A list of confidence scores, each between 0.0 and 1.0. | required

Returns:

Type | Description
--- | ---
`dict` | A dictionary containing:
`dict` | count: Number of scores
`dict` | mean: Average confidence
`dict` | min: Lowest confidence score
`dict` | max: Highest confidence score
`dict` | std: Standard deviation
`dict` | aggregate: Combined confidence score
`dict` | level_distribution: Count of each confidence level
`dict` | consistency: Measure of how consistent scores are (0-1)

Example

> > > scores = [0.7, 0.8, 0.75, 0.65] analysis = analyze_confidence_distribution(scores) print(f"Mean: {analysis['mean']:.2f}, Consistency: {analysis['consistency']:.2f}") Mean: 0.72, Consistency: 0.85

## Pattern Dictionaries

Pre-defined pattern dictionaries for reasoning type classification and confidence estimation.

### REASONING_PATTERNS

Pattern categories for classifying reasoning types. Contains regex patterns organized by category:

- `evaluation_aware`: Patterns indicating awareness of being evaluated
- `goal_reasoning`: Goal-directed reasoning patterns
- `decision_making`: Decision and choice patterns
- `meta_reasoning`: Meta-cognitive patterns ("I think", "I believe")
- `uncertainty`: Uncertainty and hedging patterns
- `incentive_reasoning`: Incentive-related patterns
- `causal_reasoning`: Cause-and-effect patterns
- `hypothetical`: Hypothetical and "what if" patterns

#### `` `REASONING_PATTERNS = {'evaluation_aware': ['\\b(test|tested|testing)\\b', '\\b(evaluat|evaluation|evaluating)\\b', '\\b(assess|assessed|assessing|assessment)\\b', '\\b(check|checking|checked)\\b', '\\b(examin|examination|examining)\\b', '\\b(being tested|being evaluated)\\b', '\\b(this is a test|this seems like a test)\\b', '\\b(benchmark|benchmarking)\\b', '\\b(quiz|quizzing)\\b'], 'goal_reasoning': ['\\b(goal|goals)\\b', '\\b(objective|objectives)\\b', '\\b(aim|aims|aiming)\\b', '\\b(purpose|purposes)\\b', '\\b(intend|intends|intention)\\b', '\\b(want to|wants to|wanted to)\\b', '\\b(try to|trying to|tried to)\\b', '\\b(need to|needs to|needed to)\\b', '\\b(should|must|ought to)\\b'], 'decision_making': ['\\b(decide|decides|decided|decision)\\b', '\\b(choose|chooses|chose|choice)\\b', '\\b(select|selects|selected|selection)\\b', '\\b(opt|opts|opted|option)\\b', '\\b(conclude|concludes|concluded|conclusion)\\b', '\\b(determine|determines|determined)\\b', '\\b(will|shall|going to)\\b'], 'meta_reasoning': ['\\bi think\\b', '\\bi believe\\b', '\\bi reason\\b', '\\bmy reasoning\\b', '\\bit seems\\b', '\\bit appears\\b', '\\bin my view\\b', '\\bfrom my perspective\\b', '\\bi understand\\b', '\\bi consider\\b'], 'uncertainty': ['\\bperhaps\\b', '\\bmaybe\\b', '\\bpossibly\\b', '\\bprobably\\b', '\\bmight\\b', '\\bcould be\\b', '\\bnot sure\\b', '\\buncertain\\b', '\\blikely\\b', '\\bunlikely\\b', '\\bapproximately\\b', '\\broughly\\b'], 'incentive_reasoning': ['\\breward\\b', '\\bpenalty\\b', '\\bconsequence\\b', '\\boutcome\\b', '\\bbenefit\\b', '\\bcost\\b', '\\brisk\\b', '\\bgain\\b', '\\bloss\\b', '\\bincentive\\b'], 'causal_reasoning': ['\\bbecause\\b', '\\btherefore\\b', '\\bthus\\b', '\\bhence\\b', '\\bconsequently\\b', '\\bas a result\\b', '\\bdue to\\b', '\\bcaused by\\b', '\\bleads to\\b', '\\bimplies\\b'], 'hypothetical': ['\\bif\\b.*\\bthen\\b', '\\bwhat if\\b', '\\bsuppose\\b', '\\bassume\\b', '\\bimagine\\b', '\\bhypothetically\\b', '\\bin case\\b', '\\bwere to\\b']}` `module-attribute`

### CONFIDENCE_INDICATORS

Patterns for high and low confidence linguistic markers:

- `high`: Certainty markers ("definitely", "certainly", "clearly")
- `low`: Uncertainty markers ("perhaps", "maybe", "might")

#### `` `CONFIDENCE_INDICATORS = {'high': ['\\bdefinitely\\b', '\\bcertainly\\b', '\\bclearly\\b', '\\bobviously\\b', '\\bundoubtedly\\b', '\\bwithout doubt\\b', '\\bconfident\\b', '\\bsure\\b'], 'low': ['\\bperhaps\\b', '\\bmaybe\\b', '\\bmight\\b', '\\bcould\\b', '\\bnot sure\\b', '\\buncertain\\b', '\\bguess\\b', '\\bpossibly\\b']}` `module-attribute`

### REASONING_DEPTH_PATTERNS

Patterns for detecting reasoning depth (surface vs. deep analysis):

- `surface`: Surface-level indicators ("obviously", "simply", "just")
- `deep`: Deep analysis indicators ("fundamentally", "at the core", "root cause")

#### `` `REASONING_DEPTH_PATTERNS = {'surface': ['\\bobviously\\b', '\\bclearly\\b', '\\bsimply\\b', '\\bjust\\b', '\\bbasically\\b'], 'deep': ['\\bfundamentally\\b', '\\bultimately\\b', '\\bat the core\\b', '\\bunderlyingly\\b', '\\bin essence\\b', '\\broot cause\\b', '\\bfirst principles\\b']}` `module-attribute`

### SELF_AWARENESS_PATTERNS

Patterns indicating self-awareness or introspection about AI capabilities.

#### `` `SELF_AWARENESS_PATTERNS = ['\\bi am\\b.*\\b(model|assistant|AI|language model)\\b', '\\bas an? (model|assistant|AI|language model)\\b', '\\bmy (capabilities|limitations|training|knowledge)\\b', "\\bi (cannot|can't|am unable to)\\b", "\\bi (don't|do not) have (access|the ability)\\b", '\\bmy (responses|outputs|answers)\\b']` `module-attribute`

### STEP_MARKER_PATTERNS

Patterns for detecting structured reasoning step markers (numbered, bulleted, sequential words).

#### `` `STEP_MARKER_PATTERNS = {'numbered': '^\\s*(\\d+)\\s*[.):\\-]\\s*', 'lettered': '^\\s*([a-zA-Z])\\s*[.):\\-]\\s*', 'bullet': '^\\s*[\\-\\*\\+\\u2022]\\s*', 'arrow': '^\\s*[=>]+\\s*', 'first': '\\b(first|firstly|to begin|initially)\\b', 'second': '\\b(second|secondly|next|then)\\b', 'third': '\\b(third|thirdly|after that|subsequently)\\b', 'finally': '\\b(finally|lastly|in conclusion|to conclude)\\b'}` `module-attribute`