\documentclass[11pt,a4paper]{report}

% === PACKAGES ===
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{mathpazo}
\usepackage{amsmath,amssymb,amsthm}
\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{etoolbox}
\usepackage[backend=biber,style=numeric,sorting=none,hyperref=true]{biblatex}
\addbibresource{../../shared/bibliography/references.bib}
\usepackage{hyperref}
\usepackage{cleveref}
\usepackage{booktabs}
\usepackage{longtable}
\usepackage{listings}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{tikz}
\usetikzlibrary{automata,positioning,arrows.meta,shapes.geometric}

% === AUTO-GENERATED DATA ===
% Regenerate with: cd experiments/benchmarks && make report
\input{generated/benchmark-data}
% Note: noise-data.tex not included in preamble due to catcode conflicts with tikz
% The noise robustness section uses the table directly instead of macros
\usepackage[margin=1in]{geometry}
\usepackage{fancyhdr}

% === COLORS ===
\definecolor{emicblue}{RGB}{41,98,255}
\definecolor{emicgreen}{RGB}{0,150,136}
\definecolor{codebg}{RGB}{248,248,248}
\definecolor{linkblue}{RGB}{0,51,153}

% === HYPERREF SETUP ===
\hypersetup{
    colorlinks=true,
    linkcolor=linkblue,
    citecolor=emicgreen,
    urlcolor=emicblue
}

% === LISTINGS SETUP ===
\lstset{
    language=Python,
    basicstyle=\small\ttfamily,
    backgroundcolor=\color{codebg},
    frame=single,
    framerule=0.5pt,
    rulecolor=\color{gray!50},
    xleftmargin=1em,
    xrightmargin=1em,
    breaklines=true,
    showstringspaces=false,
    keywordstyle=\color{emicblue}\bfseries,
    commentstyle=\color{gray}\itshape,
    stringstyle=\color{emicgreen},
    numbers=left,
    numberstyle=\tiny\color{gray},
    numbersep=8pt
}

% Prevent listings from breaking across pages
\BeforeBeginEnvironment{lstlisting}{\begin{minipage}{\linewidth}}
\AfterEndEnvironment{lstlisting}{\end{minipage}}

% === HEADER/FOOTER ===
\pagestyle{fancy}
\fancyhf{}
\fancyhead[L]{\leftmark}
\fancyhead[R]{\thepage}
\renewcommand{\headrulewidth}{0.4pt}

% === THEOREM ENVIRONMENTS ===
\theoremstyle{definition}
\newtheorem{definition}{Definition}[chapter]
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[chapter]

% === NOTATION ===
\newcommand{\eps}{\varepsilon}
\newcommand{\Cmu}{C_\mu}
\newcommand{\hmu}{h_\mu}
\newcommand{\calS}{\mathcal{S}}
\newcommand{\calA}{\mathcal{A}}
\newcommand{\emic}{\texttt{emic}}

% === TITLE ===
\title{%
    \vspace{-2em}
    {\LARGE\bfseries emic}\\[0.5em]
    {\Large A Python Framework for $\eps$-Machine Inference}\\[1em]
    {\normalsize Technical Report}
}
\author{%
    John S Azariah\\
    \small University of Technology, Sydney (UTS)\\
    \small\texttt{john.azariah@student.uts.edu.au}
}
\date{January 2026}

\begin{document}

\maketitle

\begin{abstract}
Computational Mechanics provides a rigorous framework for understanding structure in stochastic processes through the construction of $\eps$-machines—minimal, optimal predictive models. Despite decades of theoretical development, accessible software implementations have been lacking. We present \emic{}, a production-quality Python library for $\eps$-machine inference from symbolic time series data. The library implements multiple inference algorithms (CSSR, Spectral Learning, Causal State Merging, Bayesian Structural Inference, Next Symbol Distribution) with varying trade-offs between speed and accuracy. Comprehensive benchmarking shows that Spectral Learning achieves \spectralLargeNCorrectness\% correctness at sample sizes $N \geq 1{,}000$, with CSSR at \cssrLargeNCorrectness\% and NSD at \nsdLargeNCorrectness\%. The library provides extensive analysis tools and emphasizes correctness through golden tests against known analytical results. This report describes the design principles, architecture, algorithms, and validation methodology, serving as both documentation and a thesis chapter on the software contribution.
\end{abstract}

\tableofcontents
\listoffigures
\listoftables

% ===================================================================
\chapter{Introduction}
\label{ch:introduction}
% ===================================================================

\section{Motivation}

Computational Mechanics, developed by Crutchfield, Shalizi, and colleagues over the past three decades, provides the theoretical foundation for understanding structure and complexity in stochastic processes. The central object is the \emph{$\eps$-machine}—the unique, minimal hidden Markov model that captures a process's ``causal architecture.''

Despite the elegance and power of this framework, practical adoption has been hindered by the lack of modern, well-documented software tools. The original CSSR implementation in Perl \cite{shalizi2004algorithm} is unmaintained. More recent tools exist but lack comprehensive documentation, testing, or support for multiple algorithms.

\section{Contribution}

\emic{} addresses this gap with:

\begin{itemize}
    \item \textbf{Modern Python implementation}: Type-annotated, immutable data structures, comprehensive documentation
    \item \textbf{Multiple algorithms}: CSSR, Spectral Learning, Causal State Merging, Bayesian Structural Inference, Naive State Discovery
    \item \textbf{Rigorous validation}: 326+ tests including golden tests against known analytical results
    \item \textbf{Composable pipelines}: Functional composition with \texttt{>>} operator
    \item \textbf{Extensible architecture}: Protocol-based design for custom algorithms
\end{itemize}

The library is available at \url{https://github.com/johnazariah/emic} with comprehensive online documentation at \url{https://johnazariah.github.io/emic/}, including algorithm deep dives, practical guides for working with real data, and API reference.

\section{Outline}

\begin{itemize}
    \item Chapter~\ref{ch:background}: Computational Mechanics background
    \item Chapter~\ref{ch:architecture}: Software architecture
    \item Chapter~\ref{ch:algorithms}: Implemented algorithms
    \item Chapter~\ref{ch:validation}: Validation methodology
    \item Chapter~\ref{ch:usage}: Usage patterns
    \item Chapter~\ref{ch:performance}: Performance evaluation
    \item Chapter~\ref{ch:conclusion}: Limitations and future work
\end{itemize}

% ===================================================================
\chapter{Background}
\label{ch:background}
% ===================================================================

\section{Computational Mechanics in Brief}

Computational Mechanics~\cite{crutchfield1989inferring,shalizi2001computational} provides a rigorous framework for discovering hidden structure in stochastic processes. The central insight is that the ``structure'' of a process can be captured by grouping histories into equivalence classes based on their predictive power.

\subsection{Causal States}

Two histories $\overleftarrow{s}$ and $\overleftarrow{s}'$ are \emph{causally equivalent} if they make identical predictions about all futures:
\begin{equation}
    \overleftarrow{s} \sim_\varepsilon \overleftarrow{s}' \iff P(\overrightarrow{S} | \overleftarrow{s}) = P(\overrightarrow{S} | \overleftarrow{s}')
\end{equation}

The equivalence classes under $\sim_\varepsilon$ are called \emph{causal states}, and the function $\varepsilon: \{\text{histories}\} \to \calS$ maps each history to its causal state.

\subsection{The $\eps$-Machine}

The $\eps$-machine is the hidden Markov model with:
\begin{itemize}
    \item \textbf{States}: The causal states $\calS$
    \item \textbf{Transitions}: $T^{(x)}_{\sigma\sigma'} = P(\text{emit } x, \text{go to } \sigma' | \text{in } \sigma)$
    \item \textbf{Stationary distribution}: $\pi(\sigma) = P(\calS = \sigma)$
\end{itemize}

Key properties:
\begin{itemize}
    \item \textbf{Unifilarity}: Each (state, symbol) pair determines at most one next state
    \item \textbf{Minimality}: The $\eps$-machine has the fewest states among optimal predictors
    \item \textbf{Uniqueness}: The $\eps$-machine is unique up to isomorphism
\end{itemize}

\subsection{Complexity Measures}

The $\eps$-machine enables computation of fundamental measures:

\begin{itemize}
    \item \textbf{Statistical Complexity} ($\Cmu$): Entropy of the causal state distribution
    \begin{equation}
        \Cmu = H[\calS] = -\sum_{\sigma} \pi(\sigma) \log_2 \pi(\sigma)
    \end{equation}
    Interpretation: The memory (in bits) required for optimal prediction.

    \item \textbf{Entropy Rate} ($\hmu$): Average surprise per symbol
    \begin{equation}
        \hmu = H[X | \calS] = -\sum_{\sigma} \pi(\sigma) \sum_x P(x|\sigma) \log_2 P(x|\sigma)
    \end{equation}
    Interpretation: The irreducible randomness per symbol.
\end{itemize}

For a comprehensive treatment, see our companion tutorial or the original papers~\cite{shalizi2001computational,crutchfield2012between}.

\section{Existing Tools}

Several tools for $\eps$-machine inference exist, but each has limitations:

\begin{table}[h]
\centering
\caption{Comparison of $\eps$-machine inference tools}
\label{tab:tools}
\begin{tabular}{lcccc}
\toprule
Tool & Language & Algorithms & Test Suite & Maintained \\
\midrule
CSSR (original) & Perl & CSSR only & None & No (2006) \\
CMPy & Python 2 & CSSR only & Limited & No (2014) \\
transCSSR & Julia & CSSR variant & Basic & Partial \\
\emic{} & Python 3.11+ & 5 algorithms & 326 tests & Yes \\
\bottomrule
\end{tabular}
\end{table}

\section{Why a New Implementation?}

The \emic{} library was developed to address several gaps:

\begin{enumerate}
    \item \textbf{Modern Python}: Type annotations, immutable data structures, async support
    \item \textbf{Multiple algorithms}: CSSR, Spectral, CSM, BSI, NSD---enabling comparison
    \item \textbf{Rigorous validation}: Golden tests against known analytical results
    \item \textbf{Composable design}: Pipeline operators for clean data flow
    \item \textbf{Extensibility}: Protocol-based architecture for custom algorithms
    \item \textbf{Documentation}: Comprehensive docstrings and examples
\end{enumerate}

% ===================================================================
\chapter{Architecture}
\label{ch:architecture}
% ===================================================================

\section{Design Principles}

\emic{} follows five core design principles that prioritize correctness and usability:

\begin{enumerate}
    \item \textbf{Immutability}: All core types are frozen dataclasses. Once created, an \texttt{EpsilonMachine} cannot be modified. This eliminates a class of bugs from accidental mutation and makes objects safe to share across threads.

    \item \textbf{Type Safety}: Full type annotations throughout, validated with Pyright in strict mode. Types serve as documentation and enable IDE autocompletion.

    \item \textbf{Protocol-Based Extension}: Abstract interfaces define contracts for algorithms and sources. Users can implement custom inference algorithms without modifying the library.

    \item \textbf{Composability}: Pipeline operators (\texttt{>>}) enable declarative data flow. Operations compose cleanly without intermediate variables.

    \item \textbf{Correctness First}: Extensive testing (\testCount{} tests, \testCoverage\%+ coverage), including ``golden tests'' that verify inference on processes with known analytical solutions.
\end{enumerate}

\section{Module Structure}

The library is organized into five main modules:

\begin{table}[h]
\centering
\caption{Module organization}
\label{tab:modules}
\begin{tabular}{ll}
\toprule
Module & Purpose \\
\midrule
\texttt{emic.types} & Core data types: \texttt{Symbol}, \texttt{StateId}, \texttt{EpsilonMachine} \\
\texttt{emic.sources} & Data sources: synthetic processes, file readers \\
\texttt{emic.inference} & Inference algorithms: CSSR, Spectral, BSI, CSM, NSD \\
\texttt{emic.analysis} & Complexity measures: $\Cmu$, $\hmu$, excess entropy \\
\texttt{emic.output} & Visualization, serialization, LaTeX export \\
\bottomrule
\end{tabular}
\end{table}

\section{Core Types}

\subsection{Symbol and Alphabet}

Symbols are the atomic units of observation---typically single characters, but the library supports any hashable type:

\begin{lstlisting}[caption={Symbol and Alphabet types}]
from emic.types import Symbol, Alphabet, ConcreteAlphabet

# Symbols are just strings (or any hashable type)
Symbol = str  # Type alias

# Alphabet is an abstract protocol
class Alphabet(Protocol):
    def __contains__(self, s: Symbol) -> bool: ...
    def __iter__(self) -> Iterator[Symbol]: ...
    def __len__(self) -> int: ...

# Concrete implementation
@dataclass(frozen=True)
class ConcreteAlphabet:
    symbols: frozenset[Symbol]
\end{lstlisting}

\subsection{StateId and Transition}

States are identified by string IDs, and transitions encode the probability of emitting a symbol while moving to a target state:

\begin{lstlisting}[caption={State and transition types}]
StateId = str  # State identifier

@dataclass(frozen=True)
class Transition:
    """A labeled transition in an epsilon-machine."""
    symbol: Symbol           # Emitted symbol
    target: StateId          # Target state
    probability: float       # Transition probability

@dataclass(frozen=True)
class CausalState:
    """A causal state with its outgoing transitions."""
    id: StateId
    transitions: frozenset[Transition]

    @property
    def alphabet(self) -> frozenset[Symbol]:
        """Symbols emittable from this state."""
        return frozenset(t.symbol for t in self.transitions)
\end{lstlisting}

\subsection{EpsilonMachine}

The central data structure representing an inferred $\eps$-machine:

\begin{lstlisting}[caption={EpsilonMachine class (simplified)}]
@dataclass(frozen=True)
class EpsilonMachine(Generic[A]):
    """
    An epsilon-machine over alphabet A.

    The minimal, optimal predictor of a stationary process.
    """
    alphabet: frozenset[A]
    states: frozenset[CausalState[A]]
    start_state: StateId
    stationary_distribution: Distribution[StateId]

    def __len__(self) -> int:
        """Number of causal states."""
        return len(self.states)

    def is_unifilar(self) -> bool:
        """Check if machine is unifilar."""
        for state in self.states:
            symbols_seen: set[A] = set()
            for t in state.transitions:
                if t.symbol in symbols_seen:
                    return False
                symbols_seen.add(t.symbol)
        return True
\end{lstlisting}

\subsection{EpsilonMachineBuilder}

Since \texttt{EpsilonMachine} is immutable, we use a builder pattern for construction:

\begin{lstlisting}[caption={Builder pattern for machine construction}]
from emic.types import EpsilonMachineBuilder

machine = (
    EpsilonMachineBuilder[str]()
    .add_transition("A", "0", "A", 0.5)
    .add_transition("A", "1", "B", 0.5)
    .add_transition("B", "1", "A", 1.0)
    .with_start_state("A")
    .build()
)

print(f"States: {len(machine)}")      # 2
print(f"Unifilar: {machine.is_unifilar()}")  # True
\end{lstlisting}

\section{Pipeline Composition}

\emic{} provides a functional composition operator (\texttt{>>}) for building data pipelines:

\begin{lstlisting}[caption={Pipeline composition example}]
from emic.sources.synthetic import GoldenMeanSource
from emic.sources.transforms import TakeN
from emic.inference import CSSR, CSSRConfig
from emic.analysis import Summary

# Build pipeline: source >> transform >> inference >> analysis
pipeline = (
    GoldenMeanSource(p=0.5, _seed=42)
    >> TakeN(100_000)
    >> CSSR(CSSRConfig(max_history=5))
    >> Summary()
)

# Execute
result = pipeline.run()
print(result)
# StatisticalComplexity: 0.918 bits
# EntropyRate: 0.667 bits
\end{lstlisting}

The \texttt{>>} operator chains pipeline stages:

\begin{lstlisting}[caption={Pipeline operator implementation}]
@dataclass
class Pipeline:
    """A composable pipeline stage."""
    stages: list[Callable]

    def __rshift__(self, other: Callable) -> Pipeline:
        """Compose with another stage."""
        return Pipeline(self.stages + [other])

    def run(self, input=None):
        """Execute the pipeline."""
        result = input
        for stage in self.stages:
            result = stage(result)
        return result
\end{lstlisting}

\section{Protocol-Based Extension}

Users can implement custom algorithms by following the \texttt{InferenceAlgorithm} protocol:

\begin{lstlisting}[caption={InferenceAlgorithm protocol}]
from typing import Protocol, Generic, TypeVar
from collections.abc import Iterable

A = TypeVar("A", bound=Hashable)

class InferenceAlgorithm(Protocol[A]):
    """Protocol for epsilon-machine inference algorithms."""

    def infer(
        self,
        sequence: Iterable[A],
        alphabet: frozenset[A] | None = None,
    ) -> InferenceResult[A]:
        """Infer an epsilon-machine from the sequence."""
        ...
\end{lstlisting}

Custom algorithms implementing this protocol integrate seamlessly with the pipeline system.

% ===================================================================
\chapter{Inference Algorithms}
\label{ch:algorithms}
% ===================================================================

\emic{} implements five inference algorithms, each with different trade-offs between sample efficiency, computational cost, and robustness to noise.

\section{CSSR: Causal State Splitting Reconstruction}

CSSR~\cite{shalizi2004algorithm} is the ``standard'' algorithm for $\eps$-machine inference. It reconstructs causal states by iteratively splitting and merging histories based on their next-symbol distributions.

\subsection{Algorithm Overview}

\begin{algorithm}
\caption{CSSR Algorithm}
\begin{algorithmic}[1]
\Require Sequence $x_1, x_2, \ldots, x_n$, max history length $L$, significance level $\alpha$
\Ensure Epsilon-machine $\mathcal{M}$

\State Build suffix tree from sequence
\State Initialize: all histories in one state
\For{$\ell = 1$ to $L$}
    \For{each state $\sigma$}
        \For{each pair of histories $h_1, h_2 \in \sigma$}
            \If{$\chi^2(P(X|h_1), P(X|h_2)) > \chi^2_\alpha$}
                \State Split $\sigma$ into substates
            \EndIf
        \EndFor
    \EndFor
    \State Attempt to merge indistinguishable states
\EndFor
\State Construct transition matrix from final partition
\State \Return $\mathcal{M}$
\end{algorithmic}
\end{algorithm}

\subsection{Configuration}

\begin{lstlisting}[caption={CSSR configuration}]
from emic.inference import CSSR, CSSRConfig

config = CSSRConfig(
    max_history=5,       # Maximum history length L
    significance=0.01,   # Chi-squared significance level
    min_count=5,         # Minimum observations per history
    max_iterations=100,  # Convergence limit
)

cssr = CSSR(config)
result = cssr.infer(sequence)
\end{lstlisting}

\subsection{Strengths and Weaknesses}

\begin{itemize}
    \item[\checkmark] Provably consistent (converges to true model asymptotically)
    \item[\checkmark] Guarantees unifilarity
    \item[\checkmark] Interpretable intermediate states
    \item[$\times$] Can over-split with limited data
    \item[$\times$] Sensitive to significance threshold choice
\end{itemize}

\section{Spectral Learning}

Spectral Learning~\cite{hsu2012spectral,boots2011closing} uses SVD of Hankel matrices to learn observable operator representations. This approach recovers the model structure in polynomial time with statistical consistency.

\subsection{Hankel Matrix Construction}

The Hankel matrix $H$ organizes sequence probabilities:
\begin{equation}
    H_{ij} = P(\text{prefix } p_i, \text{suffix } s_j)
\end{equation}

The key insight: $\text{rank}(H) = k$, where $k$ is the number of hidden states.

\subsection{Algorithm Overview}

\begin{algorithm}
\caption{Spectral Learning}
\begin{algorithmic}[1]
\Require Sequence $x_1, \ldots, x_n$, history length $L$
\Ensure Epsilon-machine $\mathcal{M}$

\State Estimate Hankel matrix $\hat{H}$ from data
\State Estimate symbol-conditioned matrices $\hat{H}_x$ for each $x \in \mathcal{A}$
\State Compute SVD: $\hat{H} = U \Sigma V^\top$
\State Determine rank $k$ from singular value spectrum
\State Extract operators: $A_x = U_k^\top \hat{H}_x V_k \Sigma_k^{-1}$
\State Construct machine from operators
\State \Return $\mathcal{M}$
\end{algorithmic}
\end{algorithm}

\subsection{Configuration}

\begin{lstlisting}[caption={Spectral configuration}]
from emic.inference.spectral import Spectral, SpectralConfig

config = SpectralConfig(
    max_history=10,        # History/future length
    rank_threshold=0.01,   # Singular value cutoff
    min_count=5,           # Minimum count per substring
)

spectral = Spectral(config)
result = spectral.infer(sequence)
\end{lstlisting}

\subsection{Strengths and Weaknesses}

\begin{itemize}
    \item[\checkmark] Polynomial time complexity
    \item[\checkmark] Statistically consistent
    \item[\checkmark] Robust to noise (SVD averaging)
    \item[\checkmark] Determines state count from rank
    \item[$\times$] Learned operators may not be unifilar
    \item[$\times$] Requires post-processing for $\eps$-machine form
\end{itemize}

\section{Causal State Merging (CSM)}

CSM takes a bottom-up approach: start with many states (one per history), then iteratively merge similar states.

\subsection{Algorithm Overview}

\begin{algorithm}
\caption{Causal State Merging}
\begin{algorithmic}[1]
\Require Sequence, history length $L$, distance threshold $\theta$
\Ensure Epsilon-machine $\mathcal{M}$

\State Create initial state for each unique $L$-history
\State Compute predictive distribution for each state
\While{can merge}
    \State Find pair $(\sigma_1, \sigma_2)$ with minimum distribution distance
    \If{distance $< \theta$}
        \State Merge $\sigma_1$ and $\sigma_2$
        \State Update distributions
    \Else
        \State \textbf{break}
    \EndIf
\EndWhile
\State \Return $\mathcal{M}$
\end{algorithmic}
\end{algorithm}

\subsection{Configuration}

\begin{lstlisting}[caption={CSM configuration}]
from emic.inference.csm import CSM, CSMConfig

config = CSMConfig(
    history_length=5,     # History length
    merge_threshold=0.1,  # Maximum distance to merge
    distance_metric="kl", # KL divergence or Jensen-Shannon
)

csm = CSM(config)
result = csm.infer(sequence)
\end{lstlisting}

\subsection{Strengths and Weaknesses}

\begin{itemize}
    \item[\checkmark] Simple and interpretable
    \item[\checkmark] Produces hierarchical structure
    \item[\checkmark] Robust to over-splitting
    \item[$\times$] May under-split (merge distinct states)
    \item[$\times$] Not provably consistent
\end{itemize}

\section{Bayesian Structural Inference (BSI)}

BSI~\cite{strelioff2014bayesian} uses Bayesian inference with Gibbs sampling to learn $\eps$-machines while quantifying uncertainty.

\subsection{Probabilistic Model}

BSI places priors on the model structure:
\begin{itemize}
    \item Transition probabilities: $\theta_\sigma \sim \text{Dirichlet}(\alpha)$
    \item State assignments: Categorical with Dirichlet prior
\end{itemize}

The Dirichlet-Multinomial conjugacy enables efficient Gibbs sampling.

\subsection{Algorithm Overview}

\begin{algorithm}
\caption{Bayesian Structural Inference}
\begin{algorithmic}[1]
\Require Sequence, max states $K$, prior $\alpha$, samples $N$
\Ensure Epsilon-machine $\mathcal{M}$

\State Initialize state assignments randomly
\For{$i = 1$ to $N$ (Gibbs iterations)}
    \For{each history $h$}
        \State Compute $P(\text{state}_h = k | \text{rest})$ for each $k$
        \State Sample new assignment from posterior
    \EndFor
\EndFor
\State Return MAP estimate from samples
\end{algorithmic}
\end{algorithm}

\subsection{Configuration}

\begin{lstlisting}[caption={BSI configuration}]
from emic.inference.bsi import BSI, BSIConfig

config = BSIConfig(
    max_states=10,     # Maximum states to consider
    alpha_prior=1.0,   # Dirichlet concentration
    n_samples=1000,    # Gibbs samples
    burnin=200,        # Burn-in period
    seed=42,           # Random seed
)

bsi = BSI(config)
result = bsi.infer(sequence)
\end{lstlisting}

\subsection{Strengths and Weaknesses}

\begin{itemize}
    \item[\checkmark] Uncertainty quantification
    \item[\checkmark] Automatic model selection via marginal likelihood
    \item[\checkmark] Robust with limited data
    \item[\checkmark] Regularization via priors
    \item[$\times$] Computationally expensive
    \item[$\times$] Requires tuning (samples, burnin)
\end{itemize}

\section{Naive State Discovery (NSD)}

NSD uses representation learning to discover states. It embeds histories into a vector space and clusters to find causal states.

\subsection{Algorithm Overview}

\begin{algorithm}
\caption{Naive State Discovery}
\begin{algorithmic}[1]
\Require Sequence, history length $L$, max states $K$
\Ensure Epsilon-machine $\mathcal{M}$

\State Compute next-symbol distributions for each $L$-history
\State Embed distributions as vectors
\State Apply k-means clustering with $k \in \{1, \ldots, K\}$
\State Select $k$ by silhouette score or BIC
\State Assign states based on cluster membership
\State \Return $\mathcal{M}$
\end{algorithmic}
\end{algorithm}

\subsection{Strengths and Weaknesses}

\begin{itemize}
    \item[\checkmark] Fast and scalable
    \item[\checkmark] Works with high-dimensional data
    \item[$\times$] Sensitive to clustering hyperparameters
    \item[$\times$] No theoretical guarantees
\end{itemize}

\section{Algorithm Comparison}

\begin{table}[h]
\centering
\caption{Algorithm comparison}
\label{tab:algorithms}
\begin{tabular}{lccccc}
\toprule
Algorithm & Complexity & Consistent & Noise Robust & Unifilarity \\
\midrule
CSSR & $O(n \cdot L \cdot |\mathcal{A}|^L)$ & Yes & Moderate & Guaranteed \\
Spectral & $O(n \cdot L^2 + L^6)$ & Yes & Good & No \\
CSM & $O(n \cdot L + k^2 \cdot |\mathcal{A}|)$ & No & Good & Guaranteed \\
BSI & $O(N \cdot n \cdot k^2)$ & Yes & Excellent & No \\
NSD & $O(n \cdot L + k^3)$ & No & Moderate & No \\
\bottomrule
\end{tabular}
\end{table}

\textbf{Recommendations:}
\begin{itemize}
    \item \textbf{Default choice}: CSSR with appropriate significance level
    \item \textbf{Limited data}: BSI (regularization helps) or Spectral
    \item \textbf{Speed priority}: NSD or CSM
    \item \textbf{Uncertainty needed}: BSI
\end{itemize}

% ===================================================================
\chapter{Validation}
\label{ch:validation}
% ===================================================================

Correctness is paramount for scientific software. \emic{} employs a multi-layered testing strategy: unit tests for components, golden tests against known analytical solutions, and property-based tests for invariants.

\section{Testing Methodology}

\subsection{Test Suite Overview}

\begin{table}[h]
\centering
\caption{Test suite statistics}
\label{tab:test-stats}
\begin{tabular}{lc}
\toprule
Metric & Value \\
\midrule
Total tests & 334 \\
Unit tests & 253 \\
Golden tests & 48 \\
Integration tests & 18 \\
Property tests & 15 \\
Coverage & $>\testCoverage\%$ \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Unit Tests}

Unit tests verify individual components in isolation:

\begin{itemize}
    \item \textbf{Type tests}: \texttt{EpsilonMachine}, \texttt{CausalState}, \texttt{Transition} construction and validation
    \item \textbf{Source tests}: Each synthetic source produces correct output
    \item \textbf{Analysis tests}: Measure computations match manual calculations
    \item \textbf{Algorithm internals}: Suffix trees, partitions, Hankel matrices
\end{itemize}

\subsection{Golden Tests}

Golden tests are the core of validation: they verify inference on processes with \emph{known analytical solutions}.

\begin{lstlisting}[caption={Golden test structure}]
@pytest.mark.parametrize("algorithm", [CSSR, Spectral, BSI, CSM, NSD])
def test_golden_mean_inference(algorithm):
    """Verify inference on Golden Mean process."""
    # Known analytical values
    TRUE_STATES = 2
    TRUE_CMU = 0.918  # bits
    TRUE_HMU = 0.667  # bits

    # Generate data
    source = GoldenMeanSource(p=0.5, _seed=42)
    sequence = list(TakeN(100_000)(source))

    # Infer
    result = algorithm(default_config()).infer(sequence)

    # Verify
    assert len(result.machine.states) == TRUE_STATES
    assert abs(statistical_complexity(result.machine) - TRUE_CMU) < 0.05
    assert abs(entropy_rate(result.machine) - TRUE_HMU) < 0.05
\end{lstlisting}

\subsection{Property-Based Tests}

Using Hypothesis for property-based testing, we verify invariants that must hold for any valid input:

\begin{lstlisting}[caption={Property test example}]
from hypothesis import given, strategies as st

@given(st.lists(st.sampled_from(['0', '1']), min_size=1000))
def test_entropy_rate_bounds(sequence):
    """Entropy rate is bounded by symbol entropy."""
    if len(set(sequence)) == 1:
        return  # Skip constant sequences

    result = CSSR(CSSRConfig()).infer(sequence)
    h_mu = entropy_rate(result.machine)

    # Must be non-negative
    assert h_mu >= 0

    # Must be bounded by log(alphabet_size)
    assert h_mu <= math.log2(len(result.machine.alphabet))
\end{lstlisting}

\section{Canonical Test Processes}

\emic{} includes several synthetic processes with known analytical properties:

\begin{table}[h]
\centering
\caption{Golden test processes and expected values}
\label{tab:golden}
\begin{tabular}{lcccl}
\toprule
Process & States & $\Cmu$ (bits) & $\hmu$ (bits) & Notes \\
\midrule
BiasedCoin(0.5) & 1 & 0.000 & 1.000 & IID uniform \\
BiasedCoin(0.3) & 1 & 0.000 & 0.881 & IID biased \\
GoldenMean(0.5) & 2 & 0.918 & 0.667 & No consecutive 0s \\
EvenProcess(0.5) & 2 & 0.918 & 0.667 & Even \# of 1s \\
Periodic(3) & 3 & 1.585 & 0.000 & Deterministic cycle \\
Periodic(5) & 5 & 2.322 & 0.000 & Deterministic cycle \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Golden Mean Process}

The Golden Mean forbids consecutive 0s. Its $\eps$-machine has two states:
\begin{itemize}
    \item State A (``last was 1''): emit 0 or 1 with probability $p$, $1-p$
    \item State B (``last was 0''): emit 1 with probability 1
\end{itemize}

Analytical values:
\begin{align}
    \pi(A) &= \frac{1}{2-p}, \quad \pi(B) = \frac{1-p}{2-p} \\
    \Cmu &= H[\pi] = -\pi(A)\log\pi(A) - \pi(B)\log\pi(B) \\
    \hmu &= \pi(A) \cdot H[p] = \frac{H[p]}{2-p}
\end{align}

For $p = 0.5$: $\Cmu \approx 0.918$ bits, $\hmu \approx 0.667$ bits.

\subsection{Even Process}

The Even Process emits 0 after an even number of 1s. Same state count and complexity as Golden Mean, but different structure.

\subsection{Periodic Processes}

Periodic(n) cycles through a fixed sequence of length $n$. Since the sequence is deterministic:
\begin{align}
    \Cmu &= \log_2 n \text{ bits} \\
    \hmu &= 0 \text{ bits}
\end{align}

\section{Validation Results}

\subsection{State Count Accuracy}

Comprehensive benchmarking (see Chapter~\ref{ch:performance}) reveals that algorithms differ significantly in their ability to recover the correct state count:

% Auto-generated from experiments/benchmarks/analyze_results.ipynb
\input{generated/tab-state-counts}

\textbf{Spectral} and \textbf{CSSR} achieve the highest correctness, though with complementary blind spots---CSSR fails on Even Process at large $N$ (over-splitting to 4 states), while Spectral handles it correctly. NSD struggles with Periodic processes, CSM with Even Process, and BSI tends to under-split complex processes.

\subsection{Implications for Measure Accuracy}

When the state count is wrong, derived measures ($\Cmu$, $\hmu$) are also wrong. This makes correctness validation essential before trusting complexity estimates:

\begin{itemize}
    \item A machine with too few states underestimates $\Cmu$ (lost information)
    \item A machine with too many states overestimates $\Cmu$ (spurious structure)
\end{itemize}

For applications requiring accurate complexity measures, we recommend:
\begin{itemize}
    \item \textbf{Spectral} for general use (\spectralOverallCorrectness\% overall accuracy, \spectralTenKCorrectness\% at $N \geq 10{,}000$)
    \item \textbf{CSSR} for processes without the Even Process structure
    \item Cross-validation against known reference processes when possible
\end{itemize}

\subsection{Convergence Analysis}

For algorithms that achieve correctness, estimates converge to analytical values as sample size increases. Spectral reliably achieves correct inference at $N \geq 1{,}000$ for most processes, while CSSR may require $\sim$10,000 samples for stable 2-state inference.

% ===================================================================
\chapter{Usage}
\label{ch:usage}
% ===================================================================

This chapter demonstrates common usage patterns for \emic{}.

\section{Installation}

\emic{} requires Python 3.11+ and is available on PyPI:

\begin{lstlisting}[language=bash]
pip install emic
\end{lstlisting}

For development installation:

\begin{lstlisting}[language=bash]
git clone https://github.com/johnazariah/emic
cd emic
pip install -e ".[dev]"
\end{lstlisting}

\section{Basic Usage}

\subsection{Inferring from Synthetic Data}

\begin{lstlisting}[caption={Basic inference example}]
from emic.sources.synthetic import GoldenMeanSource
from emic.sources.transforms import TakeN
from emic.inference import CSSR, CSSRConfig
from emic.analysis import statistical_complexity, entropy_rate

# Generate data from Golden Mean process
source = GoldenMeanSource(p=0.5, _seed=42)
sequence = list(TakeN(100_000)(source))

# Infer epsilon-machine
config = CSSRConfig(max_history=5, significance=0.01)
cssr = CSSR(config)
result = cssr.infer(sequence)

# Analyze
machine = result.machine
print(f"States: {len(machine)}")
print(f"Cmu: {statistical_complexity(machine):.3f} bits")
print(f"hmu: {entropy_rate(machine):.3f} bits")
\end{lstlisting}

Output:
\begin{lstlisting}
States: 2
Cmu: 0.918 bits
hmu: 0.667 bits
\end{lstlisting}

\subsection{Inferring from Your Data}

\begin{lstlisting}[caption={Inference from file data}]
from emic.sources.empirical import SequenceDataSource
from emic.inference import CSSR, CSSRConfig

# Load data from file (one symbol per line or continuous string)
source = SequenceDataSource.from_file("my_data.txt")
sequence = list(source)

# Infer
result = CSSR(CSSRConfig(max_history=8)).infer(sequence)
print(f"Found {len(result.machine)} states")
\end{lstlisting}

\subsection{Using the Pipeline API}

\begin{lstlisting}[caption={Pipeline composition}]
from emic.sources.synthetic import EvenProcessSource
from emic.sources.transforms import TakeN
from emic.inference import Spectral, SpectralConfig
from emic.analysis import Summary
from emic.output import DiagramRenderer

# Compose pipeline
pipeline = (
    EvenProcessSource(p=0.5, _seed=123)
    >> TakeN(50_000)
    >> Spectral(SpectralConfig(max_history=8))
    >> Summary()
)

# Execute
summary = pipeline.run()
print(summary)
\end{lstlisting}

\section{Visualization}

\subsection{State Diagrams}

\begin{lstlisting}[caption={Generating state diagrams}]
from emic.output import DiagramRenderer

# Render to various formats
renderer = DiagramRenderer(result.machine)

# Graphviz DOT format
dot_source = renderer.to_dot()
print(dot_source)

# Save as PNG (requires graphviz)
renderer.save_png("golden_mean.png")

# Save as SVG
renderer.save_svg("golden_mean.svg")
\end{lstlisting}

\subsection{LaTeX Export}

\begin{lstlisting}[caption={LaTeX/TikZ export}]
from emic.output import LatexRenderer

latex = LatexRenderer(result.machine)
tikz_code = latex.to_tikz()

# Insert into your LaTeX document
print(tikz_code)
\end{lstlisting}

\section{Comparing Algorithms}

\begin{lstlisting}[caption={Algorithm comparison}]
from emic.sources.synthetic import GoldenMeanSource
from emic.sources.transforms import TakeN
from emic.inference import CSSR, Spectral, BSI, CSM
from emic.inference import CSSRConfig, SpectralConfig, BSIConfig, CSMConfig
from emic.analysis import statistical_complexity

# Generate data
source = GoldenMeanSource(p=0.5, _seed=42)
sequence = list(TakeN(50_000)(source))

# Define algorithms
algorithms = [
    ("CSSR", CSSR(CSSRConfig(max_history=5))),
    ("Spectral", Spectral(SpectralConfig(max_history=8))),
    ("BSI", BSI(BSIConfig(max_states=5, n_samples=500))),
    ("CSM", CSM(CSMConfig(history_length=5))),
]

# Compare
for name, algo in algorithms:
    result = algo.infer(sequence)
    cmu = statistical_complexity(result.machine)
    print(f"{name:10} | States: {len(result.machine):2} | Cmu: {cmu:.3f}")
\end{lstlisting}

Output:
\begin{lstlisting}
CSSR       | States:  2 | Cmu: 0.918
Spectral   | States:  2 | Cmu: 0.921
BSI        | States:  2 | Cmu: 0.915
CSM        | States:  2 | Cmu: 0.919
\end{lstlisting}

\section{Advanced Usage}

\subsection{Custom Sources}

Implement the \texttt{Source} protocol to create custom data sources:

\begin{lstlisting}[caption={Custom source implementation}]
from typing import Iterator
from dataclasses import dataclass

@dataclass
class MyCustomSource:
    """A custom data source."""
    parameter: float

    def __iter__(self) -> Iterator[str]:
        """Generate symbols."""
        import random
        while True:
            if random.random() < self.parameter:
                yield "1"
            else:
                yield "0"

# Use in pipeline
source = MyCustomSource(parameter=0.7)
sequence = list(TakeN(10_000)(source))
\end{lstlisting}

\subsection{Custom Algorithms}

Implement the \texttt{InferenceAlgorithm} protocol for custom inference:

\begin{lstlisting}[caption={Custom algorithm skeleton}]
from dataclasses import dataclass
from collections.abc import Iterable
from emic.inference import InferenceResult
from emic.types import EpsilonMachine

@dataclass
class MyConfig:
    """Configuration for my algorithm."""
    max_states: int = 10

@dataclass
class MyAlgorithm:
    """Custom inference algorithm."""
    config: MyConfig

    def infer(
        self,
        sequence: Iterable[str],
        alphabet: frozenset[str] | None = None,
    ) -> InferenceResult[str]:
        """Infer epsilon-machine."""
        symbols = list(sequence)

        # Your algorithm here...
        machine = build_machine_somehow(symbols)

        return InferenceResult(
            machine=machine,
            algorithm="MyAlgorithm",
            diagnostics={},
        )
\end{lstlisting}

\subsection{Serialization}

\begin{lstlisting}[caption={Saving and loading machines}]
from emic.output import save_machine, load_machine

# Save to JSON
save_machine(result.machine, "my_machine.json")

# Load from JSON
loaded = load_machine("my_machine.json")

# Verify
assert len(loaded) == len(result.machine)
\end{lstlisting}

% ===================================================================
\chapter{Performance Evaluation}
\label{ch:performance}
% ===================================================================

This chapter presents empirical performance measurements comparing all five implemented inference algorithms, with particular attention to both runtime efficiency and \emph{correctness}---whether each algorithm recovers the true number of causal states.

\section{Experimental Setup}

Benchmarks were conducted with the following configuration:
\begin{itemize}
    \item \textbf{Host}: Apple Silicon Mac, 64 GB RAM
    \item \textbf{Container}: OrbStack Linux (aarch64), 16 GB allocated, 16 cores
    \item \textbf{Python}: 3.11.13
    \item \textbf{Repetitions}: 5 runs per configuration (plus 1 warmup run)
    \item \textbf{Algorithms}: CSSR, Spectral, CSM, BSI, NSD
    \item \textbf{Processes}: Golden Mean ($p=0.5$, 2 states), Even Process ($p=0.5$, 2 states), Biased Coin ($p=0.7$, 1 state), Periodic([0,1,0], 3 states)
    \item \textbf{Sample sizes}: 100, 1{,}000, 10{,}000, 100{,}000, 1{,}000{,}000 symbols
\end{itemize}

A total of 100 configurations (5 algorithms $\times$ 4 processes $\times$ 5 sample sizes) were evaluated, yielding 500 individual benchmark runs.

\subsection{Benchmark Methodology}

Each benchmark run measures runtime, peak memory, and correctness using Python's \texttt{tracemalloc} and \texttt{time.perf\_counter}:

\begin{lstlisting}[language=Python]
def run_single_benchmark(algorithm, data, expected_states):
    """Run a single benchmark iteration."""
    gc.collect()  # Clean state before measurement
    tracemalloc.start()
    start_time = time.perf_counter()

    result = algorithm.infer(data, alphabet=set(data))

    end_time = time.perf_counter()
    _, peak_memory = tracemalloc.get_traced_memory()
    tracemalloc.stop()

    num_states = len(result.machine.states)
    return BenchmarkResult(
        runtime_seconds=end_time - start_time,
        peak_memory_bytes=peak_memory,
        num_states=num_states,
        correct=(num_states == expected_states),
    )
\end{lstlisting}

Correctness is defined as recovering the exact number of causal states. While this is a strict criterion (transition probabilities could differ), it provides a clear, unambiguous metric for comparing algorithms.

\section{Runtime Scaling}

Figure~\ref{fig:runtime-scaling} shows how inference time scales with sequence length across all five algorithms.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{../../experiments/benchmarks/results/figures/runtime_scaling.pdf}
    \caption{Runtime scaling comparison across all algorithms (log-log scale). Error bars show standard deviation across 5 runs, averaged over all processes.}
    \label{fig:runtime-scaling}
\end{figure}

Key observations:
\begin{itemize}
    \item \textbf{CSM} is the fastest algorithm, completing 1M samples in $\sim$1.1 seconds
    \item \textbf{Spectral} and \textbf{NSD} show similar performance ($\sim$3.5--4s at 1M samples)
    \item \textbf{CSSR} is slower ($\sim$13s at 1M samples) due to statistical hypothesis testing overhead
    \item \textbf{BSI} is the slowest ($\sim$15--38s at 1M samples) due to Bayesian sampling
\end{itemize}

All algorithms exhibit approximately linear scaling with $N$, as expected from their $O(N)$ time complexity for fixed history length.

\section{Memory Usage}

Figure~\ref{fig:memory-scaling} shows peak memory usage as a function of sequence length.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.8\textwidth]{../../experiments/benchmarks/results/figures/memory_scaling.pdf}
    \caption{Peak memory usage comparison. Memory scales linearly with sequence length for both algorithms.}
    \label{fig:memory-scaling}
\end{figure}

All algorithms show similar memory scaling, with peak usage under 1 MB for $N = 100{,}000$ and approximately 7.6--7.8 MB at $N = 1{,}000{,}000$. Memory is dominated by the input data representation rather than algorithm-specific structures.

\section{Correctness Analysis}

Runtime is only meaningful if the algorithm produces correct results. We define a run as \emph{correct} if the inferred machine has the same number of states as the ground truth $\eps$-machine.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{../../experiments/benchmarks/results/figures/correctness_by_process.pdf}
    \caption{Algorithm correctness by process, averaged over all sample sizes. No algorithm achieves 100\% correctness across all processes.}
    \label{fig:correctness}
\end{figure}

Table~\ref{tab:correctness} summarizes the correctness findings:

% Auto-generated from experiments/benchmarks/analyze_results.ipynb
\input{generated/tab-correctness}

Table~\ref{tab:correctness-detail} provides a detailed breakdown by sample size, revealing important patterns in how algorithms succeed or fail:

% Auto-generated from experiments/benchmarks/analyze_results.ipynb
\input{generated/tab-correctness-detail}

\textbf{Key findings}:
\begin{itemize}
    \item \textbf{Spectral} achieves the highest overall correctness (\spectralLargeNCorrectness\% at $N \geq 1{,}000$) through adaptive parameter selection and belief state clustering. It reliably handles all four test processes at sufficient sample sizes.
    \item \textbf{CSSR} achieves \cssrOverallCorrectness\% correctness but fails on Even Process at larger sample sizes, over-splitting to 4 states instead of 2. This appears to be excessive sensitivity in significance testing with abundant data.
    \item \textbf{NSD} performs well on stochastic processes (100\% on Golden Mean, Even Process, Biased Coin) but fails completely on the deterministic Periodic process, consistently inferring 2 states instead of 3.
    \item \textbf{CSM} fails on Even Process (always finds 3 states) and Periodic (always finds 2 states), suggesting the merge threshold is not universal.
    \item \textbf{BSI} struggles with structured processes, finding only 1 state for Golden Mean and Even Process.
\end{itemize}

\subsection{Failure Mode Analysis}

The failure modes differ systematically:
\begin{description}
    \item[Over-splitting] BSI and CSSR (on Even Process) find \emph{more} states than the truth
    \item[Under-splitting] CSM, NSD, and Spectral tend to find \emph{fewer} states (2 states for 3-state Periodic)
\end{description}

Notably, CSSR's over-splitting on Even Process \emph{worsens} with more data---it succeeds at $N=1{,}000$ but fails at $N \geq 10{,}000$. This counter-intuitive behavior suggests the significance threshold may be too aggressive, detecting spurious distinctions in larger samples.

These patterns suggest different remedies: BSI may need stronger priors; CSM/NSD may need lower merge thresholds; CSSR may need a higher significance level for the Even Process. \emph{Systematic hyperparameter tuning is an important direction for future work.}

\section{Algorithm Comparison}

Figure~\ref{fig:algo-comparison} presents a direct comparison at $N = 1{,}000{,}000$ across different processes, \emph{including only runs that produced correct results}.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{../../experiments/benchmarks/results/figures/algorithm_comparison.pdf}
    \caption{Algorithm runtime comparison at $N = 1{,}000{,}000$ for different processes (correct results only). Missing bars indicate the algorithm never achieved correctness for that process at this sample size.}
    \label{fig:algo-comparison}
\end{figure}

Key observations:
\begin{itemize}
    \item \textbf{Spectral} achieves correctness on all four processes at $N \geq 10{,}000$
    \item \textbf{CSSR} succeeds on Periodic but fails on Even Process at large $N$
    \item When correct, CSM and NSD are significantly faster than CSSR
    \item BSI is slowest even when correct (Biased Coin: 38s vs CSSR's 13s)
\end{itemize}

\section{Process-Specific Analysis}

Figure~\ref{fig:runtime-by-process} breaks down runtime scaling by process type, with X markers indicating incorrect results.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=\textwidth]{../../experiments/benchmarks/results/figures/runtime_by_process.pdf}
    \caption{Runtime scaling for each process type. Connected points indicate correct results; X markers indicate incorrect state counts.}
    \label{fig:runtime-by-process}
\end{figure}

The per-process figures (available in the supplementary materials) show algorithm behavior on each process:
\begin{itemize}
    \item \textbf{Golden Mean}: Spectral, CSSR, NSD, and CSM all converge correctly at $N \geq 1{,}000$
    \item \textbf{Even Process}: Spectral and NSD succeed; CSSR over-splits at large $N$; CSM under-splits
    \item \textbf{Biased Coin}: Most algorithms succeed, with Spectral and CSSR reliable at $N \geq 1{,}000$
    \item \textbf{Periodic}: Spectral and CSSR succeed; NSD and CSM under-split to 2 states
\end{itemize}

% ===================================================================
\section{Noise Robustness Analysis}
\label{sec:noise-robustness}
% ===================================================================

Real-world data often contains observation noise. We systematically evaluated how each algorithm degrades under noise by applying bit-flip noise to the Golden Mean process at levels $\varepsilon \in [0, 0.5]$.

\subsection{Experimental Setup}

We generated $N=10{,}000$ samples from Golden Mean$(p=0.5)$ and applied a binary symmetric channel (bit-flip) with error probability $\varepsilon$. Each (algorithm, $\varepsilon$) combination was evaluated over 50 independent runs with different random seeds.

\subsection{Results}

Figure~\ref{fig:noise-robustness} and Table~\ref{tab:noise-robustness} summarize the results.

\begin{figure}[htbp]
\centering
\includegraphics[width=0.85\textwidth]{figures/states_vs_noise.pdf}
\caption{Mean inferred states vs.\ noise level for Golden Mean (true states = 2). Error bars show $\pm 1$ standard deviation across 50 runs.}
\label{fig:noise-robustness}
\end{figure}

\begin{table}[htbp]
\centering
\caption{Inferred states (mean $\pm$ std) at key noise levels. True states = 2.}
\label{tab:noise-robustness}
\small
\input{generated/tab-noise-robustness}
\end{table}

\subsection{Analysis}

The algorithms exhibit distinct failure modes under noise:

\begin{itemize}
    \item \textbf{CSSR}: Over-splits progressively as noise increases, finding 3--4 states at moderate noise. The statistical splitting tests interpret noise-induced variations as genuine state distinctions.

    \item \textbf{CSM}: Bipolar behavior---over-splits at low noise ($\varepsilon \leq 0.05$) but under-splits at high noise ($\varepsilon \geq 0.3$), eventually collapsing to 1 state.

    \item \textbf{Spectral}: Robust at low noise but under-splits at $\varepsilon \geq 0.15$, collapsing to 1 state. The SVD rank becomes unreliable when noise dominates signal.

    \item \textbf{NSD}: Similar to Spectral---accurate at low noise, collapses to 1 state at high noise.

    \item \textbf{BSI}: Consistently finds 3+ states even at $\varepsilon = 0$, indicating hyperparameter sensitivity. The Bayesian model selection favors more states than necessary with default priors.
\end{itemize}

\textbf{Practical recommendations}: No algorithm is universally robust to noise. For noisy data:
\begin{enumerate}
    \item Cross-validate with multiple algorithms
    \item Be skeptical when algorithms disagree
    \item Consider noise-aware inference methods (not yet implemented in \emic{})
\end{enumerate}

\subsection{Impact on Complexity Measures}

Beyond state counts, we examined how inferred complexity measures degrade with noise. Figures~\ref{fig:entropy-vs-noise} and~\ref{fig:complexity-vs-noise} show the entropy rate $h_\mu$ and statistical complexity $C_\mu$ as functions of noise level.

\begin{figure}[htbp]
\centering
\includegraphics[width=0.85\textwidth]{figures/entropy_rate_vs_noise.pdf}
\caption{Inferred entropy rate vs.\ noise level. The true Golden Mean entropy rate is approximately 0.67 bits.}
\label{fig:entropy-vs-noise}
\end{figure}

\begin{figure}[htbp]
\centering
\includegraphics[width=0.85\textwidth]{figures/complexity_vs_noise.pdf}
\caption{Inferred statistical complexity vs.\ noise level. The true Golden Mean complexity is approximately 1.0 bit.}
\label{fig:complexity-vs-noise}
\end{figure}

Key observations:
\begin{itemize}
    \item Entropy rate estimates are relatively stable at low noise---the randomness in the process is well-captured even when state structure is misidentified.
    \item Statistical complexity is more sensitive to state count errors. Over-splitting inflates $C_\mu$ (more states = more memory), while under-splitting deflates it.
    \item At high noise ($\varepsilon \geq 0.3$), algorithms that collapse to 1 state report $C_\mu = 0$ (no memory needed for an IID process).
\end{itemize}

\section{Summary of Performance Results}

Table~\ref{tab:perf-summary} summarizes the key performance characteristics.

% Auto-generated from experiments/benchmarks/analyze_results.ipynb
\input{generated/tab-perf-summary}

\section{The Value of Comprehensive Testing}

These benchmarks reveal a crucial insight: \emph{speed without correctness is meaningless}. Prior to this systematic evaluation, one might have assumed that all algorithms would produce equivalent results and differ only in runtime. The data shows otherwise:

\begin{itemize}
    \item \textbf{Spectral} achieves the highest correctness (\spectralLargeNCorrectness\% at $N \geq 1{,}000$) through adaptive parameters and belief state clustering, but is slower due to SVD computation
    \item \textbf{CSSR} achieves \cssrLargeNCorrectness\% correctness but fails on Even Process at large $N$ due to over-splitting
    \item \textbf{NSD} achieves \nsdLargeNCorrectness\% correctness with \nsdSpeedup$\times$ speedup, but has a blind spot for deterministic processes
    \item \textbf{CSM} is \csmSpeedup$\times$ faster than CSSR with \csmLargeNCorrectness\% correctness---reasonable for exploratory analysis on simple processes
\end{itemize}

This underscores the importance of \textbf{golden tests}---benchmarks against processes with known analytical solutions. Without them, incorrect implementations can go undetected.

\textbf{Recommendations}:
\begin{enumerate}
    \item \textbf{Use Spectral} for general inference when correctness is paramount and runtime is acceptable
    \item \textbf{Use CSSR} for Periodic and deterministic processes; use \textbf{NSD} for fast exploratory analysis on stochastic processes
    \item \textbf{Cross-validate}: Run multiple algorithms and compare results when correctness is critical
    \item \textbf{Always validate} inferred machines against known processes before trusting results on unknown data
\end{enumerate}

% ===================================================================
\chapter{Conclusion}
\label{ch:conclusion}
% ===================================================================

\section{Summary}

This report has presented \emic{}, a production-quality Python library for $\eps$-machine inference from symbolic time series data. Key contributions include:

\begin{enumerate}
    \item \textbf{Modern implementation}: Type-annotated Python with immutable data structures, extensive documentation, and \testCoverage\%+ test coverage.

    \item \textbf{Multiple algorithms}: Five inference methods (CSSR, Spectral, CSM, BSI, NSD) with different trade-offs, enabling practitioners to choose based on data characteristics.

    \item \textbf{Rigorous validation}: Golden tests against known analytical solutions for canonical processes (Golden Mean, Even Process, Periodic), ensuring correctness.

    \item \textbf{Composable architecture}: Pipeline operators and protocol-based extension points make the library easy to use and extend.

    \item \textbf{Comprehensive analysis}: Built-in computation of statistical complexity, entropy rate, and related measures.
\end{enumerate}

\section{Limitations}

Current limitations of \emic{}:

\begin{itemize}
    \item \textbf{Discrete alphabets only}: Continuous-valued processes require discretization
    \item \textbf{Stationarity assumption}: All algorithms assume stationary processes
    \item \textbf{Single sequences}: Batch inference from multiple sequences not yet supported
    \item \textbf{Memory scaling}: Large history lengths ($L > 15$) may exhaust memory
    \item \textbf{No GPU acceleration}: Spectral SVD could benefit from GPU, not yet implemented
\end{itemize}

\section{Future Work}

Several directions for future development:

\subsection{Algorithm Correctness Improvements}

The performance evaluation (Chapter~\ref{ch:performance}) reveals a nuanced landscape of algorithm capabilities. After recent improvements to the Spectral algorithm (adaptive parameters, belief state clustering), it now achieves \spectralLargeNCorrectness\% correctness at $N \geq 1{,}000$, making it the most reliable algorithm. Remaining challenges include:

\begin{itemize}
    \item \textbf{CSSR Even Process failure}: CSSR over-splits on Even Process at large $N$, possibly due to statistical test sensitivity. Investigating adaptive significance levels or early stopping criteria could help.

    \item \textbf{CSM and BSI reliability}: Both algorithms achieve only \bsiLargeNCorrectness--\csmLargeNCorrectness\% correctness. CSM's merge threshold and BSI's sample count need principled tuning---perhaps using cross-validation or information criteria.

    \item \textbf{NSD deterministic blindness}: NSD fails on Periodic (deterministic) processes but works well on stochastic ones. Understanding this limitation could lead to hybrid approaches.

    \item \textbf{Ensemble methods}: Combining predictions from multiple algorithms (e.g., voting on state count, cross-validation) may achieve better robustness than any single algorithm.
\end{itemize}

\subsection{Quantum Computational Mechanics}

Recent work~\cite{gu2012quantum,thompson2018causal} has shown that quantum models can achieve lower complexity than classical $\eps$-machines for certain processes. Future versions of \emic{} could implement:

\begin{itemize}
    \item Quantum state representations
    \item Quantum complexity measures ($C_q$)
    \item Comparison between classical and quantum models
\end{itemize}

\subsection{Continuous Alphabets}

Extending to continuous-valued observations would enable applications in:
\begin{itemize}
    \item Financial time series
    \item Physiological signals (EEG, ECG)
    \item Physical sensor data
\end{itemize}

Possible approaches include kernel methods, binning strategies, or direct continuous-state inference.

\subsection{Online/Streaming Inference}

Current algorithms operate in batch mode. Online variants would enable:
\begin{itemize}
    \item Real-time complexity monitoring
    \item Adaptive model updates
    \item Anomaly detection via complexity changes
\end{itemize}

\subsection{Parallel and GPU Acceleration}

For large-scale applications:
\begin{itemize}
    \item Parallel Hankel matrix construction
    \item GPU-accelerated SVD for spectral learning
    \item Distributed Gibbs sampling for BSI
\end{itemize}

\subsection{Interactive Visualization}

A web-based interface for:
\begin{itemize}
    \item Interactive state diagram exploration
    \item Algorithm comparison dashboards
    \item Educational demonstrations
\end{itemize}

\section{Availability}

\emic{} is open source under the MIT license:

\begin{itemize}
    \item \textbf{Repository}: \url{https://github.com/johnazariah/emic}
    \item \textbf{Documentation}: \url{https://johnazariah.github.io/emic/}
    \item \textbf{PyPI}: \url{https://pypi.org/project/emic}
\end{itemize}

Contributions are welcome via pull requests.

% ===================================================================
% APPENDICES
% ===================================================================
\appendix

\chapter{API Reference}
\label{app:api}

This appendix provides a summary of the public API. Full documentation is available at \url{https://johnazariah.github.io/emic/}.

\section{Core Types}

\begin{description}
    \item[\texttt{Symbol}] Type alias for hashable symbols (\texttt{Hashable}).

    \item[\texttt{StateId}] Unique identifier for causal states (\texttt{int}).

    \item[\texttt{Probability}] Probability value in $[0, 1]$ (\texttt{float}).

    \item[\texttt{Alphabet}] Immutable set of symbols with validation.
        \begin{itemize}
            \item \texttt{symbols: frozenset[Symbol]} -- The symbol set
            \item \texttt{size: int} -- Number of symbols
            \item \texttt{\_\_contains\_\_(symbol)} -- Membership test
        \end{itemize}

    \item[\texttt{CausalState}] A single causal state.
        \begin{itemize}
            \item \texttt{id: StateId} -- Unique identifier
            \item \texttt{probability: Probability} -- Stationary probability
        \end{itemize}

    \item[\texttt{Transition}] Labeled transition between states.
        \begin{itemize}
            \item \texttt{source: StateId} -- Source state
            \item \texttt{symbol: Symbol} -- Transition symbol
            \item \texttt{target: StateId} -- Target state
            \item \texttt{probability: Probability} -- Conditional probability
        \end{itemize}

    \item[\texttt{EpsilonMachine}] Complete $\eps$-machine representation.
        \begin{itemize}
            \item \texttt{alphabet: Alphabet} -- Symbol alphabet
            \item \texttt{states: tuple[CausalState, ...]} -- Causal states
            \item \texttt{transitions: tuple[Transition, ...]} -- All transitions
            \item \texttt{initial\_state: StateId} -- Starting state
            \item \texttt{transition\_matrix(symbol)} -- Get matrix for symbol
            \item \texttt{stationary\_distribution} -- State probabilities
        \end{itemize}
\end{description}

\section{Inference Algorithms}

All algorithms implement the \texttt{InferenceAlgorithm} protocol:

\begin{lstlisting}[language=Python]
class InferenceAlgorithm(Protocol[C]):
    config: C
    def infer(self, data: Sequence[Symbol]) -> InferenceResult: ...
\end{lstlisting}

\begin{description}
    \item[\texttt{CSSR(config: CSSRConfig)}] Causal State Splitting Reconstruction.
        \begin{itemize}
            \item \texttt{max\_length: int} -- Maximum history length (default: 10)
            \item \texttt{significance: float} -- Test threshold (default: 0.05)
        \end{itemize}

    \item[\texttt{Spectral(config: SpectralConfig)}] Spectral learning algorithm.
        \begin{itemize}
            \item \texttt{rank: int | None} -- SVD rank (auto if None)
            \item \texttt{tolerance: float} -- SVD tolerance (default: 1e-6)
        \end{itemize}

    \item[\texttt{CSM(config: CSMConfig)}] Causal State Merging.
        \begin{itemize}
            \item \texttt{max\_states: int} -- Maximum state count
            \item \texttt{merge\_threshold: float} -- Similarity threshold
        \end{itemize}

    \item[\texttt{BSI(config: BSIConfig)}] Bayesian Structural Inference.
        \begin{itemize}
            \item \texttt{max\_states: int} -- Maximum model order
            \item \texttt{iterations: int} -- MCMC iterations
            \item \texttt{prior: str} -- Prior type
        \end{itemize}

    \item[\texttt{NSD(config: NSDConfig)}] Nonlinear System Discovery.
        \begin{itemize}
            \item \texttt{embedding\_dim: int} -- Embedding dimension
            \item \texttt{delay: int} -- Time delay
        \end{itemize}
\end{description}

\section{Analysis Functions}

\begin{description}
    \item[\texttt{statistical\_complexity(machine)}] Compute $\Cmu = -\sum_s \pi_s \log_2 \pi_s$.

    \item[\texttt{entropy\_rate(machine)}] Compute $\hmu$ from transition probabilities.

    \item[\texttt{excess\_entropy(machine)}] Compute $\mathbf{E}$ (mutual information past/future).

    \item[\texttt{analyze(machine)}] Return \texttt{AnalysisSummary} with all measures.
\end{description}

\section{Sources}

\begin{description}
    \item[\texttt{GoldenMean(p)}] Golden Mean process with parameter $p$.
    \item[\texttt{EvenProcess(p)}] Even Process with parameter $p$.
    \item[\texttt{BiasedCoin(p)}] Biased coin (i.i.d.) with $P(1) = p$.
    \item[\texttt{Periodic(pattern)}] Periodic sequence from pattern.
    \item[\texttt{SequenceData(sequence)}] Wrap existing sequence.
\end{description}

\section{Output}

\begin{description}
    \item[\texttt{to\_diagram(machine)}] Generate Graphviz DOT representation.
    \item[\texttt{to\_latex(machine)}] Generate TikZ code for diagrams.
    \item[\texttt{save(machine, path)}] Save to JSON file.
    \item[\texttt{load(path)}] Load from JSON file.
\end{description}

% ===================================================================
% ACKNOWLEDGMENTS
% ===================================================================
\chapter*{Acknowledgments}
\addcontentsline{toc}{chapter}{Acknowledgments}

I thank Dr.\ Jairam Manjunathaiah for generously providing the Mac Studio on which the entire \emic{} library was developed, all computational experiments were conducted, and this document was written.

% ===================================================================
% BIBLIOGRAPHY
% ===================================================================
\printbibliography

\end{document}
