# Architecture

walkingpandas uses a **split data model** that separates the static network topology from the dynamic walk data. This design allows the small network to stay in memory while walk data scales out-of-core via Parquet and DuckDB.

## Overview

```mermaid
flowchart TB
    subgraph inMemory ["In-Memory (DuckDB)"]
        nodes["wp_nodes\n(node_id, geom, attributes)"]
        edges["wp_edges\n(edge_id, source_node, target_node, geom)"]
        maps["wp_node_map / wp_edge_map\n(string ID translation)"]
    end
    subgraph outOfCore ["Out-of-Core (Parquet)"]
        walks["Walk data\n(walk_id, sequence_idx, node_id, exit_edge, ...)"]
    end
    Network --> inMemory
    WalkFrame --> outOfCore
    WalkFrame -.->|"uses"| Network
    WalkFrame -->|".compute()"| Result["pandas DataFrame"]
```

## The Network (static and in-memory)

- **Nodes**: `node_id` (BIGINT), optional coordinates/geometry, static attributes
- **Edges**: `edge_id` (BIGINT), `source_node`, `target_node`, optional geometry, static attributes
- All IDs stored as 64-bit integers for fast joins; string IDs are auto-translated via mapping tables
- Stored in DuckDB tables (with spatial indexes when geometry is present)
- Relatively small -- fits in memory

The Network can be created from:

- Parquet files or GeoDataFrames (`Network(nodes=..., edges=...)`)
- GeoDataFrames directly (`Network.from_gdfs(...)`)
- NetworkX graphs (`Network.from_networkx(...)`)
- OSMnx street networks (`Network.from_osmnx(...)`)
- Walk data (inferred topology) (`Network.from_walks(...)`)

## The walks/paths (dynamic and out-of-core)

- Stored as partitioned Parquet files
- Uses the "row-per-step" format ([superset schema](data-schema.md#superset-schema))
- Supports time-less, single-timestamp, and dwell-time data
- Handles billions of rows efficiently via DuckDB

## Lazy evaluation

WalkFrame queries are built up lazily. Each method call (`.filter()`, `.passing_through()`, `.edge_frequencies()`, etc.) records the operation in an internal query state. Only when `.compute()` is called does the library build and execute the SQL query against DuckDB.

This design means:

- **No wasted computation** -- filters and aggregations are combined into a single query
- **Method chaining** -- build complex queries fluently
- **Late optimization** -- DuckDB optimizes the full query plan

## Spatial optimization (reverse spatial filtering)

When nodes carry coordinates, spatial filters like `within_bounds()` and `intersecting()` use **reverse spatial filtering**:

1. Query the small Network (in-memory DuckDB) first to find matching node/edge IDs
2. Apply fast integer `IN` clauses against the Parquet walk data
3. Avoid expensive spatial operations on billions of rows

This strategy exploits the split data model: the spatial index lives on the small Network, while the bulk data only needs integer comparisons.
