# `r_bridge` — Design Plan

## Prompts that shaped this project

The following user prompts drove the design and implementation decisions recorded
in this document.

**Initial design**
> "Python library that runs a persistent R subprocess and exposes a Pythonic API over it."

**Verbose mode**
> "I need a verbose mode which shows the output from the R side and functions called there."

**Windows compatibility**
> "Will this also work on Windows with different path separators, line endings and encodings?"

**jsonlite availability**
> "Make sure that jsonlite is available in the R session startup."

---

## Communication Protocol

**Line-delimited JSON over stdin/stdout**, with a `__RBRIDGE__:` sentinel prefix on
every protocol line. This keeps stray `print()`/`cat()` output from corrupting the
stream. R uses `jsonlite`.

```
Python → R:  __RBRIDGE__:{"id":"<uuid>","op":"call_func","payload":{...}}\n
R → Python:  __RBRIDGE__:{"id":"<uuid>","status":"ok","payload":{...},"warnings":[...]}\n
```

Supported operations: `set_var`, `get_var`, `call_func`, `eval_expr`, `exec_script`, `ls`, `ping`, `shutdown`.

---

## Module Structure

```
r_bridge/
├── __init__.py              # Public re-exports
├── bridge.py                # RBridge class — lifecycle, send/recv, lock, timeouts, verbose mode
├── protocol.py              # Envelope construction, sentinel stripping, parsing
├── serializer.py            # Python/numpy/pandas → tagged JSON dict
├── deserializer.py          # Tagged JSON dict → Python/numpy/pandas
├── type_hints.py            # TypeHint enum (SCALAR, NUMPY, PANDAS, LIST, RAW)
├── exceptions.py            # RError, RBridgeError, RStartupError, RTimeoutError
├── utils.py                 # drain_stderr (with optional on_line callback), get_r_executable
└── _r_scripts/
    └── bridge_loop.R        # R-side REPL loop
```

---

## Type Conversion Table

### Python → R

| Python | R | Notes |
|---|---|---|
| `None` | `NULL` | JSON `null` |
| `bool` | `logical(1)` | |
| `int` | `integer(1)` | tagged to preserve `L` |
| `float` | `numeric(1)` | `nan`/`inf` tagged |
| `str` | `character(1)` | |
| `list` (homogeneous) | atomic vector | |
| `dict` | named `list()` | |
| `numpy.ndarray` 1-D | atomic vector | dtype → R type |
| `numpy.ndarray` 2-D | `matrix` | tagged with dims |
| `pandas.DataFrame` | `data.frame` | column dtypes guide R type |
| `pandas.Series` | named vector | |
| `datetime.datetime` | `POSIXct` | ISO 8601 |
| `pandas.Categorical` | `factor` | round-trippable |

### R → Python

| R | Python | Notes |
|---|---|---|
| `NULL` | `None` | |
| `logical(1)` | `bool` | |
| `logical(n>1)` | `numpy.ndarray(bool)` | |
| `integer(1)` | `int` | |
| `integer(n>1)` | `numpy.ndarray(int32)` | |
| `numeric(1)` | `float` | |
| `numeric(n>1)` | `numpy.ndarray(float64)` | |
| `character(1)` | `str` | |
| `character(n>1)` | `list[str]` | |
| `list` (named) | `dict` | |
| `list` (unnamed) | `list` | |
| `data.frame` | `pandas.DataFrame` | |
| `matrix` | `numpy.ndarray` (2-D) | |
| `factor` | `pandas.Categorical` | |
| `Date` | `datetime.date` | |
| `POSIXct`/`POSIXlt` | `datetime.datetime` (UTC) | |
| `NA` | `float('nan')` / `pd.NA` / `None` | context-dependent |
| `NaN` | `float('nan')` | |
| `Inf`/`-Inf` | `float('inf')`/`float('-inf')` | |

### Special Value Tagging

Special values use tagged dicts in the JSON payload:

```json
{"__type__": "na",        "na_type": "logical|integer|real|complex|character"}
{"__type__": "nan"}
{"__type__": "inf",       "sign": 1}
{"__type__": "complex",   "r": 1.0, "i": 2.0}
{"__type__": "factor",    "levels": [...], "codes": [...], "ordered": false}
{"__type__": "dataframe", "columns": {...}}
{"__type__": "matrix",    "data": [...], "nrow": 3, "ncol": 2, "byrow": false}
{"__type__": "posixct",   "iso8601": "2026-03-21T00:00:00Z"}
{"__type__": "date",      "iso8601": "2026-03-21"}
```

`result_type_hint` lets callers override defaults: `"scalar"`, `"numpy"`, `"pandas"`, `"list"`, `"raw"`.

---

## Public API (`RBridge` class)

### Attribute-access interface (primary)

```python
with RBridge() as r:
    r.x = 42                        # set variable in R global env
    r.df = pd.DataFrame(...)        # set a data frame
    val = r.x                       # get variable from R
    result = r.mean([1, 2, 3])      # call R function, returns Python value
    result = r.t_test(r.x, mu=0)    # keyword args become R named args
```

`__getattr__` returns either the variable value **or** a callable proxy, resolved lazily:
- If the name exists in R's global env → returns its value
- Otherwise → returns an `_RFuncProxy` that calls the R function when invoked

`_RFuncProxy.__call__(*args, result_type_hint=None, timeout=None, **kwargs)` — forwards to `call()`.

### Constructor signature

```python
class RBridge:
    def __init__(
        self,
        r_executable: str | None = None,   # path to Rscript; auto-detected if None
        startup_timeout: float = 15.0,
        call_timeout: float = 60.0,
        env: dict[str, str] | None = None,
        r_libs: list[str] | None = None,
        log_level: str = "WARNING",
        verbose: bool = False,             # print R calls and output to sys.stderr
    )
```

### Lifecycle

```python
def start(self) -> RBridge          # launch R subprocess; returns self
def stop(self, timeout: float = 5.0) -> None
def is_alive(self) -> bool
def __enter__(self) -> "RBridge"
def __exit__(self, *args) -> None
```

### Explicit methods

```python
def set(self, name: str, value: Any, /, *, type_hint=None) -> None
def get(self, name: str, *, result_type_hint=None) -> Any
def call(self, func: str, /, *args, result_type_hint=None, timeout=None, **kwargs) -> Any
def eval(self, expr: str, /, *, result_type_hint=None, timeout=None) -> Any
def execute(self, code: str, /, *, timeout=None) -> None   # run script, discard result
def ls(self) -> list[str]
def ping(self) -> float              # round-trip latency in seconds
@property
def last_stderr(self) -> str
```

### Name collision handling

Names starting with `_` or in the `_BRIDGE_ATTRS` frozenset are routed to
`object.__setattr__`; all other names go to R's global environment.

### Module-level convenience

```python
def start_bridge(**kwargs) -> RBridge: ...
def register_serializer(python_type: type, fn: Callable) -> None: ...
def register_deserializer(r_tag: str, fn: Callable) -> None: ...
```

---

## Verbose Mode

When `verbose=True`, every R operation and all R output are printed to
`sys.stderr` in real time using a human-readable prefix:

```
[R call] mean([1.0, 2.0, 3.0])   ← call_func ops
[R eval] 2 ^ 8                    ← eval_expr ops
[R set]  x = ...                  ← set_var ops
[R get]  x                        ← get_var ops
[R stdout] <line>                 ← non-protocol R stdout (e.g. cat())
[R stderr] <line>                 ← R stderr (e.g. message(), warnings)
```

Implemented via:
- `_format_verbose_call(op, payload)` private helper in `bridge.py`
- `on_line` callback passed to `drain_stderr()` for real-time stderr streaming

---

## Error Handling Strategy

| Layer | Where | What |
|---|---|---|
| 1 | R: `tryCatch` | Every op wrapped; errors → `status: "error"` response with message/call/traceback |
| 2 | R: `withCallingHandlers` | Warnings collected, muffled, returned in `warnings` field |
| 3 | Python: response check | `status == "error"` raises `RError` with message/call/traceback/warnings/stderr |
| 4 | Python: stderr drain thread | Daemon thread prevents pipe-buffer deadlocks; content attached to exceptions |
| 5 | Python: `threading.Event` timeout | `RTimeoutError` raised; bridge marked dead |
| 6 | Python: EOF detection | R process death → `RBridgeError("R process died before sending a response")` |
| 7 | Python: serializer | `TypeError` before send if type has no known conversion |

### Exception Hierarchy

```
RBridgeError (base)
├── RStartupError     — R process failed to start
├── RTimeoutError     — response not received within timeout
├── RError            — R-side error; carries .message, .call, .traceback, .warnings, .stderr
└── RProtocolError    — malformed envelope
```

---

## R-Side Script Structure (`bridge_loop.R`)

Four sections:

1. **Bootstrap** — Windows UTF-8 locale setup; auto-install `jsonlite` if missing
   (`install.packages`), exit with status 2 if still unavailable; set
   `options(warn=1, OutDec=".", useFancyQuotes=FALSE)`.
2. **Serializer helpers (R → JSON)** — `r_to_payload()`, `encode_na()`,
   `encode_factor()`, `encode_dataframe()`, `encode_matrix()`, `encode_posixct()`.
3. **Deserializer helpers (JSON → R)** — `payload_to_r()`, `decode_dataframe()`,
   `decode_matrix()`, `decode_factor()`, `decode_posixct()`.
4. **Main REPL loop** — reads sentinel-prefixed lines from stdin, dispatches to
   `handle_set_var()`, `handle_get_var()`, `handle_call_func()`,
   `handle_eval_expr()`, `handle_ls()`; wraps all execution in
   `withCallingHandlers` + `tryCatch`; writes sentinel-prefixed JSON responses to
   stdout.

Key R loop pattern:

```r
SENTINEL <- "__RBRIDGE__:"

repeat {
  line <- readLines(con = stdin(), n = 1, warn = FALSE)
  if (length(line) == 0) break          # EOF = Python died
  if (!startsWith(line, SENTINEL)) next # ignore stray output

  req <- jsonlite::fromJSON(substring(line, nchar(SENTINEL) + 1), simplifyVector = FALSE)
  # ... dispatch, execute, respond ...
}
```

- `handle_call_func` uses `do.call(match.fun(func), c(positional_args, named_args))`
- `handle_eval_expr` uses `eval(parse(text=expr), envir=.GlobalEnv)`
- `jsonlite::fromJSON(..., simplifyVector=FALSE)` prevents auto-simplification

---

## Startup Sequence

1. `start()` launches `Rscript --vanilla bridge_loop.R` with `stdin/stdout/stderr=PIPE`
2. Stderr drain thread starts (daemon); `on_line` callback wired up if `verbose=True`
3. R bootstraps: sets locale (Windows), auto-installs jsonlite if needed, sends ready signal:
   `__RBRIDGE__:{"id":"__init__","status":"ok","payload":{"ready":true,"jsonlite_version":"...","arrow_available":false}}`
4. Python reads it within `startup_timeout`, stores capabilities
5. `RStartupError` raised if signal not received in time (stderr attached)

---

## Notable Design Decisions

- **Sentinel prefix** — `__RBRIDGE__:` on every protocol line; stray `cat()`/`print()` output is logged/printed but never corrupts the stream.
- **`simplifyVector=FALSE`** — prevents `jsonlite` from auto-simplifying JSON arrays to R vectors, keeping deserialization unambiguous.
- **Single lock** — `threading.Lock` serializes concurrent calls; one in-flight request at a time.
- **Encoding** — `subprocess.Popen` uses `encoding="utf-8"` explicitly; `LANG=en_US.UTF-8` set on non-Windows only; R sets `options(OutDec=".")` and `Sys.setlocale` on Windows to prevent locale-dependent corruption of JSON.
- **`R_LIBS_USER` separator** — uses `os.pathsep` (`;` on Windows, `:` on Unix).
- **`_BRIDGE_ATTRS` frozenset** — names in this set (and names starting with `_`) are routed to `object.__setattr__`; prevents internal attributes from leaking into R's global env.
- **Verbose mode** — all output to `sys.stderr`; stderr streamed in real time via `on_line` callback on the drain thread; stdout non-protocol lines printed inside `_reader`.
- **jsonlite auto-install** — `bridge_loop.R` attempts `install.packages("jsonlite")` before giving up, so a fresh R environment works out of the box.
- **Extension points** — `register_serializer()` / `register_deserializer()` allow custom type handling without modifying the library.

---

## Not Yet Implemented

- **Arrow IPC for large DataFrames** — payload >10 MB switching to base64-encoded Arrow IPC (requires `pyarrow` + R `arrow` package). `arrow_available` is currently hardcoded `FALSE` in `bridge_loop.R`.
