# API Design: `arrays()` Strategy

**Date:** 2026-02-04
**Status:** Implemented
**Author:** Claude (with developer collaboration)

> **Update (2026-02-22):** Since this document was written:
>
> - `allow_union` parameter is now implemented in both `contents()` and
>   `arrays()` (default: `True`). The signature and "Future node type flags"
>   sections below are outdated.
> - `UnionArray` support is fully implemented via the
>   [top-down builder](../impl/2026-02-21-contents-top-down-builder.md).
> - `max_length` parameter was added to `contents()` and `arrays()`. See
>   [max-length-api](2026-02-23-max-length-api.md) for details.

## Overview

This document describes the API for the `arrays()` strategy, which generates
`ak.Array` objects via direct Content constructors. The current implementation
generates arrays with `NumpyArray` leaf contents nested in `RegularArray`,
`ListOffsetArray`, `ListArray`, and `RecordArray` wrappers. The design
anticipates progressive addition of option types, unions, and more in later
iterations.

## Background

From the [UX research](./../research/2026-01-21-ux-interview-01.md),
[type system research](./../research/2026-01-21-type-system-research.md), and
[direct constructors research](./../research/2026-02-04-direct-constructors-research.md):

- The ultimate goal is an `arrays()` strategy that generates fully general
  Awkward Arrays with multiple options to control the layout, data types, missing
  values, masks, and other array attributes. `type` and `form` parameters are in
  scope but not mandatory
- Direct constructors produce real arrays in a single step (no roundtrip through
  forms or buffers)
- Constructor validation catches invalid nesting at construction time
- `ak.Array(layout)` wrapping is zero-cost
- Target users: Awkward developers, scikit-HEP tool developers, physicists

### Why Start with Direct Constructors

Three possible generation pipelines were considered:

1. **Type -> Form -> buffers -> array** (most abstract, most roundtrips)
2. **Form -> buffers -> array** (via `ak.from_buffers()`)
3. **Content layout -> array** (direct constructors, single step)

Pipeline 3 is the most direct: each Content constructor validates its inputs, so
the strategy can lean on built-in checks. It also exercises more internal code
paths than the canonical representations produced by `from_type()`.

### Why Start with Leaf Nodes (NumpyArray and EmptyArray)

- `NumpyArray` and `EmptyArray` are the leaf nodes -- every Awkward Array tree
  terminates in one
- The existing `numpy_arrays()` strategy already generates NumPy ndarrays;
  `arrays()` reuses this work via `numpy_array_contents()`
- `EmptyArray` is a zero-length placeholder with `UnknownType` -- useful for
  testing edge cases with empty/unknown-typed arrays
- Starting with leaf node types validates the `arrays()` interface before
  adding recursive complexity
- Users get immediate value: `arrays()` with no arguments generates flat arrays,
  which is the most common case

## Design Goals

1. **Incremental extensibility**: Adding new node types must not break the
   existing interface
2. **Reuse existing strategies**: Build on `numpy_arrays()`, `supported_dtypes()`,
   etc.
3. **Familiar patterns**: Follow the parameter conventions established by
   `numpy_arrays()`, `types()`, and `numpy_forms()`
4. **Sensible defaults**: Works well out of the box; generates interesting arrays
   without configuration
5. **Composition with types/forms (future)**: The API should accommodate `type`
   and `form` parameters when those pipelines are connected

## API

> **Note:** Signatures below omit `*` separators. See
> [positional-keyword-convention](../notes/2026-02-12-positional-keyword-convention.md)
> for the keyword-only convention adopted after this document was written.
> `arrays()` falls in Group B (all keyword-only): `(draw, *, dtypes=None, ...)`.

### Main Strategy: `arrays()`

```python
@st.composite
def arrays(
    draw: st.DrawFn,
    *,
    # --- Leaf data control ---
    dtypes: st.SearchStrategy[np.dtype] | None = None,
    allow_nan: bool = False,

    # --- Size control ---
    max_size: int = 10,

    # --- Leaf type control ---
    allow_numpy: bool = True,
    allow_empty: bool = True,
    allow_string: bool = True,
    allow_bytestring: bool = True,

    # --- Nesting type control ---
    allow_regular: bool = True,
    allow_list_offset: bool = True,
    allow_list: bool = True,
    allow_record: bool = True,

    # --- Nesting control ---
    max_depth: int = 5,

    # --- Virtualization ---
    allow_virtual: bool = True,
) -> ak.Array:
```

### Parameter Details

#### `dtypes`

Strategy for NumPy dtypes used in `NumpyArray` leaf nodes.

- If `None` (default), uses `supported_dtypes()`.
- If a strategy, draws from it.
- Follows the same convention as `numpy_types(dtypes=...)` and
  `numpy_forms(dtypes=...)`.

Note: Unlike the existing `numpy_arrays()` strategy, `arrays()` does not accept
a plain `np.dtype` value -- only a strategy or `None`. This is because
`arrays()` may draw multiple dtypes for different leaf nodes in a compound
structure (e.g., different record fields). Accepting a single dtype as a
convenience shorthand can be added later without breaking the interface.

#### `allow_nan`

Generate potentially `NaN`/`NaT` values for relevant dtypes.

- `False` (default): no `NaN`/`NaT` values.
- `True`: `NaN`/`NaT` values may appear.
- Same semantics as `numpy_arrays(allow_nan=...)`.

#### `max_size`

Maximum total number of scalar values in the generated array.

- Default: `10`.
- Controls the total scalar budget across all leaf nodes, not just the outermost
  dimension length. `EmptyArray` leaves consume no budget (length 0).
- Uses a "budgeted leaf" approach: each leaf draws up to the remaining budget,
  preventing compound arrays from growing unboundedly.

#### `allow_numpy`, `allow_empty`

Control which leaf Content node types are enabled.

- Both default to `True`.
- `allow_numpy`: Generate `NumpyArray` leaves (primitive data).
- `allow_empty`: Generate `EmptyArray` leaves (zero-length, `UnknownType`).
  `EmptyArray` is unaffected by `dtypes` and `allow_nan`.
- `allow_string`: Generate string content (`ListOffsetArray` with
  `__array__="string"`). Unaffected by `dtypes` and `allow_nan`.
- `allow_bytestring`: Generate bytestring content (`ListOffsetArray` with
  `__array__="bytestring"`). Unaffected by `dtypes` and `allow_nan`.
- At least one leaf type must be enabled; disabling all raises `ValueError`.

#### `allow_regular`, `allow_list_offset`, `allow_list`

Control which structural Content node types are enabled.

- All default to `True`.
- `allow_regular`: Generate `RegularArray` wrappers (fixed-size lists).
- `allow_list_offset`: Generate `ListOffsetArray` wrappers (variable-length lists
  via offsets).
- `allow_list`: Generate `ListArray` wrappers (variable-length lists via
  starts/stops).
- When all three are `False`, only flat leaf arrays are generated (`NumpyArray`
  and/or `EmptyArray`).

- `allow_record`: Generate `RecordArray` nodes (named or tuple records with
  one or more children). Default: `True`. See
  [contents-tree-builder](../impl/2026-02-17-contents-tree-builder.md).
- `allow_virtual`: Generate virtual (lazy) arrays. Default: `True`. Virtual
  arrays show as `??` in the output and exercise lazy evaluation code paths.

Future node type flags (not yet implemented): `allow_option`, `allow_union`.

#### `max_depth`

Maximum nesting depth for structural wrappers.

- Default: `5`.
- `max_depth=0` forces leaf-only arrays (flat `NumpyArray`).
- At each level, a coin flip decides whether to go deeper or produce a leaf.
  See [contents-tree-builder](../impl/2026-02-17-contents-tree-builder.md) for
  the bottom-up tree builder algorithm.

### Return Type

`arrays()` returns `st.SearchStrategy[ak.Array]`.

**Decision:** Return `ak.Array`, not the raw `Content` layout.

**Rationale:**

- `ak.Array` is the user-facing type. Users write code against `ak.Array`, so
  the strategy should produce what they consume.
- `ak.Array(layout)` is zero-cost wrapping.
- Users needing the raw layout can access it via `result.layout`.
- Matches the precedent set by `from_numpy()`, which returns `ak.Array`.

### Parameters NOT Included

#### `allow_structured` / `allow_inner_shape`

Not included. In `numpy_arrays()`, `allow_structured=True` generates structured
NumPy arrays (with named fields), which `ak.from_numpy` converts to
`RecordArray`. In `arrays()`, records will be generated natively via
`RecordArray` when `allow_record=True`. Similarly, multi-dimensional NumpyArrays
(inner_shape) will be covered by `RegularArray` wrapping.

The `arrays()` strategy generates 1-D `NumpyArray` nodes only. Higher-dimensional
structure comes from nesting (lists, records), not from multi-dimensional NumPy
arrays.

#### `type` / `form`

Not included in the initial version. These will be added when the Type->Array
and Form->Array pipelines are connected:

```python
# Future signature extension:
def arrays(
    ...,
    type: ak.types.Type | st.SearchStrategy[ak.types.Type] | None = None,
    form: ak.forms.Form | st.SearchStrategy[ak.forms.Form] | None = None,
) -> st.SearchStrategy[ak.Array]:
```

When `type` or `form` is given, the strategy would generate arrays matching that
type or form. This can be added without breaking the current interface.

#### `allow_datetime`

Not included as a separate parameter. Users who want to exclude datetime dtypes
can filter via the `dtypes` parameter:

```python
non_datetime = st_ak.supported_dtypes().filter(lambda d: d.kind not in ('M', 'm'))
st_ak.constructors.arrays(dtypes=non_datetime)
```

This keeps the `arrays()` parameter list focused on structural concerns. The
`dtypes` parameter already provides this control.

## Implementation

### Two-layer Architecture

The implementation is split across two packages:

- **`contents/content.py`**: `contents()` strategy generates `ak.contents.Content`
  layouts using a bottom-up tree builder
- **`constructors/array_.py`**: `arrays()` is a thin wrapper that calls
  `contents()` and wraps the result in `ak.Array`

### `contents()` Strategy

The `contents()` strategy in `contents/content.py` uses a bottom-up tree
builder. A recursive `_build(depth)` function:

1. Draws "deeper?" or "bottom?" at each level (coin flip)
2. At the bottom, draws a leaf via `leaf_contents()` with a shared scalar
   budget managed by `CountdownDrawer`
3. Going up, draws "another edge?" to decide whether to add sibling children
4. Draws a node type constrained by child count (1 child: any wrapper or
   RecordArray; 2+ children: RecordArray only)

See [contents-tree-builder](../impl/2026-02-17-contents-tree-builder.md) for
full algorithm details.

### `arrays()` Strategy

`arrays()` in `constructors/array_.py` simply forwards all parameters to
`contents()` and wraps the result:

```python
@st.composite
def arrays(draw, ...) -> ak.Array:
    layout = draw(st_ak.contents.contents(...))
    return ak.Array(layout)
```

### Future Expansion

`RecordArray` support has been added via the bottom-up tree builder (see
[contents-tree-builder](../impl/2026-02-17-contents-tree-builder.md)). When
adding option types and unions, the tree builder may need constraint tracking
(e.g., option nodes cannot wrap other option nodes).

### Relationship to Existing Strategies

```text
Existing:
  supported_dtypes()  -->  numpy_arrays()   -->  from_numpy()  --> ak.Array
                       -->  numpy_types()                        (via ak.from_numpy)
                       -->  numpy_forms()

New:
  supported_dtypes()  -->  numpy_array_contents()  -->  contents()  --> Content
                                                        arrays()    --> ak.Array
```

## Module Location

### Directory Structure

```text
src/hypothesis_awkward/strategies/
+-- contents/
|   +-- __init__.py           # Re-exports contents() and individual content strategies
|   +-- content.py            # contents() — bottom-up tree builder
|   +-- leaf.py               # leaf_contents() — leaf node strategy (NumpyArray | EmptyArray)
|   +-- numpy_array.py        # numpy_array_contents()
|   +-- empty_array.py        # empty_array_contents()
|   +-- regular_array.py      # regular_array_contents()
|   +-- list_offset_array.py  # list_offset_array_contents()
|   +-- list_array.py         # list_array_contents()
|   +-- string.py             # string_contents()
|   +-- bytestring.py         # bytestring_contents()
|   +-- record_array.py       # record_array_contents()
+-- constructors/
|   +-- __init__.py           # Re-exports arrays()
|   +-- array_.py             # arrays() — delegates to contents.contents()
+-- builtins_/
+-- forms/
+-- misc/
+-- numpy/
+-- pandas/
+-- types/
+-- __init__.py               # Re-exports arrays via constructors namespace
```

`arrays()` in `constructors/array_.py` is a thin wrapper that forwards all
arguments to `contents.contents()` and wraps the result in `ak.Array`. The
`contents/` package contains the layout generation logic. As more node types are
added, new content strategies will be added to `contents/`:

```text
src/hypothesis_awkward/strategies/contents/
+-- __init__.py
+-- content.py                # Main contents() strategy (bottom-up tree builder)
+-- leaf.py                   # leaf_contents() — leaf node strategy
+-- numpy_array.py            # numpy_array_contents()
+-- empty_array.py            # empty_array_contents()
+-- regular_array.py          # regular_array_contents()
+-- list_offset_array.py      # list_offset_array_contents()
+-- list_array.py             # list_array_contents()
+-- string.py                 # string_contents()
+-- bytestring.py             # bytestring_contents()
+-- record_array.py           # record_array_contents()
+-- option.py                 # option content strategies (future)
+-- union_array.py            # union_array_contents() (future)
```

### Public API

The `constructors` subpackage is available as a namespace under `strategies/`:

```python
import hypothesis_awkward.strategies as st_ak

st_ak.constructors.arrays()
```

Not re-exported at the `st_ak` top level. The `constructors` module is imported
as a namespace (`from . import constructors`), so it is accessed as
`st_ak.constructors.arrays()`.

## Design Decisions

### 1. Implemented `allow_*` Flags Default to `True`

**Decision:** `allow_regular`, `allow_list_offset`, and `allow_list` default to
`True`. Future flags for unimplemented node types will be added when those node
types are implemented.

**Rationale:**

- Maximizes coverage by default -- `arrays()` with no arguments generates a
  variety of structural layouts.
- Users who want flat arrays can set all flags to `False`.
- New `allow_*` flags will be added (defaulting to `True`) as new node types are
  implemented.

### 2. `max_size` (Total Scalar Budget)

**Decision:** Use `max_size` to control the total number of scalar values across
all leaf `NumpyArray` nodes.

**Rationale:**

- In nested structures, controlling the outermost dimension length alone does
  not bound total array size.
- A scalar budget provides a natural cap that works regardless of nesting depth.
- The "budgeted leaf" approach (closure with `remaining` counter) ensures the
  budget is respected across multiple leaf draws.

### 3. Return `ak.Array`, Not `Content`

**Decision:** Return `ak.Array`.

**Rationale:**

- `ak.Array` is the user-facing type. Tests are written against `ak.Array`.
- `ak.Array(layout)` is zero-cost (no data copy).
- Raw layout is accessible via `result.layout`.
- Matches `from_numpy()` precedent.

### 4. No `allow_structured` Parameter

**Decision:** Omit `allow_structured`. Structured arrays (records) will come
from `allow_record=True`.

**Rationale:**

- In the `numpy_arrays()` -> `from_numpy()` path, structured NumPy dtypes
  produce RecordArray layouts. This was a side-effect of using `ak.from_numpy`.
- In `arrays()`, RecordArray is a first-class node type controlled by
  `allow_record`.
- Mixing structured dtypes with direct constructors would complicate the
  implementation without adding value.

### 5. `dtypes` Accepts Only Strategy or None, Not Plain `np.dtype`

**Decision:** `dtypes` accepts `st.SearchStrategy[np.dtype] | None`, not
`np.dtype | st.SearchStrategy[np.dtype] | None`.

**Rationale:**

- In compound arrays, multiple leaf nodes may need different dtypes.
- Accepting a plain `np.dtype` would fix all leaves to the same dtype, which is
  limiting.
- Users who want a single dtype can use `st.just(np.dtype('float64'))`.
- This can be relaxed later (accepting plain dtype as a convenience) without
  breaking the interface.

**Alternative considered:** Accepting plain `np.dtype` and wrapping in
`st.just()`, as `numpy_arrays(dtype=...)` does. Deferred to keep the initial
interface strict and revisit based on user feedback.

### 6. Separate `constructors/` Directory

**Decision:** Place `arrays()` in `strategies/constructors/`, not in
`strategies/` root or `strategies/numpy/`.

**Rationale:**

- The directory name reflects the approach (direct constructors), not the output.
  This leaves room for alternative approaches later (e.g., `builders/`,
  `from_types/`).
- `arrays()` is the main entry point, not specific to NumPy.
- It will grow to contain multiple internal modules (one per node type).
- Follows the pattern of `types/`, `forms/`, etc.
- `strategies/numpy/` is for the NumPy-specific path (`ak.from_numpy`).

### 7. Add `allow_*` Flags Only When Implemented

**Decision:** Only include `allow_*` flags for node types that are actually
implemented. Currently: `allow_numpy`, `allow_empty`, `allow_string`,
`allow_bytestring` (leaf types), `allow_regular`, `allow_list_offset`,
`allow_list`, `allow_record` (nesting types), and `allow_virtual`.

**Rationale:**

- Avoids dead parameters that have no effect.
- The signature grows as features are added, which is natural for an
  experimental strategy.
- Keyword-only arguments make this a non-breaking change.

## Usage Examples

### Basic Usage

```python
import hypothesis_awkward.strategies as st_ak
from hypothesis import given

@given(a=st_ak.constructors.arrays())
def test_something(a):
    # a is an ak.Array, possibly nested with RegularArray/ListOffsetArray/ListArray
    assert isinstance(a, ak.Array)
```

### Specific Dtype

```python
import numpy as np
from hypothesis import strategies as st
from hypothesis_awkward.util import iter_numpy_arrays

@given(a=st_ak.constructors.arrays(dtypes=st.just(np.dtype('float64'))))
def test_float_arrays(a):
    for leaf in iter_numpy_arrays(a):
        assert leaf.dtype == np.dtype('float64')
```

### Integer Dtypes Only

```python
int_dtypes = st_ak.supported_dtypes().filter(lambda d: d.kind == 'i')

@given(a=st_ak.constructors.arrays(dtypes=int_dtypes))
def test_integer_arrays(a):
    for leaf in iter_numpy_arrays(a):
        assert leaf.dtype.kind == 'i'
```

### Allow NaN

```python
float_dtypes = st_ak.supported_dtypes().filter(lambda d: d.kind == 'f')

@given(a=st_ak.constructors.arrays(dtypes=float_dtypes, allow_nan=True))
def test_nan_handling(a):
    # Test code that must handle NaN correctly
    ...
```

### Control Size

```python
@given(a=st_ak.constructors.arrays(max_size=100))
def test_larger_arrays(a):
    # Total scalar count across all leaves is at most 100
    ...
```

### Flat Arrays Only

```python
@given(a=st_ak.constructors.arrays(
    allow_regular=False, allow_list_offset=False, allow_list=False,
))
def test_flat_arrays(a):
    assert isinstance(a.layout, (ak.contents.NumpyArray, ak.contents.EmptyArray))
```

### Future: With Type Constraint

```python
# Future: generate arrays matching a specific type
@given(a=st_ak.constructors.arrays(type=ak.types.ListType(ak.types.NumpyType('float64'))))
def test_typed_arrays(a):
    assert a.type.content == ak.types.NumpyType('float64')
```

## Testing Plan

Following the patterns in
[testing-patterns.md](./../../.claude/rules/testing-patterns.md):

### `contents()` tests (`tests/strategies/contents/test_content.py`)

The `contents()` strategy has comprehensive property-based and reachability tests
that verify all options are respected: `max_size` (total scalars), per-type
gating (`allow_regular`, `allow_list_offset`, `allow_list`), dtypes via leaf
arrays, `allow_nan`, and `max_depth`. Edge case reachability tests use `find()`
to verify the strategy can produce empty content, NaN values, specific dtypes,
maximum depth, nested structures, and edge cases for each list type.

### `arrays()` tests (`tests/strategies/constructors/test_array.py`)

Since `arrays()` is a thin wrapper around `contents()`, the test mocks
`contents()` and verifies:

1. All kwargs are forwarded to `contents()` (via `assert_called_once_with`)
2. The result is an `ak.Array` wrapping the content returned by `contents()`
   (via identity check on `.layout`)

### File structure for tests

```text
tests/strategies/contents/
+-- __init__.py
+-- test_content.py           # contents() strategy tests
+-- test_empty_array.py       # empty_array_contents() tests
+-- test_numpy_array.py       # numpy_array_contents() tests
+-- test_regular_array.py     # regular_array_contents() tests
+-- test_list_offset_array.py # list_offset_array_contents() tests
+-- test_list_array.py        # list_array_contents() tests
+-- test_string.py            # string_contents() tests
+-- test_bytestring.py        # bytestring_contents() tests
+-- test_record_array.py      # record_array_contents() tests

tests/strategies/constructors/
+-- __init__.py
+-- test_array.py             # arrays() strategy tests (mocks contents())
```

## Alternatives Considered

### Alternative A: Extend `from_numpy()` Instead

Expand `from_numpy()` to support more node types.

**Rejected because:**

- `from_numpy()` is specifically about the `ak.from_numpy()` path.
- Direct constructors exercise a different code path.
- Mixing concerns would make `from_numpy()` confusing.
- A separate `arrays()` strategy has a clearer purpose.

### Alternative B: Start from Forms

Generate `Form` objects first, then use `ak.from_buffers()` to produce arrays.

**Rejected because:**

- Requires buffer generation with complex invariants (sorted offsets, etc.).
- Two-step process (form + buffers) is harder to debug.
- Direct constructors validate at construction time.
- Can be added later as an alternative pipeline if needed.

### Alternative C: Start from Types

Generate `Type` objects, convert to `Form`, then to arrays.

**Rejected because:**

- Three-step pipeline (type -> form -> buffers -> array).
- Most indirection, hardest to debug.
- One type maps to multiple forms; need to resolve ambiguity.
- Can be added later via the `type` parameter.

### Alternative D: `max_length` Instead of `max_size`

Control the outermost dimension length rather than total scalars.

**Rejected because:**

- In nested structures, outermost length alone does not bound total array size.
- A scalar budget (`max_size`) provides a natural cap regardless of nesting depth.
- The "budgeted leaf" approach made `max_size` straightforward to implement.

### Alternative E: Accept Plain `np.dtype` for `dtypes`

Accept `np.dtype | st.SearchStrategy[np.dtype] | None` as in `numpy_arrays()`.

**Deferred because:**

- In compound arrays, different leaves may need different dtypes.
- Accepting a plain dtype implies all leaves share it, which is limiting.
- Can be added later as a convenience without breaking changes.

## Open Questions

1. ~~**Should `allow_empty` be a parameter?**~~ **Resolved.** Yes, `allow_empty`
   is now implemented (default `True`). `EmptyArray` leaves are generated via
   `leaf_contents()` when `allow_empty=True` and `min_size == 0`.

2. **Should `allow_indexed` be a parameter?** `IndexedArray` is type-transparent
   (it does not change the type of its content). It adds a level of indirection
   that exercises different code paths. This is valuable for testing Awkward
   internals but may be confusing for end users. Tentative answer: Yes, add it
   when implementing option types (since `IndexedOptionArray` is the main option
   representation and `IndexedArray` shares its machinery).

3. **When `type`/`form` parameters are added, how do they interact with
   `allow_*` flags?** Tentative answer: When `type` or `form` is given, it fully
   determines the structure and `allow_*` flags are ignored (similar to how
   `numpy_forms(type_=...)` ignores other parameters).

## Completed

1. ~~Implement `arrays()` with NumpyArray-only support~~ ✓
2. ~~Add `RegularArray`, `ListOffsetArray`, `ListArray` support~~ ✓
3. ~~Add scalar budget approach (`max_size`)~~ ✓
4. ~~Add `max_depth` parameter~~ ✓
5. ~~Add tests following the testing plan~~ ✓
6. ~~Export from `strategies/__init__.py`~~ ✓

## Next Steps

1. ~~Add `RecordArray` support (`allow_record`)~~ ✓ — see
   [contents-tree-builder](../impl/2026-02-17-contents-tree-builder.md) and
   [record-array-research](../research/2026-02-17-record-array-research.md)
2. Add option type support (`allow_option`) -- `IndexedOptionArray`,
   `ByteMaskedArray`, `BitMaskedArray`, `UnmaskedArray`
3. ~~Add `UnionArray` support (`allow_union`)~~ ✓
4. ~~Add string/bytestring support (`allow_string`, `allow_bytestring`)~~ ✓ —
   see [string-bytestring-api](./../api/2026-02-13-string-bytestring-api.md)
5. Implement content nesting constraint enforcement
6. Consider connecting `type`/`form` parameters
