fm_index API documentation

# fm_index

class FMIndex:

An FM-index for efficient full-text search on a single string.

The FM-index is a compressed text index based on the Burrows–Wheeler Transform (BWT).
It supports fast substring queries whose runtime depends only on the pattern length,
not on the size of the indexed text.
Internally, several independent stages of index construction and query processing
are optimized using data-parallel execution.

### Construction

#### Time / Space Complexity

- Time: `O(N)`
- Space: `O(N)`

where:

- `N` = length of the indexed string

    from fm_index import FMIndex

    fm = FMIndex("mississippi")

### Serialization

FMIndex supports Python's pickle protocol for efficient persistence:

    import pickle

    # Save index
    with open("index.pkl", "wb") as f:
        pickle.dump(fm, f)

    # Load index
    with open("index.pkl", "rb") as f:
        fm = pickle.load(f)

FMIndex(data: str)
def item(self) -> str:

Convert the FMIndex back into the original string.

#### Complexity

- Time: `O(N)`
- Space: `O(N)`

#### Examples

    fm.item()
    # 'mississippi'

def contains(self, pattern: str) -> bool:

Check whether the indexed string contains the given pattern.

#### Complexity

- Time: `O(|pattern|)`
- Space: `O(|pattern|)`

#### Examples

    fm.contains("issi")
    # True

def count(self, pattern: str) -> int:

Count how many times a pattern appears in the indexed string.

#### Complexity

- Time: `O(|pattern|)`
- Space: `O(|pattern|)`

#### Examples

    fm.count("issi")
    # 2

def locate(self, pattern: str) -> list[int]:

Locate all starting positions of the pattern in the indexed string.
This operation may internally leverage parallel execution to efficiently
enumerate large result sets.
⚠️ Order of returned positions is not guaranteed.

#### Complexity

- Time: `O(|pattern| + |count|)`
- Space: `O(|pattern| + |count|)`

#### Examples

    fm.locate("issi")
    # [4, 1]

def iter_locate(self, pattern: str) -> Generator[int, NoneType, NoneType]:

Lazily locate all starting positions of the pattern in the indexed string.

This method yields the same positions as `locate`, but returns them one by one as an iterator instead of allocating a list.

⚠️ Order of yielded positions is not guaranteed.

#### Complexity

- Time: `O(|pattern|)` for initialization, `O(1)` per yielded position
- Space: `O(|pattern|)`

#### Examples

    iter = fm.iter_locate("issi")
    next(iter)
    # 4
    next(iter)
    # 1

def startswith(self, prefix: str) -> bool:

Check if the indexed string starts with the given prefix.

#### Complexity

- Time: `O(|prefix|)`
- Space: `O(|prefix|)`

#### Examples

    fm.startswith("mi")
    # True

def endswith(self, suffix: str) -> bool:

Check if the indexed string ends with the given suffix.

#### Complexity

- Time: `O(|suffix|)`
- Space: `O(|suffix|)`

#### Examples

    fm.endswith("pi")
    # True

class MultiFMIndex:

A multi-document FM-index for fast substring search across multiple strings.

Internally, all strings are concatenated with separators and indexed as a single FM-index, while preserving the ability to map matches back to their original documents. Query processing across documents is internally parallelized where applicable, making multi-document search efficient in practice.

### Construction

#### Time / Space Complexity

- Time: `O(S + D log D)`
- Space: `O(S + D log D)`

where:

- `D` = number of indexed strings
- `S` = total length of all indexed strings

    from fm_index import MultiFMIndex

    mfm = MultiFMIndex(["abcabcabcabc", "xxabcabcxxabc", "abcababcabc"])

### Serialization

MultiFMIndex supports Python's pickle protocol for efficient persistence:

    import pickle

    # Save index
    with open("index.pkl", "wb") as f:
        pickle.dump(mfm, f)

    # Load index
    with open("index.pkl", "rb") as f:
        mfm = pickle.load(f)

MultiFMIndex(data: Iterable[str])
def item(self) -> list[str]:

Convert the index back into the original list of strings.

#### Complexity

- Time: `O(S)`
- Space: `O(S)`

#### Examples

    mfm.item()
    # ['abcabcabcabc', 'xxabcabcxxabc', 'abcababcabc']

def contains(self, pattern: str) -> bool:

Check if the pattern exists as a full document in the index.

#### Complexity

- Time: `O(|pattern|)`
- Space: `O(|pattern|)`

#### Examples

    mfm.contains("abcabcabcabc")
    # True

def count_all(self, pattern: str) -> int:

Count total occurrences of a pattern across all documents.

#### Complexity

- Time: `O(|pattern|)`
- Space: `O(|pattern|)`

#### Examples

    mfm.count_all("abc")
    # 10

def count(self, pattern: str, doc_id: int | None = None) -> dict[int, int] | int:

Count occurrences per document or within a specific document.

#### Complexity

When `doc_id` is None:

- Time: `O(|pattern| + |output| log D)`
- Space: `O(|pattern| + |output|)`

When `doc_id` is specified:

- Time: `O(|pattern| + log D)`
- Space: `O(|pattern|)`

#### Examples

    # Count across all documents (returns {doc_index: count})
    mfm.count("abc")
    # {0: 4, 1: 3, 2: 3}

    # Count within a specific document (returns int)
    mfm.count("abc", doc_id=0)
    # 4

def topk(self, pattern: str, k: int) -> list[tuple[int, int]]:

Return the top-k documents with the highest number of occurrences of the given pattern.

#### Complexity

- Time: `O(|pattern| + |total_count| log |total_count| log D)`
- Space: `O(|pattern| + k)`

#### Examples

    # Get top 2 documents containing the pattern
    mfm.topk("abc", k=2)
    # [(0, 4), (1, 3)]

    # If fewer than k documents match, all matching documents are returned
    mfm.topk("abc", k=5)
    # [(0, 4), (1, 3), (2, 3)]

def locate( self, pattern: str, doc_id: int | None = None) -> dict[int, list[int]] | list[int]:

Locate occurrences per document or within a specific document. Internally, result enumeration and aggregation may be parallelized. ⚠️ Order is not guaranteed.

#### Complexity

When `doc_id` is None:

- Time: `O(|pattern| + |total_count| log D)`
- Space: `O(|pattern| + |total_count|)`

When `doc_id` is specified:

- Time: `O(|pattern| + |count_in_doc| log D)`
- Space: `O(|pattern| + |count_in_doc|)`

#### Examples

    # Locate across all documents (returns {doc_index: [positions]})
    mfm.locate("abc")
    # {0: [9, 6, 3, 0], 1: [10, 2, 5], 2: [8, 0, 5]}

    # Locate within a specific document (returns list[int])
    mfm.locate("abc", doc_id=0)
    # [9, 6, 3, 0]

def iter_locate( self, pattern: str, doc_id: int | None = None) -> Generator[tuple[int, int], NoneType, NoneType]:

Lazily locate all occurrences of the pattern across documents or within a specific document.

Yields `(doc_index, position)` pairs without constructing an intermediate result dictionary.

⚠️ Order of yielded results is not guaranteed.

### Complexity

When `doc_id` is None:

- Time: `O(|pattern|)` to initialize, then `O(log D)` per yielded occurrence.
- Space: `O(|pattern|)`

When `doc_id` is specified:

- Time: `O(|pattern| + log D)` to initialize, then `O(log D)` per yielded occurrence within the document.
- Space: `O(|pattern|)`

#### Examples

    # Iterate across all documents
    iter = mfm.iter_locate("abc")
    next(iter)
    # (2, 8)
    next(iter)
    # (1, 10)

    # Iterate within a specific document
    iter = mfm.iter_locate("abc", doc_id=0)
    next(iter)
    # (0, 9)
    next(iter)
    # (0, 6)
    ...

def startswith(self, prefix: str) -> list[int]:

List document indices whose content starts with the prefix.

#### Complexity

- Time: `O(|prefix| + |output|)`
- Space: `O(|prefix| + |output|)`

#### Examples

    mfm.startswith("abc")
    # [2, 0]

def endswith(self, suffix: str) -> list[int]:

List document indices whose content ends with the suffix.

#### Complexity

- Time: `O(|suffix| + |output| log D)`
- Space: `O(|suffix| + |output|)`

#### Examples

    mfm.endswith("abc")
    # [2, 1, 0]