# nidx: NucliaDB index

This is the search index component of NucliaDB. It doesn't store the documents itself: it only manages
the indexes needed in order to perform fast searches on them.

Each index is divided into smaller segments that end up stored in object store (S3, GCS, etc.). This allows us to
support fast indexing of documents (create and upload segments) while a background process merges multiple segments
into bigger ones to allow searches to be faster (since they have to hit all segments). The metadata about these
segments is stored in a PostgreSQL database.

## Components

The index is divided in various components that can be deployed and scaled separately:

- **indexer**: Takes documents (received from a NATS consumer), indexing them and uploading the segments.
- **scheduler**: Periodically scans the database to look for segments needing to be merged and schedules jobs.
- **worker**: Executes the merge jobs: download the segments, merge them and upload them again.
- **searcher**: Continually syncs a set of indexes (downloads them to disk) and provides search over them.
- **api**: Provides the gRPC API to manage indexes, provide stats, search...

## Organization

The code is divided into a set of crates:

- **nidx**: The main crate, contains all the code common to all indexes, including all components listed above.
- **Index crates** (nidx_text, nidx_paragraph, nidx_relation, nidx_vector): Specific code for each kind of index.
- **nidx_types**: Common types shared by `nidx` and the index crates.
- **nidx_protos**: Protobuf definitions for the API with the rest of the system.

The consequence of this is that the index crates are very focused on working with a particular index and don't do any
networking or interfacting other than with the nidx crate. The `nidx` crate takes charge of all segment management,
including storing all the files needed/generated by the index crates.

## Development

This is a cargo workspace and can use cargo for building & running as usual. Two notable dependencies are needed for
development:
- For compiling `nidx_protos` we need a protobuf compiler (`protoc`).
- We do compile-time checking of SQL queries with sqlx. We commit `.sqlx` to the repository which avoids doing this
  normally but, if changing the queries, you'll need to run `cargo sqlx prepare
  --all -- --tests` and commit the `.sqlx` dir, which
  contains a cached version of the information needed for compile-time checking. See the sqlx
  [docs](https://github.com/launchbadge/sqlx/blob/main/sqlx-cli/README.md#enable-building-in-offline-mode-with-query)
  for more information about offline mode.
