# FASDR: Fast and Simple Dense Retrieval

🚧 WORK IN PROGRESS 🚧

FASDR is a simple and lightweight library for fast and efficient document retrieval. It is designed to be easy to setup and use, built on top of popular, trusted components (`scipy`, `spacy`, `transformers`) to ensure it can be seamlessly integrated into existing projects and "just work". It's especially well suited for small-to-medium corpora, such as retrieval-augmented prompting of FOSS documentation.

### Features

* Fast and efficient dense retrieval using KDTree data structures.
* Simple interface for indexing and searching documents and sentences.
* Support for various file formats and customizable indexing options.
* Integration with the SpaCy and Sentence-BERT libraries for natural language processing and sentence embeddings.

## Installation

First, install `fasdr` via pip:

<!--To do: publish to PyPI-->
```
pip install git+https://github.com/dmarx/fast-and-simple-dense-retrieval
```

Next, download sentence tokenization language model for
spacy:

<!--To do: language agnostic sentencizer? language detection?-->
```
python -m spacy download en_core_web_trf
```

## Quick Start

### Indexing documents

### Quick Start

To get started with FASDR, you can create a `DocumentIndex` object by passing in the root directory containing the documents you want to index:

```python
from fasdr import DocumentIndex

index = DocumentIndex("/path/to/documents")
```

Once you have created the DocumentIndex object, you can search for documents or sentences using the search_documents and search_sentences methods:

```python
# Find the top five documents relevant to the query "climate change"
results = index.search_documents("climate change", k=5)

# Find the top five sentences relevant to the query "climate change"
results = index.search_sentences("climate change", k=5)
```

You can customize the behavior of the DocumentIndex object by specifying options such as the model name and the file extensions to include in the index:

```python
index = DocumentIndex(
    "/path/to/documents",
    model_name="all-MiniLM-L6-v2",
    extensions=[".txt", ".md", ".pdf"]
)
```

## Design

FASDR is designed to be fast and simple, with a focus on ease of use and minimal setup. It uses FAISS for similarity search, which is a highly optimized library for dense vector search, and SpaCy with the Sentence-BERT component for embedding text. The library is built around two main classes:

* `Document`: Represents a single document and its embeddings.  
* `DocumentIndex`: Represents an index of documents and their embeddings.  

`Document` objects are created by passing in the path to the document file, and can be used to search for similar sentences within the document. `DocumentIndex` objects are created by passing in the root directory containing the documents to index, and can be used to search for similar documents or sentences across all the indexed documents.