# TODO

## Bou:

#### Features
Choose among these features the one you are most confident about getting done in a week. All should be implemented using the llama server at https://amd16.tail9f615d.ts.net/ and the meilisearch index on amd16 (not yet publicly available)
- [ ] implement a `def walk_path` function that recursively walks a directory tree and uses `rdrag.util.make_chunks()` to return a list of dictionaries (chunks of all the md and txt files it finds) -- then run add_docs() on that list of dictionaries to add them to the meilisearch default index
- [ ] run add_wiki_pages(pages) 
- [ ] add 1000+ wikipedia pages to an meilisearch index named "wikipedia" (walking the 'links' attribute of a wikipedia.WikipediaPage instance)
- [ ] finish implementing `rdrag.util.search()` such that it can answer a simple question based on 2 or 3 paragraphs from the wikipedia articles 
- [ ] add docstrings and/or doctests (where practical) to all rdrag.util.* functions
- [ ] Read _Retrieval Augmented Generation_ ch 4-5 PDF (will send you draft textbook from Manning by Benjamin Auffarth)

#### Details

- [ ] tailscale up
- [ ] play with llamacpp server at https://amd16.tail9f615d.ts.net/
- [ ] ssh user@amd16 -P $PASSWORD_STARTS_WITH_OK
- [ ] ssh-copy-id user@amd16  # 

- [ ] change base_url to https://amd16.tail9f615d.ts.net/v1
- [ ] test it in ragflow
- [ ] clone gitlab.com:tangibleai/rdrag && cd rdrag
- [ ] source .workon  # creates venv and install rdrag
- [ ] cd tests and look at test_llamacpp.py
- [ ] util.add_wiki_pages()
- [ ] use rdrag.util.search()
- [ ] integrate llama into the rdrag.util.ask()
 
## Hobs:

#### Features
0. upgrade meilisearch to avoid .facets=true error in meilisearch python api and pin python version to working meilisearch version
1. implement llama vector store in meilisearch
2. compare meili keyword-only search to _semantic (vector) search_
3. compare meili keyword-only search to _hybrid search_
4. tune the keyword-only search with synonyms, stopwords and _hybrid search_
5. implement FAIS index for semantic search using same llama vectors from 1.
6. compare FAIS to keyword-only search and hybrid search in meilisearch 
7. implement public reverse proxy in nginx an amd16 port 443 so that both meilisearch and llamacpp are accessible publicly (or at least by Bou and Glenn on the tailnet)
8. add wikipedia page table parsing/chunking to nlpia2-wikipedia
9. add markdown output to wikipedia page object in nlpia2-wikipedia
