# IDEA-014: Docs Knowledge Base Tooling for Agent Researchers

**Date:** 2026-03-04
**Source:** user
**Status:** raw

## Idea

Instead of agents doing live WebFetch on every request, build a persistent, searchable knowledge base from official documentation sites (Claude Code, OpenAI Codex, etc.) that researcher agents can query instantly via MCP. Explore and evaluate ready-made solutions before building anything custom.

## Problem

Currently `claude-researcher` and `codex-researcher` do live WebFetch against docs sites on every call. This has several issues:
- Slow — HTTP roundtrip + full page fetch per query
- Brittle — docs URLs change, sites go down, rate limits
- Context-heavy — fetching full pages wastes context window on noise
- No semantic search — agents can only fetch exact URLs, can't search by concept

## What we want

An agent should be able to call something like:
```
search_docs("how does PreToolUse hook work with non-zero exit codes")
```
...and get back exactly the relevant paragraphs with source links — not a full page dump.

## Landscape of solutions

### 1. DeepWiki (deepwiki.com) — zero-effort, already done

DeepWiki is a service by Cognition AI (creators of Devin). It automatically generates an interactive, searchable wiki from any public GitHub repository.

- **How it works**: crawls the repo (code + README + docs), generates architecture diagrams, creates a Q&A interface
- **Access**: `deepwiki.com/owner/repo` — e.g. `deepwiki.com/anthropics/claude-code`
- **For iEvo**: deepwiki may already have `anthropics/claude-code` indexed. Agents could query it via WebFetch on a specific deepwiki URL
- **Limitation**: only covers GitHub repos, not arbitrary docs sites like `code.claude.com/docs`
- **iEvo own wiki**: ievo-ai/cli could get its own deepwiki automatically — useful for onboarding users and external contributors

### 2. Context7 (context7.com) — MCP server for library docs

Context7 is an MCP server that maintains up-to-date, pre-indexed documentation for popular libraries and frameworks.

- **How it works**: you connect the MCP server to your Claude session, then call `get_library_docs("claude-code")` or `resolve-library-id` → get back current, chunked docs
- **MCP integration**: `npx -y @upstash/context7-mcp@latest` — one line to add to `.mcp.json`
- **Key benefit**: docs are always current, maintained by Context7, no crawling needed
- **Check**: whether Claude Code and OpenAI Codex are already in their index — if yes, this is the zero-effort solution
- **URL**: https://context7.com

### 3. Firecrawl — crawl any site → clean markdown

Firecrawl (firecrawl.dev) is a web crawling service designed specifically for LLM ingestion.

- **How it works**: give it a URL or sitemap, it crawls the site and returns clean markdown (strips nav, ads, boilerplate)
- **API**: REST API + Python/JS SDKs
- **MCP server**: `firecrawl-mcp` — already exists, agents can call `crawl`, `scrape`, `search` directly
- **Use for iEvo**: crawl `code.claude.com/docs/en/` and `developers.openai.com/codex/` once → store results → agents search the stored markdown
- **Pricing**: has a free tier; self-hostable open source version available
- **URL**: https://firecrawl.dev

### 4. Crawl4AI — open source Firecrawl alternative

- **How it works**: same idea as Firecrawl but fully open source, runs locally
- **LLM-friendly**: outputs structured markdown, respects robots.txt, handles JS-rendered pages
- **No API cost**: self-hosted, free
- **Python package**: `pip install crawl4ai`
- **URL**: https://github.com/unclecode/crawl4ai

### 5. Full stack: Crawl → Embed → Vector DB → MCP

For maximum control and offline capability:

```
Step 1: Crawl docs site
  Firecrawl / Crawl4AI → markdown chunks

Step 2: Embed chunks
  OpenAI text-embedding-3-small or local model (nomic-embed)
  → float vectors per chunk

Step 3: Store in vector DB
  Chroma (local, zero-infra) or Qdrant (production-ready, Docker)

Step 4: Expose as MCP server
  Custom MCP server: search(query) → top-K chunks + source URLs
  Agents call it like any other tool
```

**Vector DB options:**
- **Chroma** — local, zero setup, Python-native, good for dev/small scale
- **Qdrant** — production-ready, Docker or cloud, better for large corpora and concurrent access
- **sqlite-vec** — SQLite extension, zero infra, good if we want single-file storage

### 6. Exa (exa.ai) — neural web search

Not a knowledge base but worth mentioning: Exa is a neural search engine for the web. Instead of keyword search, it understands semantic queries.

- Agents could call `exa_search("claude code hooks exit code behavior")` and get relevant pages from the live web
- Has an MCP server
- Good complement to a local knowledge base for queries that need current info

## Recommended path for iEvo

**Phase 1 (zero effort, try immediately):**
1. Check if Context7 already has Claude Code and Codex docs indexed → if yes, just add to `.mcp.json`
2. Check deepwiki.com/anthropics/claude-code → if indexed, add as source in `claude-researcher`

**Phase 2 (if Context7 doesn't cover it):**
3. Use Firecrawl MCP to crawl `code.claude.com/docs/en/` and `developers.openai.com/codex/` on-demand
4. Store crawl results as markdown files in `.ievo/knowledge/<source>/`
5. Agents search the local files with Grep — simple, no vector DB needed for small corpora

**Phase 3 (when scale demands it):**
6. Add embeddings + Chroma/sqlite-vec for semantic search
7. Wrap in MCP server, expose `search_docs(query, source?)` tool
8. Update `claude-researcher` and `codex-researcher` to use it

## iEvo's own DeepWiki

Separately from researcher tooling: once ievo-ai/cli is public, it automatically gets a DeepWiki page. This is worth adding to onboarding docs — new contributors and users can explore the codebase interactively at `deepwiki.com/ievo-ai/cli`.

## Related

- IDEA-013: iEvo MCP server for self-documentation
- IDEA-012: OpenAI Codex multi-agent config format integration
- agents: `claude-researcher.md`, `codex-researcher.md`
