Metadata-Version: 2.3
Name: zmp-knowledge-store-mcp-server
Version: 0.2.6
Summary: MCP Server for zmp knowledge base
Author: stiron
Author-email: lesch1@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: accelerate (>=1.7.0,<2.0.0)
Requires-Dist: boto3 (>=1.38.24,<2.0.0)
Requires-Dist: chromadb (>=1.0.11,<2.0.0)
Requires-Dist: docling (>=2.36.1,<3.0.0)
Requires-Dist: docling-core (>=2.32.0,<3.0.0)
Requires-Dist: fastapi (>=0.110.0,<1.0.0)
Requires-Dist: fastmcp (>=2.6.1,<3.0.0)
Requires-Dist: httpx (>=0.28.1,<0.29.0)
Requires-Dist: huggingface-hub (>=0.32.2,<0.33.0)
Requires-Dist: langchain (>=0.3.25,<0.4.0)
Requires-Dist: llama-index (>=0.12.48,<0.13.0)
Requires-Dist: markdown (>=3.8,<4.0)
Requires-Dist: mcp[cli] (>=1.9.1,<2.0.0)
Requires-Dist: numpy (<2.0.0)
Requires-Dist: openai (>=1.88.0,<2.0.0)
Requires-Dist: pdf2image (>=1.17.0,<2.0.0)
Requires-Dist: pillow (>=11.2.1,<12.0.0)
Requires-Dist: protobuf (>=3.20,<4.0)
Requires-Dist: pymupdf (>=1.26.0,<2.0.0)
Requires-Dist: python-dotenv (>=1.1.0,<2.0.0)
Requires-Dist: qdrant-client (<1.14.2)
Requires-Dist: ranx (>=0.3.20,<0.4.0)
Requires-Dist: sentence-transformers (>=4.1.0,<5.0.0)
Requires-Dist: torch (>=2.7.0,<3.0.0)
Requires-Dist: transformers (>=4.52.3,<5.0.0)
Requires-Dist: uvicorn (>=0.34.2,<0.35.0)
Description-Content-Type: text/markdown

# ZMP Knowledge Store MCP Server

![Platform Badge](https://img.shields.io/badge/platform-zmp-red)
![Component Badge](https://img.shields.io/badge/component-knowledge_store_mcp_server-red)
![CI Badge](https://img.shields.io/badge/ci-github_action-green)
![License Badge](https://img.shields.io/badge/license-MIT-green)

A high-performance backend service for managing knowledge store content, supporting multi-tenant ChromaDB integration, real-time progress tracking, and secure authentication.

---

## Python File Structure

```
zmp_knowledge_store/
├── __init__.py              # Package initialization & metadata
├── config.py                # Configuration management
├── knowledge_store.py       # Main knowledge store logic (ingestion, search, backend selection)
├── qdrant_adapter.py        # Qdrant vector DB integration
├── chromadb_adapter.py      # ChromaDB vector DB integration
├── keyword_extractor.py     # Advanced keyword extraction
├── utils.py                 # Utility functions

tests/
├── test_client.py           # Test client for validation
├── test_mcp_ingest_setup_a_project.py      # Ingestion & search test for setup-a-project example
├── test_mcp_ingest_group_management.py     # Ingestion & search test for group-management example
├── ...                      # Other tests (see tests/ directory)

examples/
├── zcp/getting-started/setup-a-project/
│   ├── setup-a-project.mdx
│   └── *.png (all referenced images)
├── zcp/advanced-guide/security-and-access-control/role-group-and-permissions/
│   ├── group-management.mdx
│   └── *.png (all referenced images)
```

---

## Key Features in the Source Code

### 1. Unified Knowledge Store (`knowledge_store.py`)
- Handles ingestion, chunking, and search for solution documentation
- Supports both Qdrant and ChromaDB as vector stores
- Multi-backend SmolDocling model support (MLX for Apple Silicon, Transformers for cross-platform)
- Robust image and page number assignment logic for MDX/PDF ingestion
- Metadata enrichment and error handling

### 2. Configuration Management (`config.py`)
- Environment-based configuration
- Embedding model and backend selection
- Default values and cluster mode support

### 3. Vector Store Adapters
- `qdrant_adapter.py`: Qdrant integration (search, upsert, collection management)
- `chromadb_adapter.py`: ChromaDB integration (search, upsert, collection management)

### 4. Keyword Extraction (`keyword_extractor.py`)
- Multiple extraction methods (KeyBERT, spaCy, NLTK, custom)
- Solution-specific optimization (ZCP, AMDP, APIM)
- Adaptive keyword counts (5-12 based on content)
- Domain vocabulary and technical term boosting

### 5. Utilities (`utils.py`)
- Chunking, document creation, and helper functions

### 6. Tests & Example Data
- Comprehensive test suite in `tests/` (pytest-based)
- Example ingestion sets in `examples/` for both setup-a-project and group-management

---

## MCP Tools Implemented

| Tool Name           | Request Schema                | Response Schema                        | Description |
|---------------------|------------------------------|----------------------------------------|-------------|
| ingest_documents    | `{documents: list, solution?: str}`   | `{ success: bool, results: list, total_page_count?: int }`              | Ingest documents with metadata and keyword extraction. |
| search_knowledge    | `{query: str, n_results?: int}`       | `{ query: str, results: list }`    | Search the knowledge store for relevant info. |
| log_chat_history    | `{query: str, response: str, user_id?: str, session_id?: str}` | `{ success: bool, id?: str, error?: str }` | Log a user query/response pair to chat history. Deduplication by (query, user_id). |
| search_chat_history | `{query: str, user_id?: str, n_results?: int}` | `{ query: str, user_id?: str, results: list }` | Hybrid search over chat history, filtered by clustering and semantic similarity. |

---

## Usage Examples

```python
# Ingest documents (see examples/ for sample MDX and images)
result = await client.call_tool("ingest_documents", {
    "documents": [...],
    "solution": "zcp"
})

# Search knowledge
result = await client.call_tool("search_knowledge", {
    "query": "Group Management",
    "n_results": 3
})

# Log chat history
result = await client.call_tool("log_chat_history", {
    "query": "What is a group?",
    "response": "A group is ...",
    "user_id": "user1"
})

# Search chat history
result = await client.call_tool("search_chat_history", {
    "query": "Delete a group",
    "n_results": 5
})
```

---

## Example Ingestion Sets

- `examples/zcp/getting-started/setup-a-project/`: MDX and images for "Setup a Project" documentation
- `examples/zcp/advanced-guide/security-and-access-control/role-group-and-permissions/`: MDX and images for "Group Management" documentation

---

## Running the Tests

- All tests are in the `tests/` directory and use `pytest`.
- Key ingestion and search tests:
  - `test_mcp_ingest_setup_a_project.py`: Ingests and verifies the setup-a-project example
  - `test_mcp_ingest_group_management.py`: Ingests and verifies the group-management example
- To run all tests:

```bash
pytest tests/
```

---

## Troubleshooting: Embedding Dimension Mismatch

If you see errors like:

```
Collection expecting embedding with dimension of 384, got 768
```

- This means your embedding model (e.g., `all-mpnet-base-v2`) outputs a different dimension than the collection was created with (e.g., `all-MiniLM-L6-v2`).
- **Solution:** Delete the ChromaDB collection and re-ingest, or update your config to match the expected dimension.

---

## SmolDocling Backend Selection & Platform Compatibility

- The backend for SmolDocling is controlled by the `SMOLDOCLING_BACKEND` environment variable:
  - `mlx`: Use MLX backend (Apple Silicon only)
  - `transformers`: Use HuggingFace Transformers backend (cross-platform)
  - `auto`: Auto-select based on platform
- MLX backend is recommended for Apple Silicon (macOS); Transformers backend is recommended for Linux/Kubernetes or non-Apple hardware.
- See `PLATFORM_COMPATIBILITY.md` for more details.

---

## Environment Configuration

Set the following environment variables in your `.env` file or deployment environment:

(See previous section for full table of variables)

---

## Deployment Guide

### Installation

(Instructions TBD)

