# Condensed: Faiss Quantizers

Summary: This tutorial provides implementation details for Faiss quantization techniques, specifically Scalar Quantization and Product Quantization (PQ) for high-dimensional vector compression and similarity search. It demonstrates how to implement basic quantizers and their corresponding indexes, with code examples for both standalone quantizers and index-based approaches. The tutorial covers key parameters like dimension (d), sub-vectors (M), and bits per quantizer (nbits), helping with tasks such as vector compression, nearest neighbor search, and large-scale similarity search optimization. Notable features include memory-efficient vector storage, fast similarity search capabilities, and IVF clustering for improved search performance on large datasets.

*This is a condensed version that preserves essential implementation details and context.*

Here's the condensed tutorial focusing on essential implementation details:

# Faiss Quantizers Implementation Guide

## Setup
```python
import faiss
import numpy as np

# Sample data preparation
data = np.random.random((1000, 128))
```

## 1. Scalar Quantizer

Key concepts:
- Transforms 32-bit floats to smaller representations (e.g., 8-bit integers)
- 4x reduction in size
- Distributes each dimension into 256 buckets

### Implementation Options

1. **Basic Scalar Quantizer**
```python
d = 128
qtype = faiss.ScalarQuantizer.QT_8bit

quantizer = faiss.ScalarQuantizer(d, qtype)
quantizer.train(data)
compressed_data = quantizer.compute_codes(data)
```

2. **Scalar Quantizer Index**
```python
index = faiss.IndexScalarQuantizer(d, qtype, faiss.METRIC_L2)
index.train(data)
index.add(data)
```

## 2. Product Quantizer (PQ)

Key concepts:
- Optimized for speed and memory efficiency
- Divides vectors into sub-vectors
- Uses codebooks with centroids
- Stores centroid indices instead of full vectors

### Implementation Options

1. **Basic Product Quantizer**
```python
d = 128  # dimension
M = 8    # number of sub-vectors
nbits = 4  # bits per subquantizer

quantizer = faiss.ProductQuantizer(d, M, nbits)
quantizer.train(data)
compressed_data = quantizer.compute_codes(data)
```

2. **PQ Index**
```python
index = faiss.IndexPQ(d, M, nbits, faiss.METRIC_L2)
index.train(data)
index.add(data)
```

3. **PQ IVF Index**
```python
nlist = 5  # number of clusters
quantizer = faiss.IndexFlat(d, faiss.METRIC_L2)
index = faiss.IndexIVFPQ(quantizer, d, nlist, M, nbits, faiss.METRIC_L2)
index.train(data)
index.add(data)
```

### Critical Parameters
- `d`: vector dimension
- `M`: number of sub-vectors (must satisfy d % M == 0)
- `nbits`: bits per subquantizer (determines number of centroids: 2^nbits)
- `nlist`: number of clusters for IVF
- `qtype`: quantization type (e.g., QT_8bit)
- `metric`: similarity metric (METRIC_L2 or IP)

### Search Example
```python
k = 3  # number of nearest neighbors
D, I = index.search(data[:1], k)  # Returns distances and indices
```

Best Practices:
1. Choose quantizer based on memory/speed requirements
2. PQ is preferred for memory-critical applications
3. Ensure dimension is divisible by M for PQ
4. Consider using IVF version for large datasets