# ClusterTK - TODO

## Текущий статус: v0.12.0 🚀

**Latest Release:** v0.12.0 - https://pypi.org/project/clustertk/0.12.0/
**Recent Releases:**
- v0.12.0 (2025-10-30) - Algorithm Parameters & Noise Point Tracking
- v0.11.1 (2025-10-30) - SHAP hotfix
- v0.11.0 (2025-10-30) - Smart Feature Selection & Cluster Balance
- v0.10.2 (2025-10-30) - True NumPy vectorization
- v0.10.1 (2025-10-30) - Feature importance memory fix
**GitHub:** https://github.com/alexeiveselov92/clustertk

## ✅ Что завершено

### v0.1.0-v0.1.1 - Базовый функционал
- [x] Полный pipeline кластерного анализа
- [x] Preprocessing (4 модуля)
- [x] Feature selection (2 модуля)
- [x] Dimensionality reduction (PCA, t-SNE, UMAP)
- [x] Clustering (KMeans, GMM)
- [x] Evaluation (метрики + optimal k finder)
- [x] Interpretation (ClusterProfiler)
- [x] Публикация на PyPI

### v0.2.0 - Дополнительные алгоритмы
- [x] HierarchicalClustering (Ward, Complete, Average linkage)
- [x] DBSCANClustering с автоподбором параметров (eps, min_samples)

### v0.3.0 - Visualization & Naming
- [x] 11 visualization функций в 4 модулях
- [x] Интеграция с Pipeline (.plot_*() методы)
- [x] ClusterNamer с 3 стратегиями (top_features, categories, combined)
- [x] Условные импорты для опциональных зависимостей

### v0.3.1-v0.3.5 - Bugfixes & Improvements
- [x] v0.3.1: Hotfix README (plot_cluster_profiles → plot_cluster_heatmap)
- [x] v0.3.2: Исправлена нормализация для малого числа кластеров
- [x] v0.3.3-v0.3.5: Улучшения в visualization модуле

### v0.4.0-v0.4.2 - Visualization Fix & Documentation
- [x] v0.4.0: Попытка исправить дублирование (неудачная)
- [x] v0.4.1: Правильное решение через plt.close(fig)
- [x] v0.4.2: Обновлен Quick Start + навигация в README
- [x] Исследована причина дублирования (pyplot global state)
- [x] Изучен подход seaborn (возвращает Axes, не Figure)
- [x] Добавлены Table of Contents и Quick Links
- [x] Pipeline Components теперь в collapsed section

### v0.5.0 - Export & Reports ✅
- [x] Реализован `export_results()` для CSV формата
- [x] Реализован `export_results()` для JSON формата
- [x] Реализован `export_report()` - HTML отчёты с embedded plots
- [x] Реализованы `save_pipeline()` и `load_pipeline()` методы
- [x] Добавлена зависимость joblib
- [x] Обновлён README с примерами экспорта
- [x] Протестирован весь функционал экспорта

### v0.6.0 - Documentation Structure ✅
- [x] Создана docs/ директория с полной структурой
- [x] Сокращён README с 495 до 196 строк
- [x] Созданы все основные документы:
  - docs/installation.md - установка и требования
  - docs/quickstart.md - быстрый старт за 5 минут
  - docs/user_guide/ - 8 разделов (preprocessing, clustering, evaluation, etc.)
  - docs/examples.md - реальные примеры использования
  - docs/faq.md - часто задаваемые вопросы
  - docs/index.md - главная страница документации
- [x] README теперь содержит только Quick Start и ссылки на docs/
- [ ] TODO (optional): GitHub Pages или MkDocs setup

## ✅ v0.7.0 - Algorithm Comparison (Completed)
- [x] Метод `pipeline.compare_algorithms()` - автоматическое сравнение KMeans/GMM/Hierarchical/DBSCAN
- [x] Таблица с метриками для каждого алгоритма (DataFrame с результатами)
- [x] Рекомендация лучшего алгоритма на основе weighted scoring (40/30/30)
- [x] Визуализация сравнения алгоритмов (plot_algorithm_comparison)
- [x] Тесты функционала (test_comparison.py - все пройдены)
- [x] Документация с примерами (README, clustering.md, examples.md, FAQ)

## ✅ v0.8.0 - HDBSCAN & Test Suite (Completed)

### HDBSCAN Algorithm ✅
- [x] Реализация HDBSCANClustering класса
- [x] Автоподбор параметров (min_cluster_size, min_samples)
- [x] Интеграция в Pipeline
- [x] Добавление в compare_algorithms()
- [x] Документация и примеры

### Test Suite ✅
- [x] Unit tests для preprocessing модулей (missing, scaling)
- [x] Unit tests для clustering алгоритмов (kmeans)
- [x] Unit tests для evaluation (metrics)
- [x] Integration tests для Pipeline (fit, transform, full workflow)
- [x] Pytest infrastructure с pytest.ini
- [x] 39 тестов, 39% coverage (clustering 66-76%, preprocessing 61-69%)
- [x] Fixtures для различных сценариев данных

### Build & Release ✅
- [x] Build packages успешно (wheel + source distribution)
- [x] Packages прошли twine check
- [x] Git tag v0.8.0 создан и pushed
- [x] Version numbers обновлены (setup.py, pyproject.toml, __init__.py)
- [x] **Published to PyPI** ✅ https://pypi.org/project/clustertk/0.8.0/
- [ ] **TODO: Create GitHub Release** (осталось только это!)
  - URL: https://github.com/alexeiveselov92/clustertk/releases/new
  - Tag: v0.8.0
  - Title: v0.8.0 - HDBSCAN Algorithm & Test Suite
  - Description: Major features - HDBSCAN clustering algorithm, comprehensive test suite (39 tests, 39% coverage)
  - Attach: dist/clustertk-0.8.0-py3-none-any.whl, dist/clustertk-0.8.0.tar.gz

## 🎯 Приоритеты для v0.9.0

### HIGH PRIORITY

1. **Enhanced Test Coverage** 🔥
   - [ ] Добавить тесты для outliers, transforms модулей
   - [ ] Тесты для GMM, Hierarchical, DBSCAN, HDBSCAN
   - [ ] Тесты для optimal_k
   - [ ] Тесты для export/report функционала
   - [ ] Тесты для visualization модуля
   - [ ] Цель: coverage >50%

2. **CI/CD Setup** 🔥
   - [ ] GitHub Actions для автоматических тестов
   - [ ] Автоматическая проверка code style (black, flake8)
   - [ ] Coverage reports (codecov integration)
   - [ ] Автоматическая публикация на PyPI при release tag

## ✅ v0.9.0 - Feature Importance & Stability Analysis (Completed!)

### Feature Importance Analysis ✅
   - [x] Permutation importance - measures impact on clustering quality
   - [x] Feature contribution - variance ratio analysis
   - [x] SHAP values integration (optional dependency)
   - [x] Pipeline integration via `analyze_feature_importance()`
   - [x] Documentation examples (interpretation.md, examples.md, faq.md)
   - [x] Tests for feature importance (21 tests, 83% coverage)

### Cluster Stability Analysis ✅
   - [x] Bootstrap resampling implementation
   - [x] Overall stability via pairwise ARI
   - [x] Per-cluster stability scores (pair consistency)
   - [x] Per-sample confidence scores
   - [x] Stable/unstable cluster identification
   - [x] Pipeline integration via `analyze_stability()`
   - [x] Documentation examples (evaluation.md, examples.md)
   - [x] Tests for stability analysis (20 tests, 94% coverage)

### Release ✅
   - [x] Version bump to 0.9.0
   - [x] CHANGELOG.md updated
   - [x] Published to PyPI: https://pypi.org/project/clustertk/0.9.0/
   - [x] Git tag v0.9.0 created
   - [ ] GitHub Release (create manually at https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.9.0)

---

## ✅ v0.10.0 - Stability Analysis Performance Optimization (Completed!)

### Major Performance Improvements ✅
   - [x] **Memory optimization** (~64x reduction for 80k samples)
     - [x] Streaming computation instead of storing all bootstrap results
     - [x] Memory: O(n_samples + window) instead of O(n_samples × n_iterations)
     - [x] 80k samples: from 32+ GB (OOM) → <500 MB
   - [x] **Speed optimization** (100-1000x faster)
     - [x] Replaced nested Python loops with NumPy broadcasting
     - [x] Fast lookup via np.searchsorted() instead of np.where() in loops
     - [x] Adaptive pair sampling for large clusters
     - [x] 80k samples, 20 iterations: ~6 seconds
   - [x] **Sliding window approach**
     - [x] Only keeps last 10 iterations in memory
     - [x] Same statistical validity with fraction of memory
   - [x] **New parameters for tuning**
     - [x] max_comparison_window (default: 10)
     - [x] max_pairs_per_cluster (default: 5000)

### Implementation Details ✅
   - [x] Rewritten ClusterStabilityAnalyzer with streaming methods
   - [x] _update_sample_confidence_streaming() - vectorized updates
   - [x] _update_cluster_stability_streaming() - adaptive sampling
   - [x] _compute_ari_fast() - optimized ARI computation
   - [x] _finalize_cluster_stability() - streaming aggregation
   - [x] Removed old inefficient methods

### Testing & Validation ✅
   - [x] Tested on 1k samples (0.95s, perfect results)
   - [x] Tested on 10k samples (1.22s, perfect results)
   - [x] Tested on 80k samples (6.15s, works without OOM!)
   - [x] Backward compatible API (no breaking changes)

### Release ✅
   - [x] Version bump to 0.10.0 (setup.py, pyproject.toml)
   - [x] CHANGELOG.md updated with detailed technical info
   - [x] TODO.md updated
   - [x] Build and publish to PyPI: https://pypi.org/project/clustertk/0.10.0/
   - [x] Git commit and tag v0.10.0 pushed
   - [ ] Create GitHub Release manually at: https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.10.0
     - Title: "v0.10.0 - Major Performance Optimization for Large Datasets"
     - Copy release notes from commit message or PyPI page
     - Attach: dist/clustertk-0.10.0-py3-none-any.whl, dist/clustertk-0.10.0.tar.gz

---

## ✅ v0.10.1 - Feature Importance Memory Fix (Completed!)

### Critical Fix ✅
   - [x] **Permutation Importance OOM fix**
     - [x] Identified problem: silhouette_score O(n²) pairwise distances
     - [x] 80k samples = 51+ GB memory requirement → OOM
     - [x] Implemented automatic sampling to 10k samples
     - [x] Result: 80k samples works in ~20s instead of OOM
   - [x] **Feature Contribution optimization**
     - [x] Replaced nested loops with vectorized groupby
     - [x] 10x speedup: 0.3s → 0.03s for 80k samples
     - [x] Memory-efficient single-pass computation

### Testing & Validation ✅
   - [x] Tested on 80k samples (works without OOM!)
   - [x] All 21 unit tests pass (76% coverage)
   - [x] Backward compatible API

### Release ✅
   - [x] Version bump to 0.10.1 (setup.py, pyproject.toml)
   - [x] CHANGELOG.md updated
   - [x] Build and publish to PyPI: https://pypi.org/project/clustertk/0.10.1/
   - [x] Git commit and tag v0.10.1 pushed
   - [ ] Create GitHub Release manually at: https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.10.1

---

## ✅ v0.10.2 - True NumPy Vectorization (Completed!)

### Performance Improvement ✅
   - [x] **Feature Contribution True Vectorization**
     - [x] Replaced pandas groupby with pure NumPy bincount
     - [x] Performance: 0.0165s → 0.0134s (1.23x faster on 80k samples)
     - [x] Benefits:
       - True vectorization without hidden loops
       - No pandas DataFrame creation overhead
       - Pure NumPy C-level operations
     - [x] All 21 unit tests pass (77% coverage)

### Technical Details ✅
   - [x] Identified issue: pandas groupby is not true vectorization
   - [x] Implemented np.bincount for cluster statistics computation
   - [x] Benchmarked three approaches: pandas, NumPy list comprehension, NumPy bincount
   - [x] Winner: NumPy bincount (1.23x faster than pandas)

### Release ✅
   - [x] Version bump to 0.10.2 (setup.py, pyproject.toml)
   - [x] CHANGELOG.md updated
   - [x] Build and publish to PyPI: https://pypi.org/project/clustertk/0.10.2/
   - [x] Git commit and tag v0.10.2 pushed
   - [ ] Create GitHub Release manually at: https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.10.2

---

## ✅ v0.11.0 - Smart Feature Selection & Cluster Balance (Completed!)

### Smart Correlation Filter ✅
   - [x] **SmartCorrelationFilter** - intelligent feature selection from correlated pairs
     - [x] Hopkins statistic strategy - keep features with better clusterability
     - [x] Variance ratio strategy - keep features with better separation
     - [x] Backward compatible (mean_corr strategy = old behavior)
     - [x] get_feature_scores() - view clusterability scores
     - [x] get_selection_summary() - detailed selection decisions
   - [x] **Pipeline integration**
     - [x] Added smart_correlation parameter (default: True)
     - [x] Added correlation_strategy parameter (default: 'hopkins')
     - [x] Verbose output shows smart selection reasoning
     - [x] Backward compatible (can disable with smart_correlation=False)

### Cluster Balance Metric ✅
   - [x] **cluster_balance_score()** - measure cluster size distribution
     - [x] Uses normalized Shannon entropy [0, 1]
     - [x] 1.0 = perfectly balanced, ~0 = highly imbalanced
     - [x] Handles DBSCAN noise points (-1 labels)
   - [x] **Integration into evaluation**
     - [x] Added to compute_clustering_metrics() (include_balance=True by default)
     - [x] Added to get_metrics_summary() with quality thresholds
     - [x] Integrated into OptimalKFinder voting (4th metric)
     - [x] Integrated into compare_algorithms() weighted scoring (15% weight)

### Tests ✅
   - [x] test_smart_correlation.py - 9 comprehensive tests (90% coverage)
   - [x] test_cluster_balance.py - 12 comprehensive tests (100% coverage)
   - [x] All 21 new tests pass ✅

### Release ✅
   - [x] Version bump to 0.11.0 (setup.py, pyproject.toml)
   - [x] CHANGELOG.md updated
   - [x] TODO.md updated
   - [x] Build and publish to PyPI: https://pypi.org/project/clustertk/0.11.0/
   - [x] Git commit and tag v0.11.0 pushed
   - [ ] Create GitHub Release manually at: https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.11.0

---

## ✅ v0.11.1 - SHAP Multidimensional Array Fix (Completed!)

### Bug Fix ✅
   - [x] **SHAP ValueError fix**
     - [x] Fixed "Per-column arrays must each be 1-dimensional" error
     - [x] Added flatten() to ensure mean_shap is always 1D
     - [x] Handles both list of arrays and single array from SHAP
     - [x] All 21 unit tests pass (76% coverage)

### Release ✅
   - [x] Version bump to 0.11.1 (setup.py, pyproject.toml)
   - [x] CHANGELOG.md updated
   - [x] Build and publish to PyPI: https://pypi.org/project/clustertk/0.11.1/
   - [x] Git commit and tag v0.11.1 pushed
   - [ ] Create GitHub Release manually at: https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.11.1

---

## ✅ v0.12.0 - Algorithm Parameters & Noise Point Tracking (Completed!)

### Algorithm Parameter Flexibility ✅
   - [x] **clustering_params parameter** - pass custom parameters to any algorithm
     - [x] Added Dict[str, Any] clustering_params to Pipeline.__init__()
     - [x] Parameters override defaults in algorithm initialization
     - [x] Works with all algorithms: kmeans, gmm, hierarchical, dbscan, hdbscan
     - [x] Examples: min_cluster_size for HDBSCAN, eps/min_samples for DBSCAN, n_init for KMeans
   - [x] **Pipeline integration**
     - [x] Updated all algorithm initialization to merge user params with defaults
     - [x] Backward compatible (clustering_params optional, defaults to {})

### Noise Points Tracking ✅
   - [x] **Noise point statistics** - track DBSCAN/HDBSCAN outliers
     - [x] Added n_noise and noise_ratio to metrics_ dict
     - [x] compute_clustering_metrics() filters noise before calculation
     - [x] get_metrics_summary() shows noise statistics with quality assessment
   - [x] **ClusterProfiler updates**
     - [x] Filters noise points before computing profiles
     - [x] Added n_noise_ and noise_ratio_ attributes
     - [x] Updated docstrings and examples
   - [x] **Pipeline updates**
     - [x] print_cluster_summary() shows noise statistics
     - [x] export_report() includes noise in summary and cluster sizes table
     - [x] Noise row highlighted in HTML reports

### Documentation ✅
   - [x] **Updated docs/user_guide/clustering.md**
     - [x] Added "Customizing Algorithm Parameters" section
     - [x] Updated all algorithm sections with clustering_params examples
     - [x] Added noise statistics examples for DBSCAN/HDBSCAN
   - [x] **Updated docs/quickstart.md**
     - [x] Added HDBSCAN example with clustering_params
     - [x] Added noise point statistics examples
   - [x] **Updated docs/examples.md**
     - [x] Enhanced anomaly detection example with noise tracking
     - [x] Added HDBSCAN alternative with custom parameters
     - [x] Updated geographic clustering with noise visualization
   - [x] **Updated docs/faq.md**
     - [x] Added "How do I customize algorithm parameters?" section
     - [x] Added "When should I use clustering_params?" section
     - [x] Added "What are noise points?" section
     - [x] Updated DBSCAN troubleshooting with clustering_params

### Release ✅
   - [x] Version bump to 0.12.0 (setup.py, pyproject.toml)
   - [x] CHANGELOG.md updated
   - [x] TODO.md updated
   - [x] Build and publish to PyPI: https://pypi.org/project/clustertk/0.12.0/
   - [x] Git commit and tag v0.12.0 pushed
   - [ ] Create GitHub Release manually at: https://github.com/alexeiveselov92/clustertk/releases/new?tag=v0.12.0

---

## 🎯 v0.13.0 - Multivariate Outlier Detection (HIGH PRIORITY) 🔥

### Проблема
**Текущая ситуация:**
- K-Means часто создаёт 1 огромный кластер (95%+ данных) + несколько маленьких кластеров
- Silhouette score высокий (~0.9), но результат бесполезный
- Причина: маленькие кластеры = замаскированные outliers
- Текущий `OutlierHandler` работает **univariate** (по каждому признаку отдельно) → не ловит multivariate outliers

### Решение
**MultivariateOutlierDetector** - новый preprocessing модуль

### Архитектура:

```python
class MultivariateOutlierDetector:
    """
    Detect multivariate outliers before clustering.

    Methods:
    - 'isolation_forest' - IsolationForest (лучше для high-dimensional)
    - 'lof' - LocalOutlierFactor (лучше для low-dimensional, плотных данных)
    - 'elliptic_envelope' - EllipticEnvelope (только для нормально распределённых)
    - 'auto' - автовыбор на основе анализа данных

    Actions:
    - 'remove' - удалить outliers
    - 'flag' - пометить для отдельного анализа
    - None - только детекция
    """
```

### Интеграция в Pipeline:

```python
ClusterAnalysisPipeline(
    # Существующий univariate (per-feature)
    handle_outliers='robust',              # IQR/z-score per feature

    # НОВОЕ: multivariate (full space)
    detect_multivariate_outliers='auto',   # NEW!
    multivariate_outlier_action='remove',  # or 'flag' or None
    contamination=0.1,                     # expected outlier ratio
    verbose=True
)
```

### Порядок выполнения:
1. Missing values
2. Scaling (StandardScaler/RobustScaler)
3. **Multivariate outlier detection** ← NEW!
4. PCA (dimensionality reduction)
5. Feature selection (correlation/variance)
6. Clustering

### Auto-режим логика:

```python
def _select_multivariate_method(X, n_samples, n_features):
    """Auto-select best method based on data characteristics."""

    # High-dimensional (many features): IsolationForest
    if n_features > 20:
        return 'isolation_forest'

    # Low-dimensional + dense: LocalOutlierFactor
    elif n_features <= 20 and n_samples > 1000:
        return 'lof'

    # Check if data is approximately normal
    # (can use scipy.stats.normaltest on PCA components)
    if is_approximately_normal(X):
        return 'elliptic_envelope'

    # Default: IsolationForest (most robust)
    return 'isolation_forest'
```

### Задачи:

- [ ] Создать `clustertk/preprocessing/multivariate_outliers.py`
- [ ] Реализовать `MultivariateOutlierDetector` класс
- [ ] Добавить параметры в `ClusterAnalysisPipeline.__init__()`
- [ ] Интегрировать в preprocessing workflow (после scaling, до PCA)
- [ ] Добавить `_select_multivariate_method()` для auto-режима
- [ ] Тесты: test_multivariate_outliers.py
- [ ] Документация: обновить preprocessing.md
- [ ] Примеры использования в quickstart.md и examples.md
- [ ] Обновить CHANGELOG.md

### Ожидаемый результат:
- Улучшенное качество кластеризации (нет доминирующих кластеров)
- Автоматическая детекция multivariate outliers
- Возможность анализа outliers отдельно (action='flag')

---

## 🎯 Приоритеты для v0.14.0+

### MEDIUM PRIORITY

4. **More Clustering Algorithms**
   - [ ] Spectral Clustering
   - [ ] OPTICS (Ordering Points To Identify Clustering Structure)
   - [ ] Mini-Batch K-Means (для больших данных)

### LOW PRIORITY (Backlog)

5. **Advanced Features**
   - [ ] Ensemble clustering (voting между алгоритмами)
   - [ ] Автоматический feature engineering (polynomial, interactions)
   - [ ] Stability analysis (bootstrap resampling)
   - [ ] Time series clustering support

## 📋 Backlog (для будущих версий)

- Полная Sphinx документация с API reference
- Оптимизация производительности для больших данных
- GPU support (cuML integration)
- Incremental/streaming clustering
- Time series clustering support
- Interactive web UI

## 🐛 Известные проблемы

Нет критических проблем в текущей версии.

## 💡 Идеи для улучшения

1. **User Experience:**
   - Прогресс-бар для долгих операций (tqdm)
   - Более информативные error messages
   - Warnings для потенциально проблемных настроек

2. **Performance:**
   - Кэширование промежуточных результатов
   - Параллелизация где возможно
   - Опциональная поддержка Dask для больших данных

3. **Features:**
   - Автоматический feature engineering
   - Anomaly detection внутри кластеров
   - Stability analysis (bootstrap resampling)
   - Cluster evolution tracking (для временных данных)

## 📝 Заметки по разработке

### Приоритизация
- **v0.4.0-v0.4.2** ✅ - исправлено дублирование графиков + улучшена навигация
- **v0.5.0** ✅ - export/reports (CSV, JSON, HTML, save/load)
- **v0.6.0** 🎯 - docs/ структура + algorithm comparison + enhanced analysis
- **v0.7.0** - HDBSCAN/Spectral + базовые тесты + CI/CD
- **v0.8.0+** - стабилизация API + advanced features
- **v1.0.0** - стабильный API, полная документация, production-ready

### Принципы
- Backward compatibility важна
- Опциональные зависимости остаются опциональными
- Код на английском, docstrings обязательны
- Type hints везде

### Для новых фич
Перед добавлением новой фичи проверь:
1. Соответствует ли общей архитектуре?
2. Нужны ли новые зависимости? (минимизировать!)
3. Как это повлияет на API?
4. Есть ли похожее в sklearn? (следовать их паттернам)
