# AMGD: Adaptive Momentum Gradient Descent

[![PyPI version](https://badge.fury.io/py/amgd_optimizer.svg)](https://badge.fury.io/py/amgd_optimizer)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![Documentation](https://readthedocs.org/projects/amgd-optimizer/badge/?version=latest)](https://amgd-optimizer.readthedocs.io/)

A Python implementation of **Adaptive Momentum Gradient Descent (AMGD)** for high-dimensional sparse Poisson regression with L1 and Elastic Net regularization. AMGD combines adaptive learning rates with momentum-based updates and specialized soft-thresholding to achieve superior performance in feature selection and optimization efficiency.

## 🚀 Key Features 

- **Optimization** for Poisson regression with automatic feature selection
- **Superior convergence** compared to Adam, AdaGrad, and GLMnet
- **Adaptive soft-thresholding** for effective L1 and Elastic Net regularization
- **High-dimensional support** with excellent scalability (tested up to 1000+ features)
- **Built-in benchmarking** tools for algorithm comparison
- **Extensible framework** supporting custom penalties and GLM families
- **Comprehensive visualization** for convergence analysis and coefficient paths

## 📊 Performance Highlights

Based on extensive empirical evaluation:

- **38% faster convergence** than Adam on average
- **27% better feature selection precision** compared to standard methods
- **Improved sparsity** while maintaining predictive accuracy
- **Robust performance** across different noise levels and data dimensions

## 📦 Installation

### From PyPI

```bash
pip install amgd
```

### From Source

```bash
git clone https://github.com/yourusername/amgd.git 
cd amgd
pip install -e .
```

## ⚡ Quick Start

### Basic Poisson Regression

```python
from amgd.models import PoissonRegressor
from sklearn.model_selection import train_test_split

# Prepare your data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Create and train model
model = PoissonRegressor(
    optimizer='amgd',
    penalty='l1',
    lambda1=0.1,
    max_iter=1000
)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate
score = model.score(X_test, y_test)
print(f"Test score: {score:.4f}")
```

### Use Elastic Net for Grouped Feature Selection

```python
model = PoissonRegressor(
    optimizer='amgd',
    penalty='elasticnet',
    lambda1=0.05,  # L1 penalty
    lambda2=0.05,  # L2 penalty
    max_iter=1000
)
model.fit(X_train, y_train)

# Check sparsity
sparsity = 1 - (np.sum(model.coef_ != 0) / len(model.coef_))
print(f"Sparsity: {sparsity:.2%}")
```

## 📖 Comprehensive Examples

### 1. Ecological Health Analysis

```python
from amgd.benchmarks.datasets import load_ecological_dataset
from amgd.models import PoissonRegressor
from amgd.visualization import plot_coefficient_path

# Load ecological dataset
X, y, feature_names = load_ecological_dataset()

# Hyperparameter tuning
lambda_values = np.logspace(-4, 1, 50)
best_score = float('inf')
best_lambda = None

for lambda_val in lambda_values:
    model = PoissonRegressor(
        optimizer='amgd',
        penalty='l1',
        lambda1=lambda_val
    )
    model.fit(X_train, y_train)
    val_score = -model.score(X_val, y_val)

    if val_score < best_score:
        best_score = val_score
        best_lambda = lambda_val

# Train final model
final_model = PoissonRegressor(
    optimizer='amgd',
    penalty='l1',
    lambda1=best_lambda,
    max_iter=1000
)
final_model.fit(X_train, y_train)

# Visualize coefficient paths
plot_coefficient_path(
    lambda_values,
    coefficients,
    feature_names=feature_names,
    title="Coefficient Paths for Biodiversity Prediction"
)
```

### 2. Algorithm Comparison

```python
from amgd.benchmarks import compare_optimizers

results = compare_optimizers(
    X, y,
    optimizers=['amgd', 'adam', 'adagrad'],
    penalties=['l1', 'elasticnet'],
    cv_folds=5,
    test_size=0.2
)

print(results['test_results'])
```

### 3. Custom Regularization

```python
from amgd.core.penalties import PenaltyBase
from amgd.models import GLM

class AdaptiveLassoPenalty(PenaltyBase):
    def __init__(self, lambda1, weights):
        self.lambda1 = lambda1
        self.weights = weights

    def __call__(self, x):
        return self.lambda1 * np.sum(self.weights * np.abs(x))

    def proximal_operator(self, x, step_size):
        threshold = self.lambda1 * self.weights * step_size
        return np.sign(x) * np.maximum(np.abs(x) - threshold, 0)

model = GLM(
    optimizer='amgd',
    family='poisson',
    link='log'
)
```

## 🧮 Algorithm Details

AMGD integrates three key methods:

### 1. Adaptive Learning Rate Decay
```
αₜ = α / (1 + ηt)
```

### 2. Momentum Updates with Bias Correction
```
mₜ = ζ₁mₜ₋₁ + (1 - ζ₁)∇f(βₜ)
vₜ = ζ₂vₜ₋₁ + (1 - ζ₂)(∇f(βₜ))²
m̂ₜ = mₜ / (1 - ζ₁ᵗ)
v̂ₜ = vₜ / (1 - ζ₂ᵗ)
```

### 3. Adaptive Soft-Thresholding
```
βⱼ ← sign(βⱼ) · max(|βⱼ| - αₜλ/(|βⱼ| + ε), 0)
```
## 🛠️ Advanced Features

### Warm Starting

```python
model = PoissonRegressor(
    optimizer='amgd',
    warm_start=True,
    max_iter=100
)
model.fit(X_train, y_train)
model.max_iter = 500
model.fit(X_train, y_train)
```

### Custom Convergence Criteria

```python
from amgd.core.convergence import RelativeChangeCriterion

criterion = RelativeChangeCriterion(tol=1e-8, patience=5)
optimizer = AMGDOptimizer(convergence_criterion=criterion)
```

### Visualization Tools

```python
from amgd.visualization import plot_convergence, plot_feature_importance

plot_convergence(model.loss_history_, log_scale=True)
plot_feature_importance(
    model.coef_,
    feature_names=feature_names,
    top_k=20
)
```

## 🔧 API Reference

### Core Classes

- `AMGDOptimizer`: Main optimization algorithm
- `PoissonRegressor`: Poisson regression with AMGD
- `GLM`: General framework for GLMs with various families

### Key Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| alpha     | 0.01    | Initial learning rate |
| beta1     | 0.9     | First moment decay rate |
| beta2     | 0.999   | Second moment decay rate |
| lambda1   | 0.0     | L1 regularization strength |
| lambda2   | 0.0     | L2 regularization strength |
| T         | 20.0    | Gradient clipping threshold |
| eta       | 0.0001  | Learning rate decay factor |

## 📚 Documentation

Full documentation is available at [https://amgd.readthedocs.io](https://amgd.readthedocs.io)

- Installation Guide
- Tutorial
- API Reference
- Theory & Algorithm

## 🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

```bash
# Run tests
pytest tests/

# Run linting
flake8 src/

# Build documentation
cd docs && make html
```

## 📝 Citation

If you use AMGD in your research, please cite our paper:



```bibtex
@article{bakari2025amgd,
  title={Adaptive Momentum Gradient Descent Algorithm: A New Algorithm in Regularized Poisson Regression},
  author={Bakari, Ibrahim and Özkale, M. Revan},
  journal={Journal Name},
  year={2025}
  volume={XX},
  number={X},
  pages={XXX-XXX},
  doi={10.XXXX/XXXXX}
}
```

## 📄 License

This project is licensed under the MIT License 

## 🙏 Acknowledgments

- Inspired by advances in adaptive optimization methods
- Built on top of NumPy, SciPy, and scikit-learn
- **Çukurova University** - Department of Statistics
- **Research Community** - For valuable feedback and suggestions

## 📧 Contact

**Author**: Ibrahim Bakari  
**Email**: acbrhmbakari@gmail.com  
**Issues**: [GitHub Issues](https://github.com/elbakari01/amgd/issues)





