Algorithm Details
=================

The Adaptive Momentum Gradient Descent (AMGD) algorithm is designed specifically for regularized Poisson regression in high-dimensional settings. It integrates adaptive learning rates, momentum updates, and adaptive soft-thresholding to overcome limitations of existing optimization methods like Adam and AdaGrad.

This section outlines the core components of AMGD along with its pseudocode as described in Algorithm 1 of the paper.

---

Core Components of AMGD
-----------------------

### 1. Adaptive Learning Rate

To prevent premature convergence due to rapidly diminishing learning rates (a known issue with AdaGrad), AMGD uses an adaptive learning rate schedule:

.. math::
    \alpha_t = \frac{\alpha}{1 + \eta \cdot t}

where:
- :math:`\alpha`: Initial learning rate
- :math:`\eta`: Decay factor
- :math:`t`: Iteration number

This ensures that the learning rate decreases gradually over time, allowing for fine-tuning in later stages while still enabling rapid initial progress.

---

### 2. Momentum Updates with Bias Correction

AMGD employs exponential moving averages of past gradients (first moment) and squared gradients (second moment), similar to the Adam optimizer:

.. math::
    m_t &= \zeta_1 \cdot m_{t-1} + (1 - \zeta_1) \cdot g_t \\
    v_t &= \zeta_2 \cdot v_{t-1} + (1 - \zeta_2) \cdot g_t^2

where:
- :math:`m_t`: First moment estimate at iteration :math:`t`
- :math:`v_t`: Second moment estimate at iteration :math:`t`
- :math:`g_t`: Gradient at iteration :math:`t`
- :math:`\zeta_1, \zeta_2`: Momentum decay parameters (typically set to 0.9 and 0.999)

Bias correction is applied to account for zero initialization:

.. math::
    \hat{m}_t = \frac{m_t}{1 - \zeta_1^t}, \quad
    \hat{v}_t = \frac{v_t}{1 - \zeta_2^t}

These corrected estimates are used in the parameter update step.

---

### 3. Adaptive Soft-Thresholding

A key innovation of AMGD is its direct handling of L1 regularization through adaptive soft-thresholding:

.. math::
    \beta_j^{(t+1)} = \text{sign}(\beta_j^{(t)}) \cdot \max\left(|\beta_j^{(t)}| - \alpha_t \cdot \frac{\lambda_1}{|\beta_j^{(t)}| + \varepsilon}, 0\right)

where:
- :math:`\beta_j^{(t)}`: Coefficient value for feature :math:`j` at iteration :math:`t`
- :math:`\lambda_1`: Regularization strength
- :math:`\varepsilon`: Small constant to avoid division by zero (e.g., 0.01)

This mechanism shrinks smaller coefficients more aggressively than larger ones, effectively performing feature selection while maintaining interpretability.

---

Pseudocode for AMGD Algorithm
-----------------------------

Below is the full pseudocode adapted from Algorithm 1 in paper( Adaptive Momentum Gradient Descent.

.. code-block:: text

    Input: Training data (X, y); learning rate α; momentum parameters ζ₁, ζ₂;
           regularization parameters λ₁, λ₂; gradient clipping threshold T;
           tolerance tol; max iterations M; decay rate η; small ϵ
    
    Initialize: β ← β₀, m ← 0, v ← 0, prev_loss ← ∞
    
    for t = 1 to M do
        αₜ ← α / (1 + η·t)  // Adaptive learning rate
        
        linear_pred ← Xβ
        linear_pred ← clip(linear_pred, -20, 20)
        µ ← exp(linear_pred)  // Predicted mean
        
        grad_ll ← Xᵀ(µ - y)  // Gradient of negative log-likelihood
        
        if penalty == 'L1':
            grad ← grad_ll
        elif penalty == 'elasticnet':
            grad ← grad_ll + λ₂ * β
        end if
        
        grad ← clip(grad, T)  // Apply gradient clipping
        
        m ← ζ₁·m + (1 - ζ₁)·grad  // Momentum update
        v ← ζ₂·v + (1 - ζ₂)·(grad)^2  // Squared gradient update
        
        m̂ ← m / (1 - ζ₁^t), v̂ ← v / (1 - ζ₂^t)  // Bias correction
        
        β ← β - αₜ · m̂ / (√v̂ + ϵ)  // Parameter update
        
        if penalty == 'L1' or 'elasticnet':
            denom ← |β| + 0.01
            β ← sign(β) · max(|β| - αₜ·λ₁ / denom, 0)  // Adaptive soft-thresholding
        end if
        
        LL ← poisson_log_likelihood(β, X, y)
        
        if penalty == 'L1':
            reg_pen ← λ₁·∑|βⱼ|
        elif penalty == 'elasticnet':
            reg_pen ← λ₁·∑|βⱼ| + (λ₂/2)·∑βⱼ²
        end if
        
        total_loss ← LL + reg_pen
        
        if |prev_loss - total_loss| < tol:
            break
        end if
        
        prev_loss ← total_loss
    end for
    
    return β  // Optimized coefficients

---

Theoretical Properties
----------------------

### Convergence Guarantees

Under convexity assumptions for the objective function :math:`f(\beta) = -\ell(\beta) + \lambda P(\beta)` (with :math:`P(\beta)` being either L1 or L2 norm), the sequence of iterates :math:`\{\beta^{(t)}\}` generated by AMGD converges to an optimal solution provided that the learning rate satisfies:

.. math::
    \sum_{t=1}^\infty \alpha_t = \infty, \quad \sum_{t=1}^\infty \alpha_t^2 < \infty

This condition holds for the adaptive learning rate schedule used in AMGD.

### Feature Selection Consistency

For L1-regularized Poisson regression, AMGD identifies the optimal feature subset :math:`S^*` that minimizes expected prediction error:

.. math::
    S^* = \arg\min_{S \subseteq \{1,2,\dots,p\}} \mathbb{E}[L(y, f_S(x))] + \alpha |S|

This formulation ensures a balance between model fit and complexity, validating the use of L1 regularization in count data modeling.

---

Conclusion
----------

AMGD combines the strengths of adaptive learning rates, momentum-based acceleration, and proximal thresholding into a unified framework. Its theoretical guarantees and empirical performance make it particularly suitable for high-dimensional sparse modeling tasks such as ecological health prediction, genomic studies, and network analysis involving count data.