# SandevistanLR — ECG-Inspired Cyclical Learning Rate

### Official ECG template (P–QRS–T) — reference

![Official ECG P–QRS–T components](https://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/SinusRhythmLabels.svg/1280px-SinusRhythmLabels.svg.png)

*Reference ECG diagram (Public Domain) from **SinusRhythmLabels.svg** (Wikimedia Commons).*

### SandevistanLR waveform (ECG-shaped learning rate)

![SandevistanLR Preview](figures/01_default_normal_x1.png)

![Python](https://img.shields.io/badge/python-3.8%2B-success.svg)
![PyTorch](https://img.shields.io/badge/pytorch-2.x-orange.svg)
![TensorFlow](https://img.shields.io/badge/tensorflow-2.x-orange.svg)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
[![Contributions Welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](CONTRIBUTING.md)

A **physiologically-inspired learning-rate scheduler** where the LR waveform follows a **realistic ECG template (P–QRS–T)** instead of triangle/cosine curves.

Inspired by **CLR**, **1cycle**, and **warm restarts**, SandevistanLR introduces four training modes:

- **normal** — standard ECG rhythm  
- **run** — faster rhythm + exercise-like morphology changes  
- **die** — flatline (very low constant LR)  
- **aed** — flatline + controlled shock spikes (“defibrillation”)  

> **Status:** This project is **actively under development**.  
> The goal is to produce a **research-grade scheduler** with strong empirical evidence across vision + tabular benchmarks, with results packaged for a **conference-ready submission** (e.g., CVPR / NeurIPS / ICLR workshops).

> **Motivation (LLMs & short runs):** SandevistanLR is **step-based** (per-batch updates), which makes it especially promising for **large-scale training and fine-tuning** where runs are often limited to **very few epochs (sometimes only 1)**.  
> In these regimes, per-epoch LR schedules can be too coarse; step-level “heartbeat” cycles provide controlled exploration and stabilization even in short training budgets.

---

## Preset Configurations (Recommended)

SandevistanLR ships with **ready-to-use preset configurations** that cover multiple training behaviors such as:

- warmup → normal → cooldown
- athlete-like training cycles
- intermittent flatlines
- die → aed → recovery protocols
- interval training
- stress-test / arrhythmia-style schedules

### Where to find the presets

- **Code (presets):** [`src/sandevistanlr/configs.py`](src/sandevistanlr/configs.py)  
  This file defines a dictionary of named configs (`CONFIGS`) and their corresponding schedule plans and parameters.
  The preset configurations are intentionally provided as strong starting points.  
SandevistanLR is fully **parameterizable**, allowing users to generate **infinitely many custom schedules** by tuning LR bounds, cycle length, ECG morphology, and mode transitions to match their training budget and stability requirements.


### Visual previews

All preset waveforms have been pre-generated and are available in [`figures/`](figures).

Below are a few previews directly rendered inside the README:

#### Default ECG (normal)
![01 Default ECG (normal)](figures/01_default_normal_x1.png)

#### Warmup → Normal → Cooldown
![02 Warmup Normal Cooldown](figures/02_warmup_normal_cooldown.png)

#### Die → AED → Recovery
![05 Die AED Recovery](figures/05_die_aed_recovery.png)

#### High-energy Run mode
![10 High Energy Run Mode](figures/10_high_energy_run_mode.png)

These images are intended to provide an immediate visual understanding of each schedule’s behavior before integrating it into training.


---


## Install

### Option 1 — PyPI
```bash
pip install sandevistanlr
```

### Option 2 — From source
```bash
git clone https://github.com/takfarine/SandevistanLR.git
```

---

## Usage

### Option 1 — PyTorch (per-step / per-batch)
```python
import torch
from sandevistanlr import SandevistanLR

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

sched = SandevistanLR(
    lr_min=1e-6,
    lr_max=3e-4,
    steps_per_cycle=2000,
    mode="normal",  # normal | run | die | aed
)

global_step = 0
for epoch in range(EPOCHS):
    for batch in loader:
        # update LR per step
        lr = sched(global_step)
        for pg in optimizer.param_groups:
            pg["lr"] = lr

        loss = train_step(batch)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        global_step += 1
```

### Option 2 — TensorFlow / Keras (manual LR assignment)
```python
import tensorflow as tf
from sandevistanlr import SandevistanLR

optimizer = tf.keras.optimizers.AdamW(learning_rate=1e-3)

sched = SandevistanLR(
    lr_min=1e-6,
    lr_max=3e-4,
    steps_per_cycle=2000,
    mode="run",
)

global_step = tf.Variable(0, dtype=tf.int64)

@tf.function
def train_step(x, y):
    # update LR per step
    lr = sched(int(global_step.numpy()))
    optimizer.learning_rate.assign(lr)

    with tf.GradientTape() as tape:
        pred = model(x, training=True)
        loss = loss_fn(y, pred)

    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

    global_step.assign_add(1)
    return loss
```

---

## Code Examples

End-to-end examples are provided under [`code_examples/`](code_examples):

- **PyTorch** — `code_examples/pytorch_demo.ipynb`  
- **TensorFlow / Keras** — `code_examples/tensorflow_demo.ipynb`  
- **Waveform plots** — `code_examples/plot_waveforms.ipynb`  

---

## References

### Learning-rate scheduling
- Leslie N. Smith, **Cyclical Learning Rates for Training Neural Networks**, 2015.  
  https://arxiv.org/abs/1506.01186

- Leslie N. Smith and Nicholay Topin, **Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates**, 2017.  
  https://arxiv.org/abs/1708.07120

- Ilya Loshchilov and Frank Hutter, **SGDR: Stochastic Gradient Descent with Warm Restarts**, 2016.  
  https://arxiv.org/abs/1608.03983

- fastai documentation, **The 1cycle policy**.  
  https://docs.fast.ai/callback.schedule.html

### ECG foundations and original electrocardiography work
- Willem Einthoven, **The String Galvanometer and the Measurement of the Action Currents of the Heart** (Nobel Lecture, 1925).  
  https://www.nobelprize.org/prizes/medicine/1924/einthoven/lecture/  
  PDF: https://www.nobelprize.org/uploads/2018/06/einthoven-lecture.pdf

- M. Rivera-Ruiz et al., **Einthoven’s String Galvanometer: The First Electrocardiograph**, 2008.  
  https://pmc.ncbi.nlm.nih.gov/articles/PMC2435435/

- C. Fisch, **Centennial of the String Galvanometer and the Electrocardiogram**, *Journal of the American College of Cardiology*, 2000.  
  https://www.sciencedirect.com/science/article/pii/S0735109700009761

- Wikimedia Commons (Public Domain), **SinusRhythmLabels.svg** (ECG components diagram).  
  https://commons.wikimedia.org/wiki/File:SinusRhythmLabels.svg

---

## License

Apache License 2.0 — see [LICENSE](LICENSE)
