Migration Guide: v0.3.6 Defense Layers
=======================================

.. versionadded:: 0.3.6

This guide helps you understand the behavioral changes introduced by the 4-layer
defense strategy in v0.3.6 and how to adapt your code if needed.

Overview of Changes
-------------------

Version 0.3.6 introduces a **4-layer defense strategy** for the Adaptive Hybrid
Streaming Optimizer (``method='hybrid_streaming'``) that prevents Adam warmup
divergence when initial parameters are already near optimal.

**The good news**: Defense layers are beneficial for stability and are **enabled
by default**. Most users require **no code changes**.

**The heads-up**: You may observe different behavior if:

1. Your initial parameters are already very close to optimal
2. You're running warm-start or refinement workflows
3. You're monitoring iteration counts or loss trajectories

What Changed?
-------------

Behavior Changes
~~~~~~~~~~~~~~~~

If you're using ``method='hybrid_streaming'`` in ``curve_fit()``:

1. **Warm Start Optimization** (Layer 1)

   - **Before 0.3.6**: Adam warmup always ran for configured iterations
   - **After 0.3.6**: If initial loss < 1% of data variance, Adam warmup is **skipped**
   - **Impact**: Fewer iterations, faster convergence, better stability

2. **Automatic Learning Rate Adjustment** (Layer 2)

   - **Before 0.3.6**: Fixed learning rate (0.001)
   - **After 0.3.6**: LR automatically selected based on initial loss quality:

     - Excellent fit (< 10% variance): LR = 1e-6 (ultra-conservative)
     - Good fit (10-100% variance): LR = 1e-5 (conservative)
     - Poor fit (≥ 100% variance): LR = 0.001 (exploration)

   - **Impact**: More conservative updates near optimum, preventing divergence

3. **Cost Guard Protection** (Layer 3)

   - **Before 0.3.6**: No protection against cost increases
   - **After 0.3.6**: Warmup aborts if loss increases > 5% from initial
   - **Impact**: Prevents divergence, returns best parameters found

4. **Step Clipping** (Layer 4)

   - **Before 0.3.6**: No limit on Adam update magnitude
   - **After 0.3.6**: Updates clipped to max norm of 0.1
   - **Impact**: Prevents large jumps that overshoot optimum

Compatibility
~~~~~~~~~~~~~

.. list-table::
   :widths: 40 60
   :header-rows: 1

   * - API Component
     - Compatibility
   * - ``curve_fit()`` signature
     - ✅ 100% backward compatible
   * - ``HybridStreamingConfig`` defaults
     - ⚠️ Changed (defense layers enabled)
   * - Return values (popt, pcov)
     - ✅ 100% compatible
   * - Telemetry functions
     - ✅ New (opt-in)
   * - Preset methods
     - ✅ New (opt-in)

Code Migration Patterns
------------------------

No Changes Required (Default)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Most users don't need to change anything**. Defense layers improve stability:

.. code-block:: python

    from nlsq import curve_fit

    # This continues to work exactly as before
    popt, pcov = curve_fit(model, x, y, p0=initial_guess, method="hybrid_streaming")

    # Defense layers are automatically enabled for better stability

Opt-Out: Disable Defense Layers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you need the pre-0.3.6 behavior (e.g., for regression testing):

.. code-block:: python

    from nlsq import curve_fit, HybridStreamingConfig

    # Completely disable all defense layers
    config = HybridStreamingConfig.defense_disabled()

    popt, pcov = curve_fit(
        model, x, y, p0=initial_guess, method="hybrid_streaming", config=config
    )

Customize Defense Sensitivity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Fine-tune defense layers for your use case:

.. code-block:: python

    from nlsq import HybridStreamingConfig

    # Option 1: Use preset for your scenario
    config = HybridStreamingConfig.defense_strict()  # Near-optimal
    config = HybridStreamingConfig.defense_relaxed()  # Exploration
    config = HybridStreamingConfig.scientific_default()  # Physics models

    # Option 2: Customize individual layers
    config = HybridStreamingConfig(
        # Layer 1: Relax warm start threshold
        warm_start_threshold=0.001,  # Trigger less often
        # Layer 2: Increase learning rates
        warmup_lr_refinement=1e-5,
        warmup_lr_careful=1e-4,
        # Layer 3: Allow more cost increase
        cost_increase_tolerance=0.2,  # 20% instead of 5%
        # Layer 4: Allow larger steps
        max_warmup_step_size=0.5,
    )

    popt, pcov = curve_fit(model, x, y, method="hybrid_streaming", config=config)

Monitor Defense Layer Activations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Track which layers are triggering in production:

.. code-block:: python

    from nlsq import curve_fit, get_defense_telemetry, reset_defense_telemetry

    # Reset telemetry before monitoring session
    reset_defense_telemetry()

    # Run batch of fits
    for dataset in datasets:
        popt, pcov = curve_fit(model, x, y, p0=guess, method="hybrid_streaming")

    # Get telemetry report
    telemetry = get_defense_telemetry()
    print(telemetry.get_summary())

    # Check activation rates
    rates = telemetry.get_trigger_rates()
    if rates["layer1_warm_start_rate"] > 80:
        print("Warning: Warm start triggering frequently")
        print("Consider lowering warm_start_threshold")

Common Scenarios
----------------

Scenario 1: Warm Start Refinement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Situation**: Refining parameters from a previous fit.

**Pre-0.3.6 Issue**: Adam warmup would push away from good initial guess.

**0.3.6 Solution**: Layer 1 automatically skips warmup.

.. code-block:: python

    # Initial fit
    popt_v1, _ = curve_fit(model, x1, y1, p0=[1.0, 2.0])

    # Refinement with new data (0.3.6 automatically uses strict defense)
    popt_v2, pcov = curve_fit(
        model, x2, y2, p0=popt_v1, method="hybrid_streaming"  # Good initial guess
    )

    # Layer 1 will skip warmup since popt_v1 is already near optimal

**Expected behavior**:

- Fewer iterations (warmup skipped)
- Better final accuracy (no divergence from good guess)
- Faster overall time

Scenario 2: Multi-Scale Parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Situation**: Parameters span many orders of magnitude.

**Pre-0.3.6 Issue**: Aggressive LR caused instability for small parameters.

**0.3.6 Solution**: Layer 2 scales LR appropriately, Layer 4 clips large steps.

.. code-block:: python

    def multi_scale_model(x, amplitude, decay_rate, offset):
        # amplitude ~ 1e6, decay_rate ~ 1e-6, offset ~ 100
        return amplitude * jnp.exp(-decay_rate * x) + offset


    # With bounds to help normalization
    bounds = ([1e5, 1e-7, 0], [1e7, 1e-5, 1000])

    popt, pcov = curve_fit(
        multi_scale_model,
        x,
        y,
        p0=[1e6, 1e-6, 100],
        bounds=bounds,
        method="hybrid_streaming",
    )

    # Layer 2 will use conservative LR
    # Layer 4 will clip large updates

Scenario 3: Production Monitoring
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Situation**: Running hundreds of fits in production, need to monitor stability.

**0.3.6 Solution**: Use telemetry to track defense layer activations.

.. code-block:: python

    import time
    from nlsq import get_defense_telemetry, reset_defense_telemetry

    # Production fitting loop
    reset_defense_telemetry()
    start_time = time.time()

    for sample_id, (x, y) in enumerate(data_stream):
        try:
            popt, pcov = curve_fit(
                model,
                x,
                y,
                method="hybrid_streaming",
                config=HybridStreamingConfig.scientific_default(),
            )
            save_results(sample_id, popt, pcov)
        except Exception as e:
            log_error(sample_id, e)

        # Report telemetry every hour
        if time.time() - start_time > 3600:
            telemetry = get_defense_telemetry()

            # Push to monitoring dashboard
            metrics = telemetry.export_metrics()
            push_to_prometheus(metrics)

            # Reset for next window
            reset_defense_telemetry()
            start_time = time.time()

Troubleshooting
---------------

Problem: Results Changed After Upgrade
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Symptom**: Fitted parameters slightly different after upgrading to 0.3.6.

**Likely cause**: Defense layers preventing divergence that occurred in 0.3.5.

**Diagnosis**:

.. code-block:: python

    # Check which layers triggered
    telemetry = get_defense_telemetry()
    rates = telemetry.get_trigger_rates()

    if rates["layer1_warm_start_rate"] > 0:
        print("Warm start detected - warmup was skipped")
    if rates["layer3_cost_guard_rate"] > 0:
        print("Cost guard triggered - divergence prevented")

**Resolution**:

- The new results are likely **more accurate** (defense prevented divergence)
- If you need exact pre-0.3.6 behavior, use ``defense_disabled()``
- If changing behavior is a problem, pin to ``nlsq==0.3.5``

Problem: Warmup Always Skipped
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Symptom**: Layer 1 triggers on every fit, no Adam warmup runs.

**Diagnosis**:

.. code-block:: python

    telemetry = get_defense_telemetry()
    if telemetry.get_trigger_rates()["layer1_warm_start_rate"] == 100.0:
        print("Warm start threshold too high")

**Resolution**:

.. code-block:: python

    # Option 1: Lower threshold
    config = HybridStreamingConfig(warm_start_threshold=0.001)

    # Option 2: Disable Layer 1
    config = HybridStreamingConfig(enable_warm_start_detection=False)

    # Option 3: Use relaxed preset
    config = HybridStreamingConfig.defense_relaxed()

Problem: Convergence Too Slow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Symptom**: More iterations needed after upgrading.

**Likely cause**: Layer 2 using conservative LR when exploration would be better.

**Diagnosis**:

.. code-block:: python

    rates = telemetry.get_trigger_rates()
    if rates["layer2_refinement_rate"] > 50:
        print("Using ultra-conservative LR frequently")

**Resolution**:

.. code-block:: python

    # Increase refinement/careful LRs
    config = HybridStreamingConfig(
        warmup_lr_refinement=1e-5,  # Increase from 1e-6
        warmup_lr_careful=1e-4,  # Increase from 1e-5
    )

    # Or use relaxed preset
    config = HybridStreamingConfig.defense_relaxed()

Problem: Cost Guard Aborts Too Early
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Symptom**: Layer 3 triggers frequently, warmup doesn't complete.

**Diagnosis**:

.. code-block:: python

    if telemetry.get_trigger_rates()["layer3_cost_guard_rate"] > 20:
        print("Cost guard very active - may be protecting from divergence")

**Resolution**:

.. code-block:: python

    # Option 1: Increase tolerance (allow more cost increase)
    config = HybridStreamingConfig(cost_increase_tolerance=0.2)

    # Option 2: Disable cost guard (not recommended)
    config = HybridStreamingConfig(enable_cost_guard=False)

    # Note: High cost guard rate may indicate poor initial guess

Testing Your Migration
-----------------------

Regression Test Suite
~~~~~~~~~~~~~~~~~~~~~

Add tests to verify behavior with defense layers:

.. code-block:: python

    import pytest
    from nlsq import curve_fit, HybridStreamingConfig, reset_defense_telemetry


    def test_defense_layers_enabled_by_default():
        """Verify defense layers are active by default."""
        reset_defense_telemetry()

        popt, pcov = curve_fit(model, x, y, method="hybrid_streaming")

        # Defense layers should be enabled
        config = HybridStreamingConfig()
        assert config.enable_warm_start_detection is True
        assert config.enable_adaptive_warmup_lr is True
        assert config.enable_cost_guard is True
        assert config.enable_step_clipping is True


    def test_defense_disabled_preset():
        """Verify defense_disabled() reverts to pre-0.3.6 behavior."""
        config = HybridStreamingConfig.defense_disabled()

        assert config.enable_warm_start_detection is False
        assert config.enable_adaptive_warmup_lr is False
        assert config.enable_cost_guard is False
        assert config.enable_step_clipping is False


    def test_warm_start_triggers_correctly():
        """Verify Layer 1 skips warmup for near-optimal guess."""
        from nlsq import get_defense_telemetry

        reset_defense_telemetry()

        # Fit with very good initial guess
        popt, pcov = curve_fit(
            model, x, y, p0=near_optimal_params, method="hybrid_streaming"
        )

        telemetry = get_defense_telemetry()
        rates = telemetry.get_trigger_rates()

        # Warm start should have triggered
        assert rates["layer1_warm_start_rate"] > 0

Benchmark Comparison
~~~~~~~~~~~~~~~~~~~~

Compare performance before/after:

.. code-block:: python

    import time
    from nlsq import curve_fit, HybridStreamingConfig


    def benchmark_defense_layers():
        """Compare defense enabled vs disabled."""

        # With defense (0.3.6 default)
        start = time.time()
        popt_def, _ = curve_fit(model, x, y, method="hybrid_streaming")
        time_defense = time.time() - start

        # Without defense (pre-0.3.6 behavior)
        config_old = HybridStreamingConfig.defense_disabled()
        start = time.time()
        popt_old, _ = curve_fit(model, x, y, method="hybrid_streaming", config=config_old)
        time_no_defense = time.time() - start

        print(f"Defense enabled:  {time_defense:.3f}s")
        print(f"Defense disabled: {time_no_defense:.3f}s")
        print(f"Overhead: {(time_defense/time_no_defense - 1)*100:.1f}%")

Further Reading
---------------

- :doc:`../guides/defense_layers` - Complete defense strategy documentation
- :doc:`../api/nlsq.adaptive_hybrid_streaming` - API reference
- :doc:`../api/nlsq.hybrid_streaming_config` - Configuration reference
- :doc:`../guides/troubleshooting` - General troubleshooting guide

Summary
-------

**Key Takeaways**:

1. ✅ **No code changes required** for most users
2. ✅ **Improved stability** - defense prevents warmup divergence
3. ✅ **Opt-out available** - use ``defense_disabled()`` if needed
4. ✅ **Telemetry included** - monitor defense activations
5. ✅ **Fully backward compatible** - API unchanged

**Recommended Actions**:

1. **Test your existing code** with 0.3.6 (should work without changes)
2. **Monitor telemetry** to understand defense layer usage
3. **Adjust sensitivity** if needed using presets or custom config
4. **Report issues** at https://github.com/imewei/nlsq/issues

**When to disable defense layers**:

- Regression testing against 0.3.5 results
- Benchmarking raw optimizer performance
- Debugging convergence issues

**When to use strict defense**:

- Warm-start scenarios
- Refinement workflows
- Near-optimal initial guesses
- Ill-conditioned problems
