Durations and SMDP Discounting - SMDPfier
Description: SMDP-level behavior for Gymnasium environments

Skip to content

# Durations and SMDP Discounting

Duration in SMDPfier v0.2.0+ is simple: each primitive action = 1 tick, and option duration equals the number of primitive actions executed (k_exec).

## 🎯 Time Semantics (v0.2.0+)

The key principle: Each primitive action = 1 tick of time.

Concept | Value
--- | ---
Tick | One primitive action execution
Option Duration | Number of primitive actions executed (k_exec)
Complete Execution | duration = `len(option.actions)`
Partial Execution | duration = number of actions before termination

### Simple Example

    from smdpfier import SMDPfier, Option
    import gymnasium as gym

    env = gym.make("CartPole-v1")
    options = [
        Option([0, 1, 0], "three-step"),      # 3 actions = 3 ticks
        Option([1, 1, 0, 0], "four-step"),    # 4 actions = 4 ticks
        Option([0, 1], "two-step"),           # 2 actions = 2 ticks
    ]

    smdp_env = SMDPfier(env, options_provider=options, action_interface="index", max_options=3)

    # Execute first option (3 actions)
    obs, reward, term, trunc, info = smdp_env.step(0)

    # Check duration
    assert info["smdp"]["duration"] == 3  # 3 actions executed = 3 ticks
    assert info["smdp"]["k_exec"] == 3    # Same value

### Visual Example

    Option: [0, 1, 0, 1, 1]  (5 actions)

    Normal Execution:
    ┌─────┬─────┬─────┬─────┬─────┐
    │ a=0 │ a=1 │ a=0 │ a=1 │ a=1 │
    └─────┴─────┴─────┴─────┴─────┘
    tick:   1     2     3     4     5    → duration = 5

    Early Termination (after 3 actions):
    ┌─────┬─────┬─────┐  X
    │ a=0 │ a=1 │ a=0 │ episode ends
    └─────┴─────┴─────┘
    tick:   1     2     3                → duration = 3

## 🔄 SMDP Discounting

With simple duration = k_exec, SMDP discounting becomes straightforward:

### Basic Discounting

    gamma = 0.99

    # Execute option
    obs, reward, term, trunc, info = env.step(action)
    duration = info["smdp"]["duration"]

    # Apply SMDP discount
    discounted_reward = reward * (gamma ** duration)

### Multi-Step Example

    # Execute sequence of options
    gamma = 0.99
    total_time = 0
    cumulative_return = 0

    for action in [0, 1, 2]:
        obs, reward, term, trunc, info = env.step(action)
        duration = info["smdp"]["duration"]

        # Apply time-cumulative discount
        discount_factor = gamma ** total_time
        cumulative_return += reward * discount_factor

        total_time += duration
        if term or trunc:
            break

    print(f"Total time: {total_time} ticks")
    print(f"Cumulative return: {cumulative_return}")

### Comparison: MDP vs SMDP

    # MDP: Each primitive action discounts by γ
    mdp_return = r₀ + γ¹·r₁ + γ²·r₂ + γ³·r₃ + γ⁴·r₄

    # SMDP: Each option discounts by γ^{duration}
    # Options with lengths [3, 2, 4]:
    smdp_return = R₀ + γ³·R₁ + γ⁵·R₂
    #                   ↑      ↑
    #                   3    3+2

## 📊 Reward Aggregation

By default, SMDPfier sums the per-step rewards to produce the macro reward:

    # Default behavior (v0.2.0+)
    macro_reward = sum(primitive_rewards)

    # Example:
    # Option executes 3 steps with rewards [1.0, 1.0, 1.0]
    # macro_reward = 1.0 + 1.0 + 1.0 = 3.0

### Custom Aggregators

You can customize reward aggregation:

    from smdpfier import SMDPfier
    from smdpfier.defaults import mean_rewards, discounted_sum

    # Average per-step rewards
    env = SMDPfier(env, options_provider=options, reward_agg=mean_rewards)
    # 3 steps with rewards [1.0, 1.0, 1.0] → macro_reward = 1.0

    # Discount per-step rewards
    env = SMDPfier(env, options_provider=options, reward_agg=discounted_sum(gamma=0.99))
    # 3 steps with rewards [1.0, 1.0, 1.0] → macro_reward = 1.0 + 0.99 + 0.98 = 2.97

    # Custom aggregator
    def custom_agg(rewards):
        return sum(r * (i + 1) for i, r in enumerate(rewards))  # Weighted sum

    env = SMDPfier(env, options_provider=options, reward_agg=custom_agg)

## ⚠️ Early Termination

When the episode terminates before an option completes, duration reflects actual execution:

    option = Option([0, 1, 0, 1, 1], "five-step")

    # Episode terminates after 3rd action
    # - k_exec = 3
    # - duration = 3 (not 5!)
    # - rewards = [r₁, r₂, r₃] (only 3 rewards)
    # - terminated_early = True

    obs, reward, term, trunc, info = env.step(option)
    if info["smdp"]["terminated_early"]:
        print(f"Option interrupted after {info['smdp']['k_exec']} of {info['smdp']['option']['len']} actions")
        print(f"Duration: {info['smdp']['duration']} ticks (not {info['smdp']['option']['len']})")

### Why This Matters for Discounting

    # Option length: 5 actions
    # Terminated after 3 actions

    # ✅ Correct: Use actual duration
    duration = info["smdp"]["duration"]  # = 3
    next_discount = gamma ** duration    # = γ³

    # ❌ Wrong: Use planned duration
    planned = info["smdp"]["option"]["len"]  # = 5
    wrong_discount = gamma ** planned         # = γ⁵ (incorrect!)

## 🧪 Full Example

    import gymnasium as gym
    from smdpfier import SMDPfier, Option

    # Setup
    env = gym.make("CartPole-v1")
    options = [
        Option([0, 0, 1], "left-left-right"),    # 3 ticks
        Option([1, 1, 0, 0], "right-right-left-left"),  # 4 ticks
    ]

    smdp_env = SMDPfier(
        env,
        options_provider=options,
        action_interface="index",
        max_options=2
    )

    # Episode
    obs, info = smdp_env.reset()
    gamma = 0.99
    total_time = 0
    returns = []

    for step in range(10):
        action = smdp_env.action_space.sample()
        obs, reward, term, trunc, info = smdp_env.step(action)

        smdp = info["smdp"]
        duration = smdp["duration"]

        print(f"Step {step}:")
        print(f"  Option: {smdp['option']['name']}")
        print(f"  Duration: {duration} ticks")
        print(f"  Rewards: {smdp['rewards']}")
        print(f"  Macro reward: {reward}")
        print(f"  Discounted: {reward * (gamma ** total_time):.4f}")

        total_time += duration
        returns.append(reward * (gamma ** total_time))

        if term or trunc:
            break

    print(f"\nTotal time: {total_time} ticks")
    print(f"Total return: {sum(returns):.4f}")

## 📝 Key Takeaways

1. Each primitive action = 1 tick: Time is simple and natural
2. Duration = k_exec: No separate duration function needed
3. Early termination: Duration reflects actual execution, not planned
4. SMDP discounting: Use `γ^{duration}` for each option
5. Macro reward: Sum of per-step rewards by default (customizable)

## 🔄 Migrating from 0.1.x

In v0.1.x, you had to specify a `duration_fn`. In v0.2.0+, this is automatic:

    # 0.1.x - Old way
    from smdpfier.defaults import ConstantActionDuration

    env = SMDPfier(
        env,
        options_provider=options,
        duration_fn=ConstantActionDuration(1),  # Each action = 1 tick
        # ...
    )

    # 0.2.0+ - New way (automatic!)
    env = SMDPfier(
        env,
        options_provider=options,
        # duration_fn removed - each action automatically = 1 tick
        # ...
    )

See Migration Guide for complete upgrade instructions.

Next: FAQ | Previous: API Reference - When different actions have different inherent durations

### Duration Type Comparison

Scenario | Scalar Duration | List Duration
--- | --- | ---
Option: `[0,1,0]` | `10` → 10 ticks total | `[3,4,3]` → 10 ticks total
Early Termination | Apply partial duration policy | Sum executed portion
Complexity | Simple | More detailed
Use Case | Macro-actions | Action-specific costs

## SMDP Discounting

The power of SMDPs lies in proper temporal discounting using elapsed time rather than step counts.

### Standard MDP vs SMDP

Standard MDP Discounting:

    # Each step advances time by exactly 1 unit
    return = r₁ + γ¹·r₂ + γ²·r₃ + γ³·r₄ + ...
    #           ↑    ↑    ↑    ↑
    #          t=1  t=2  t=3  t=4

SMDP Discounting:

    # Each option advances time by its duration
    # Options with durations [5, 3, 7, 2] ticks:
    return = r₁ + γ⁵·r₂ + γ⁸·r₃ + γ¹⁵·r₄ + γ¹⁷·r₅
    #           ↑     ↑     ↑      ↑      ↑
    #          t=0   t=5   t=8   t=15   t=17

### Practical SMDP Discounting Implementation

    import gymnasium as gym
    from smdpfier import SMDPfier, Option
    from smdpfier.defaults import ConstantOptionDuration

    # Setup
    env = SMDPfier(
        gym.make("CartPole-v1"),
        options_provider=[
            Option([0, 0], "left-left"),     # 2 steps, 10 ticks
            Option([1, 1, 1], "right-x3"),   # 3 steps, 10 ticks
            Option([0, 1], "left-right"),    # 2 steps, 10 ticks
        ],
        duration_fn=ConstantOptionDuration(10)
    )

    # Episode with SMDP discounting
    gamma = 0.99
    discounted_return = 0
    elapsed_time = 0

    obs, info = env.reset()
    for step in range(3):
        action = step % 3  # Cycle through options
        obs, reward, term, trunc, info = env.step(action)

        # Get duration from SMDP info
        duration = info["smdp"]["duration_exec"]

        # Apply SMDP discounting
        discount_factor = gamma ** elapsed_time
        discounted_return += discount_factor * reward

        # Update elapsed time for next option
        elapsed_time += duration

        print(f"Option {action}: reward={reward}, duration={duration}, "
              f"discount=γ^{elapsed_time-duration}={discount_factor:.4f}")

        if term or trunc:
            break

    print(f"Total SMDP discounted return: {discounted_return:.4f}")

Sample Output:

    Option 0: reward=2.0, duration=10, discount=γ^0=1.0000
    Option 1: reward=3.0, duration=10, discount=γ^10=0.9044
    Option 2: reward=1.0, duration=10, discount=γ^20=0.8179
    Total SMDP discounted return: 4.6223

## Partial Duration Policies

When an episode terminates early (before an option completes), SMDPfier applies a partial duration policy to determine the executed duration.

### Policy Comparison

Consider an option with 3 actions and scalar duration 10, terminating after 2 actions:

Policy | Formula | Result | Use Case
--- | --- | --- | ---
`"proportional"` | `(k_exec / option_len) * planned_duration` | `(2/3) * 10 = 6` | Default - proportional time
`"full"` | `planned_duration` | `10` | Option "completes" conceptually
`"zero"` | `0` | `0` | No time passes on failure

### Example: Partial Duration Calculation

    from smdpfier.defaults import ConstantOptionDuration

    # Option with 4 actions, 12 ticks planned
    option = Option([0, 1, 0, 1], "four-action-option")
    duration_fn = ConstantOptionDuration(12)

    # Episode terminates after 3 actions (k_exec = 3)

    # Proportional policy (default):
    duration_exec = (3 / 4) * 12 = 9 ticks

    # Full policy:
    duration_exec = 12 ticks

    # Zero policy:
    duration_exec = 0 ticks

### Policy Selection Guide

Choose | When
--- | ---
`"proportional"` | Time should scale with execution (most realistic)
`"full"` | Options have setup costs regardless of completion
`"zero"` | Failed options should not consume time

## Advanced Duration Functions

### Custom Duration Functions

    def custom_duration_fn(option, obs, info):
        """Duration based on option length and environment state."""
        base_duration = len(option.actions) * 2

        # Adjust based on environment state
        if obs[0] > 0:  # Cart position in CartPole
            return base_duration + 5  # Takes longer when cart is right
        else:
            return base_duration

    # Usage
    env = SMDPfier(env, options_provider=options, duration_fn=custom_duration_fn)

### State-Dependent Durations

    from smdpfier.defaults import MapActionDuration

    # Different actions have different costs
    action_duration_map = {
        0: 2,  # Left action: 2 ticks
        1: 5,  # Right action: 5 ticks (more expensive)
    }

    duration_fn = MapActionDuration(action_duration_map, default_duration=3)

    # Option [0, 1, 0] → durations [2, 5, 2] → total 9 ticks

## Summary

Concept | Key Point
--- | ---
Steps vs Ticks | Steps = execution, Ticks = time metadata
Duration Types | Scalar (per-option) vs List (per-action)
SMDP Discounting | Use γ^{duration_exec}, not γ^{steps}
Partial Policies | Handle early termination gracefully
Custom Functions | Tailor durations to your domain

Remember: Duration is metadata for discounting - it never affects how many environment steps are executed!

Next: Index vs Direct Interfaces | See Also: API Reference