Index vs Direct Interfaces - SMDPfier
Description: SMDP-level behavior for Gymnasium environments

Skip to content

# Index vs Direct Interfaces

SMDPfier provides two interfaces for option selection, each optimized for different use cases. Understanding when to use each interface is crucial for effective SMDP implementation.

## Interface Comparison

Aspect | Index Interface | Direct Interface
--- | --- | ---
Action Space | `Discrete(max_options)` | Same as Option actions
Actions | Integer indices (0, 1, 2, ...) | Option objects
Best For | Reinforcement Learning | Scripting/Testing
Action Masking | Built-in support | Not applicable
Dynamic Options | Supported with overflow handling | Straightforward
RL Integration | Seamless | Requires adaptation
Debugging | Index-based (less intuitive) | Object-based (more intuitive)

## Index Interface

The index interface transforms SMDPfier into a discrete action space where each action is an integer index selecting an available option.

### When to Use Index Interface

✅ Choose Index Interface When: - Training RL agents (most algorithms expect discrete actions) - Need action masking based on environment state - Working with dynamic option sets - Integrating with existing RL frameworks (Stable-Baselines3, RLLib, etc.) - Want built-in overflow handling for dynamic options

### Basic Index Interface Setup

    import gymnasium as gym
    from smdpfier import SMDPfier, Option
    from smdpfier.defaults import ConstantOptionDuration

    # Define static options
    options = [
        Option([0, 0, 1], "left-left-right"),
        Option([1, 1, 0], "right-right-left"),
        Option([0, 1, 0], "left-right-left"),
    ]

    # Create SMDPfier with index interface
    env = SMDPfier(
        gym.make("CartPole-v1"),
        options_provider=options,
        duration_fn=ConstantOptionDuration(5),
        action_interface="index",  # Default
        max_options=len(options)   # Must specify for static options
    )

    print(f"Original action space: Discrete(2)")
    print(f"SMDP action space: {env.action_space}")  # Discrete(3)

    # Use integer actions
    obs, info = env.reset()
    action = 1  # Select second option ("right-right-left")
    obs, reward, term, trunc, info = env.step(action)

    print(f"Executed option: {info['smdp']['option']['name']}")

### Action Masking with Index Interface

Action masking allows you to restrict which options are available based on the current environment state.

    def cart_availability(obs):
        """Restrict options based on cart position."""
        cart_position = obs[0]

        if cart_position > 0.5:
            return [0, 2]  # Only left-based options when cart is far right
        elif cart_position < -0.5:
            return [1]     # Only right-based options when cart is far left
        else:
            return [0, 1, 2]  # All options available in center

    env = SMDPfier(
        gym.make("CartPole-v1"),
        options_provider=options,
        duration_fn=ConstantOptionDuration(5),
        action_interface="index",
        max_options=3,
        availability_fn=cart_availability
    )

    obs, info = env.reset()
    print(f"Available actions: {info['smdp']['action_mask']}")
    # Might show: [1, 1, 1] (all available) or [1, 0, 1] (middle option masked)

### Dynamic Options with Index Interface

Dynamic options change based on the current state, requiring careful overflow handling.

    from smdpfier.defaults.options import RandomStaticLen

    def dynamic_options_generator(obs, info):
        """Generate different options based on cart velocity."""
        velocity = obs[1]

        if abs(velocity) > 0.1:  # Fast movement
            # Short options for quick corrections
            return [
                Option([0], "quick-left"),
                Option([1], "quick-right"),
            ]
        else:  # Slow movement
            # Longer options for building momentum
            return [
                Option([0, 0, 0], "triple-left"),
                Option([1, 1, 1], "triple-right"),
                Option([0, 1, 0], "left-right-left"),
                Option([1, 0, 1], "right-left-right"),
            ]

    env = SMDPfier(
        gym.make("CartPole-v1"),
        options_provider=dynamic_options_generator,
        duration_fn=ConstantOptionDuration(3),
        action_interface="index",
        max_options=4  # Maximum expected options
    )

    obs, info = env.reset()
    # If generator returns 2 options but max_options=4:
    # - Actions 0,1 are valid
    # - Actions 2,3 are masked out
    # - info["smdp"]["action_mask"] = [1, 1, 0, 0]

### Overflow Handling

When dynamic options exceed `max_options`, SMDPfier applies truncate policy by default:

    # Generator returns 6 options, but max_options=4
    available_options = dynamic_generator(obs, info)  # Returns 6 options
    # Result: First 4 options are used, 2 are dropped
    # info["smdp"]["num_dropped"] = 2

## Direct Interface

The direct interface allows you to pass `Option` objects directly to `env.step()`, providing an intuitive and flexible approach.

### When to Use Direct Interface

✅ Choose Direct Interface When: - Scripting or testing specific option sequences - Building non-RL controllers or heuristics - Debugging option behavior - Prototyping before RL training - Need full control over option selection

### Basic Direct Interface Setup

    import gymnasium as gym
    from smdpfier import SMDPfier, Option
    from smdpfier.defaults import ConstantActionDuration

    # Define options
    options = [
        Option([0, 0, 1], "left-left-right"),
        Option([1, 1, 0], "right-right-left"),
        Option([0, 1, 0, 1], "alternating"),
    ]

    # Create SMDPfier with direct interface
    env = SMDPfier(
        gym.make("CartPole-v1"),
        options_provider=options,
        duration_fn=ConstantActionDuration(2),  # 2 ticks per action
        action_interface="direct"
    )

    print(f"Action space: {env.action_space}")  # Same as original env

    # Use Option objects directly
    obs, info = env.reset()
    option = options[1]  # Select "right-right-left"
    obs, reward, term, trunc, info = env.step(option)

    print(f"Executed {info['smdp']['k_exec']} steps")
    print(f"Duration: {info['smdp']['duration_exec']} ticks")

### Dynamic Options with Direct Interface

    def get_option_for_state(obs):
        """Select option based on current state."""
        cart_position, cart_velocity = obs[0], obs[1]

        if cart_position > 0 and cart_velocity > 0:
            return Option([0, 0], "strong-left")  # Moving right, correct strongly
        elif cart_position < 0 and cart_velocity < 0:
            return Option([1, 1], "strong-right")  # Moving left, correct strongly
        else:
            return Option([0, 1], "gentle-correction")  # Gentle correction

    # Simple control loop
    obs, info = env.reset()
    for step in range(100):
        option = get_option_for_state(obs)
        obs, reward, term, trunc, info = env.step(option)

        if term or trunc:
            break

### Continuous Actions with Direct Interface

The direct interface works seamlessly with continuous action spaces:

    import gymnasium as gym
    from smdpfier import SMDPfier, Option
    from smdpfier.defaults import ConstantOptionDuration

    # Continuous action options
    continuous_options = [
        Option([[-1.0], [0.0], [1.0]], "left-center-right"),
        Option([[0.5], [0.5]], "gentle-right"),
        Option([[-2.0]], "hard-left"),
    ]

    env = SMDPfier(
        gym.make("Pendulum-v1"),
        options_provider=continuous_options,
        duration_fn=ConstantOptionDuration(5),
        action_interface="direct"
    )

    obs, info = env.reset()
    option = continuous_options[0]
    obs, reward, term, trunc, info = env.step(option)

## Interface Selection Guide

### Choose Index Interface When:

    # RL Training
    env = SMDPfier(..., action_interface="index", max_options=10)
    agent.learn(env)  # Works with any RL algorithm

    # Action Masking Needed
    env = SMDPfier(..., action_interface="index", availability_fn=mask_fn)

    # Dynamic Options with Overflow
    env = SMDPfier(..., action_interface="index", max_options=20)
    # Handles varying option counts gracefully

### Choose Direct Interface When:

    # Scripted Control
    for situation in test_cases:
        option = select_option_for_situation(situation)
        obs, reward, term, trunc, info = env.step(option)

    # Debugging Specific Options
    problem_option = Option([0, 1, 0], "problematic-sequence")
    obs, reward, term, trunc, info = env.step(problem_option)

    # Prototyping Before RL
    def human_policy(obs):
        return Option([best_action_for(obs)], "human-choice")

## Configuration Examples

### Index Interface Configuration

    # Static options with masking
    env = SMDPfier(
        base_env,
        options_provider=static_options,
        duration_fn=ConstantOptionDuration(10),
        action_interface="index",
        max_options=len(static_options),
        availability_fn=masking_function,
        precheck=True  # Validate options before execution
    )

    # Dynamic options with overflow handling
    env = SMDPfier(
        base_env,
        options_provider=dynamic_generator,
        duration_fn=RandomActionDuration(2, 5),
        action_interface="index",
        max_options=15,  # Allow up to 15 options
        # Overflow: truncate to first 15, record num_dropped
    )

### Direct Interface Configuration

    # Simple direct interface
    env = SMDPfier(
        base_env,
        options_provider=option_list,
        duration_fn=ConstantActionDuration(3),
        action_interface="direct"
        # No max_options needed
        # No availability_fn needed
    )

## Summary

Use Case | Recommended Interface | Why
--- | --- | ---
RL Training | Index | Discrete action space expected
Action Masking | Index | Built-in masking support
Dynamic Options | Index | Overflow handling
Scripting/Testing | Direct | Intuitive option passing
Debugging | Direct | Clear option identification
Continuous Actions | Direct | Natural continuous support
Prototyping | Direct | Flexible experimentation

Most Common Pattern: 1. Start with direct interface for prototyping and testing 2. Switch to index interface when training RL agents

Next: Masking and Precheck | See Also: API Reference