Vision API Reference — ManipulaPy 1.1.0 Documentation
Description: This page documents ManipulaPy.vision, the module for computer vision capabilities including stereo vision, object detection, camera calibration, and PyBullet integration with optional YOLO object ...

# Vision API Reference

This page documents ManipulaPy.vision, the module for computer vision capabilities including stereo vision, object detection, camera calibration, and PyBullet integration with optional YOLO object detection.

Tip

For conceptual explanations, see Vision User Guide.

—

## Quick Navigation

—

## Vision Class

—

## Camera Management

### Configuration Methods

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 68)

Title underline too short.

    Configuration Methods
    --------------------

### Extrinsic Matrix Computation

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 101)

Title underline too short.

    Extrinsic Matrix Computation
    ---------------------------

—

## Image Acquisition

### PyBullet Integration

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 139)

Title underline too short.

    PyBullet Integration
    -------------------

—

## Object Detection

### YOLO Integration

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 169)

Title underline too short.

    YOLO Integration
    ---------------

—

## Stereo Vision Pipeline

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 210)

Title underline too short.

    Stereo Vision Pipeline
    =====================

### Rectification Setup

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 213)

Title underline too short.

    Rectification Setup
    ------------------

### Image Processing

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 233)

Title underline too short.

    Image Processing
    ---------------

### 3D Reconstruction

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 277)

Title underline too short.

    3D Reconstruction
    ----------------

—

## PyBullet Debug Interface

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 319)

Title underline too short.

    PyBullet Debug Interface
    =======================

### Debug Slider Setup

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 322)

Title underline too short.

    Debug Slider Setup
    -----------------

—

## Resource Management

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 364)

Title underline too short.

    Resource Management
    ==================

### Cleanup Methods

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 367)

Title underline too short.

    Cleanup Methods
    --------------

—

## Utility Functions

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 394)

Title underline too short.

    Utility Functions
    ================

—

## Data Structures and Configuration

### Internal Storage Format

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 420)

Title underline too short.

    Internal Storage Format
    ----------------------

Camera Storage (self.cameras):

    self.cameras[index] = {
        "name": str,
        "translation": [x, y, z],
        "rotation": [roll_deg, pitch_deg, yaw_deg],
        "fov": float,
        "near": float,
        "far": float,
        "intrinsic_matrix": np.ndarray(3, 3),
        "distortion_coeffs": np.ndarray(5,),
        "use_opencv": bool,
        "device_index": int,
        "extrinsic_matrix": np.ndarray(4, 4)
    }

Stereo Configuration Attributes:

    # Stereo processing state
    self.stereo_enabled: bool
    self.left_cam_cfg: dict
    self.right_cam_cfg: dict

    # Rectification maps
    self.left_map_x: np.ndarray
    self.left_map_y: np.ndarray
    self.right_map_x: np.ndarray
    self.right_map_y: np.ndarray

    # 3D reconstruction
    self.Q: np.ndarray  # 4x4 disparity-to-depth matrix
    self.stereo_matcher: cv2.StereoSGBM

### Default Camera Configuration

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 460)

Title underline too short.

    Default Camera Configuration
    ---------------------------

When no camera_configs provided, uses:

    default_config = {
        "name": "default_camera",
        "translation": [0, 0, 0],
        "rotation": [0, 0, 0],
        "fov": 60,
        "near": 0.1,
        "far": 5.0,
        "intrinsic_matrix": [[500, 0, 320],
                            [0, 500, 240],
                            [0, 0, 1]],
        "distortion_coeffs": [0, 0, 0, 0, 0],
        "use_opencv": False,
        "device_index": 0
    }

—

## Error Handling and Validation

### YOLO Model Management

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 487)

Title underline too short.

    YOLO Model Management
    --------------------

- Loading Failure: Sets yolo_model = None, continues operation

- Detection Fallback: Returns empty arrays when YOLO unavailable

- Input Validation: Checks rgb_image and depth_image for None/invalid

### Stereo Processing Errors

- Configuration Validation: Checks required keys in stereo configs

- Runtime State Checking: Validates stereo_enabled before operations

- Map Initialization: Ensures rectification maps computed before use

### OpenCV Device Handling

- Device Access Failure: Raises RuntimeError with descriptive message

- Capture Validation: Uses cap.isOpened() to verify device accessibility

- Resource Cleanup: Automatic release in destructor and explicit method

### PyBullet Integration Safety

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 508)

Title underline too short.

    PyBullet Integration Safety
    --------------------------

- Parameter Reading: Safe handling of missing debug parameters

- Matrix Validation: Debug logging for view/projection matrix inspection

- Client State: No assumptions about PyBullet connection status

—

## Performance Considerations

### Memory Management

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 520)

Title underline too short.

    Memory Management
    ----------------

- Image Arrays: Contiguous memory layout for OpenCV operations

- Rectification Maps: Persistent storage for repeated stereo processing

- Point Clouds: Filtered arrays to reduce memory footprint

### Computational Efficiency

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 527)

Title underline too short.

    Computational Efficiency
    -----------------------

- YOLO Inference: Single forward pass per image

- Stereo Matching: StereoSGBM optimized for quality/speed balance

- Depth Scaling: Vectorized operations for range conversion

### Threading Considerations

System Message: WARNING/2 (/home/abo/Desktop/ManipulaPy/docs/source/api/vision.rst, line 534)

Title underline too short.

    Threading Considerations
    -----------------------

- OpenCV Capture: Single-threaded device access

- YOLO Processing: GPU acceleration when available

- PyBullet Integration: Main thread simulation access required

—

## See Also

- Perception API Reference – Higher-level perception capabilities using Vision

- Utils Module – Mathematical utilities for transformations

- Simulation API Reference – PyBullet simulation integration

- Potential Field API Reference – Obstacle avoidance using vision data

## External Dependencies

- OpenCV – Computer vision algorithms and stereo processing

- PyBullet – Physics simulation and camera rendering

- Ultralytics YOLO – Object detection framework

- NumPy – Numerical array operations

- Matplotlib – Debug visualization