
# DO NOT EDIT THIS FILE -- AUTOGENERATED BY PANTS
# Target: src/ai/backend/agent:dist

from setuptools import setup

setup(**{
    'author': 'Lablup Inc. and contributors',
    'classifiers': [
        'Intended Audience :: Developers',
        'Operating System :: MacOS :: MacOS X',
        'Operating System :: POSIX :: Linux',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3',
        'Environment :: No Input/Output (Daemon)',
        'Topic :: Scientific/Engineering',
        'Topic :: Software Development',
        'Development Status :: 5 - Production/Stable',
        'Programming Language :: Python :: 3.13',
        'License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)',
    ],
    'description': 'Backend.AI Agent',
    'entry_points': {
        'backendai_cli_v10': [
            'ag = ai.backend.agent.cli.__main__:main',
            'ag.start-server = ai.backend.agent.server:main',
        ],
        'backendai_network_agent_v1': [
            'overlay = ai.backend.agent.docker.intrinsic:OverlayNetworkPlugin',
            'host = ai.backend.agent.docker.intrinsic:HostNetworkPlugin',
        ],
    },
    'install_requires': (
        'aiodocker==0.24.0',
        'aiofiles~=24.1.0',
        'aiohttp_cors~=0.8.1',
        'aiohttp~=3.12.15',
        'aiomonitor~=0.7.0',
        'aiotools~=2.2.3',
        'async_timeout~=4.0',
        'attrs>=25.3',
        """backend.ai-cli==25.16.0
""",
        """backend.ai-common==25.16.0
""",
        """backend.ai-kernel-binary==25.16.0
""",
        """backend.ai-kernel-helper==25.16.0
""",
        """backend.ai-kernel==25.16.0
""",
        'backend.ai-krunner-static-gnu==4.4.0',
        """backend.ai-logging==25.16.0
""",
        """backend.ai-plugin==25.16.0
""",
        'cachetools~=5.5.0',
        'callosum~=1.0.3',
        'cattrs~=24.1.1',
        'click~=8.1.7',
        'etcd-client-py~=0.4.1',
        'janus~=2.0',
        'kubernetes-asyncio~=33.3.0',
        'kubernetes~=33.1.0',
        'more-itertools~=10.5.0',
        'networkx~=3.3.0',
        'prometheus-client~=0.21.1',
        'psutil~=7.0',
        'pydantic[email]~=2.11.3',
        'pyzmq~=26.4',
        'ruamel.yaml~=0.18.10',
        'setproctitle~=1.3.5',
        'setuptools~=80.0.0',
        'tenacity>=9.0',
        'tomlkit~=0.13.2',
        'trafaret~=2.1',
        'types-aiofiles',
        'types-cachetools',
        'types-psutil',
        'types-setuptools',
        'typing_extensions~=4.11',
        'uvloop~=0.22.1; sys_platform != "Windows"',
    ),
    'license': 'LGPLv3',
    'long_description': """# Backend.AI Agent

## Purpose

The Agent is a compute node component responsible for container lifecycle management. It runs computing sessions as containers, monitors resource usage, and reports status to the Manager.

## Key Responsibilities

### 1. Container Lifecycle Management
- Create and start containers from session specifications
- Initialize kernels and execute runtime configuration
- Monitor container health and status
- Delete containers upon session termination
- Handle container restarts and error recovery

### 2. Resource Monitoring
- Track CPU, memory, GPU usage per container
- Monitor disk I/O and network traffic
- Collect accelerator metrics (CUDA, ROCm, TPU)
- Report resource availability to Manager
- Enforce resource limits via cgroups

### 3. Local Storage Management
- Manage scratch storage for temporary files
- Mount virtual folders (vfolders) to containers
- Handle file permissions and ownership
- Clean up storage after session termination

### 4. Service Port Management
- Expose container service ports (Jupyter, SSH, TensorBoard, etc.)
- Forward traffic from App Proxy to containers
- Manage port allocation from configured ranges
- Handle SSL/TLS termination when needed

### 5. Agent Status Reporting
- Periodically report agent status to Manager
- Send heartbeat signals indicating availability
- Update occupied and available resource slots
- Report errors and exceptional conditions

## Architecture

```
┌────────────────────────────────────────┐
│         Agent Server (agent.py)        │
├────────────────────────────────────────┤
│    Container Backend (docker/)         │  ← Docker
├────────────────────────────────────────┤
│  Resource Monitor (watcher/ stats.py)  │  ← CPU, GPU, Memory
├────────────────────────────────────────┤
│   Storage Manager (scratch.py fs.py)   │  ← Local and mounted storage
├────────────────────────────────────────┤
│      Plugin System (plugin/)           │  ← Accelerator plugins
└────────────────────────────────────────┘
```

## Directory Structure

```
agent/
├── docker/              # Docker container backend
│   ├── agent.py        # Docker-specific agent implementation
│   └── resources.py    # Docker resource management
├── watcher/             # Resource monitoring
│   ├── cpu.py          # CPU monitoring
│   ├── mem.py          # Memory monitoring
│   └── gpu.py          # GPU monitoring
├── plugin/              # Accelerator plugins
│   ├── cuda.py         # NVIDIA CUDA support
│   ├── rocm.py         # AMD ROCm support
│   └── tpu.py          # Google TPU support
├── observer/            # Metrics observers
│   └── stat.py         # Statistics collection
├── cli/                 # CLI commands
├── config/              # Configuration
├── agent.py             # Main agent logic
├── server.py            # RPC server entry point
├── kernel.py            # Kernel (container) management
├── resources.py         # Resource allocation
├── stats.py             # Statistics aggregation
├── scratch.py           # Scratch storage management
└── fs.py                # Filesystem utilities
```

## Core Concepts

### Kernels
Kernels represent running containers executing computing sessions:
- **Kernel ID**: Unique identifier for the container
- **Session ID**: Associated session in Manager
- **Image**: Container image (e.g., `lablup/python:3.11-ubuntu20.04`)
- **Resources**: Allocated CPU, memory, GPU slots
- **Status**: PREPARING, RUNNING, RESTARTING, TERMINATING, TERMINATED
- **Service Ports**: Exposed ports for services (Jupyter, SSH, etc.)

### Container Backend
The Agent supports multiple container runtimes:
- **Docker**: Standard Docker containers

The Docker backend implements these interfaces:
- `create_kernel()`: Create new container
- `destroy_kernel()`: Remove container
- `restart_kernel()`: Restart container
- `get_kernel_status()`: Query container status
- `execute_code()`: Execute code in container

### Resource Allocation
Resources are allocated from the agent's capacity:
- **CPU Slots**: Measured in cores (e.g., 2.0 cores)
- **Memory Slots**: Measured in bytes (e.g., 4GB)
- **GPU Slots**: Number of GPU devices (e.g., 1 GPU)
- **Accelerator Slots**: Custom accelerator resources

The Agent tracks:
- **Total Capacity**: Maximum available resources
- **Occupied Slots**: Currently allocated resources
- **Available Slots**: Remaining resources for new sessions

### Scratch Storage
Each kernel receives local scratch storage:
- **Location**: `/tmp/backend.ai/scratches/{kernel_id}`
- **Quota**: Configurable size limit per kernel
- **Cleanup**: Automatically removed after kernel termination
- **Performance**: Local SSD for fast I/O

### Virtual Folder Mounting
VFolders are mounted to containers at runtime:
- **Mount Point**: `/home/work/{vfolder_name}`
- **Permissions**: RO, RW, or RW-DELETE
- **Backend**: Storage Proxy manages actual storage
- **Protocol**: NFS, SMB, or direct mount

## Resource Monitoring

### CPU Monitoring
- Track per-container CPU usage via cgroups
- Measure CPU time in user and system modes
- Calculate CPU utilization percentages
- Enforce CPU quotas and limits

### Memory Monitoring
- Track RSS (Resident Set Size) per container
- Measure cache and swap usage
- Detect OOM (Out-of-Memory) conditions
- Enforce memory limits via cgroups

### GPU Monitoring
- Query NVIDIA GPUs via NVML (nvidia-ml-py)
- Query AMD GPUs via ROCm SMI
- Track GPU utilization and memory usage
- Measure GPU temperature and power consumption

### Disk I/O Monitoring
- Track read/write operations per container
- Measure I/O bandwidth usage
- Monitor disk space consumption
- Enforce I/O throttling when configured

## Plugin System

The Agent uses plugins for accelerator support:

### CUDA Plugin
- Detect NVIDIA GPUs via `nvidia-smi`
- Allocate GPU devices to containers
- Set `CUDA_VISIBLE_DEVICES` environment variable
- Monitor GPU metrics via NVML

### ROCm Plugin
- Detect AMD GPUs via `rocm-smi`
- Allocate GPU devices to containers
- Set `HIP_VISIBLE_DEVICES` environment variable
- Monitor GPU metrics via ROCm

### TPU Plugin
- Detect Google TPUs
- Configure TPU access for TensorFlow
- Monitor TPU utilization

## Communication Protocols

### Manager → Agent (ZeroMQ RPC)
- **Port**: 6011 (default)
- **Protocol**: ZeroMQ request-response
- **Operations**:
  - `create_kernel`: Create new container
  - `destroy_kernel`: Terminate container
  - `restart_kernel`: Restart container
  - `execute_code`: Execute code in container
  - `get_status`: Query agent and kernel status

### Agent → Manager (HTTP Watcher API)
- **Port**: 6009 (default)
- **Protocol**: HTTP
- **Operations**:
  - Heartbeat signals
  - Resource usage reporting
  - Kernel status updates
  - Error notifications

### Agent → Storage Proxy
- **Protocol**: HTTP or NFS/SMB
- **Operations**:
  - Mount vfolder
  - Unmount vfolder
  - Query vfolder metadata

## Container Execution Flow

```
1. Manager sends create_kernel RPC
   ↓
2. Agent validates resource availability
   ↓
3. Agent pulls container image (if needed)
   ↓
4. Agent creates scratch directory
   ↓
5. Agent mounts vfolders via Storage Proxy
   ↓
6. Agent creates container with resources
   ↓
7. Agent starts container and runs init script
   ↓
8. Agent registers service ports
   ↓
9. Agent reports kernel status to Manager
   ↓
10. Container runs until termination
   ↓
11. Agent cleans up resources upon termination
```

## Service Ports

Containers can expose service ports:
- **Jupyter Notebook**: Port 8080 (HTTP)
- **Jupyter Lab**: Port 8090 (HTTP)
- **SSH**: Port 2200 (TCP)
- **TensorBoard**: Port 6006 (HTTP)
- **Custom Services**: User-defined ports

Service ports are:
- **Allocated** from configured range (default 30000-31000)
- **Registered** with App Proxy for external access
- **Forwarded** from agent host to container
- **Cleaned up** upon container termination

## Configuration

See `configs/agent/halfstack.toml` for configuration file examples.

### Key Configuration Items

**Agent Basic Settings**:
- Agent ID and region information
- Resource slot definitions
- Container runtime configuration (Docker)

**Storage Settings**:
- Scratch storage root path
- Per-session quota

**Service Port Settings**:
- Container port range

**Resource Monitoring**:
- Watcher interval settings

## Infrastructure Dependencies

### Required Infrastructure

#### Container Runtime
- **Purpose**: Container creation and management
- **Supported Runtimes**: Docker

#### Redis (Event and State Management)
- **Purpose**:
  - Send heartbeat events (for Manager registration)
  - Update agent status
  - Manage background tasks
- **Note**: Agent registers with Manager via Redis and receives RPC commands from Manager.

#### Manager Connection
- **Protocol**: ZeroMQ RPC (6011), HTTP Watcher API (6009)
- **Purpose**: Receive session commands from Manager, report status
- **Note**: Agent only receives commands from Manager, not directly from users or other components.

#### etcd (Global Configuration)
- **Purpose**:
  - Retrieve global configuration (container registry, storage volumes, etc.)
  - Auto-discover Manager address

### Optional Infrastructure (Observability)

Optional infrastructure for Agent monitoring.

#### Prometheus (Metrics Collection)
- **Purpose**:
  - Monitor agent resource utilization
  - Collect kernel (container) metrics
  - Track heartbeat and health status
- **HTTP Service Port**: 6003 (separate from RPC port 6011)
- **Exposed Endpoint**: `http://localhost:6003/metrics`
- **Key Metrics**:
  - `backendai_agent_heartbeat` - Last heartbeat timestamp
  - `backendai_agent_cpu_usage` - CPU usage percentage
  - `backendai_agent_mem_usage` - Memory usage (bytes)
  - `backendai_agent_gpu_usage` - GPU usage percentage
  - `backendai_kernel_count` - Number of running kernels
- **Service Discovery**: Automatically registered via Manager's HTTP SD
  - Agent auto-registers with Manager at startup
  - Manager provides agent information to Prometheus SD

#### Loki (Log Aggregation)
- **Purpose**:
  - Centralized agent log collection
  - Aggregate kernel execution logs
  - Track errors and debugging information
- **Log Transmission Methods**:
  - Direct HTTP API usage
  - Use Promtail or Fluent Bit
- **Log Labels**:
  - `agent_id` - Agent identifier
  - `kernel_id` - Kernel identifier
  - `level` - Log level

#### Container Runtime Metrics
- **Docker**: Container metrics via cAdvisor or Docker API
- **cgroups**: Direct cgroups filesystem reading

### Container Runtime Requirements

#### Docker
- **Minimum Version**: Docker 20.10+
- **Required Features**:
  - cgroups v2 support (resource limits)
  - GPU support (NVIDIA Docker Runtime or CDI)

### Storage Requirements

#### Local Scratch Storage
- **Purpose**: Temporary files, build cache
- **Recommended Specifications**:
  - Fast SSD (NVMe recommended)
  - Minimum 100GB free space
  - Configurable per-session quota

#### VFolder Mount
- **Protocol**: NFS, SMB, direct mount
- **Dependencies**: Storage Proxy or direct storage access
- **Recommendations**:
  - High-performance network (10GbE or higher)
  - Low-latency storage

### Halfstack Configuration

**Recommended**: Use the `./scripts/install-dev.sh` script for development environment setup.

#### Starting Development Environment
```bash
# Setup development environment via script (recommended)
./scripts/install-dev.sh

# Start Agent
./backend.ai ag start-server
```

#### Observability Integration
Agent automatically registers with Prometheus via Manager's Service Discovery.

## Metrics and Monitoring

### Prometheus Metrics
Agent exposes metrics:

#### RPC Related Metrics
Metrics related to processing RPC commands received from Manager.

**`backendai_rpc_requests`** (Counter)
- **Description**: Total RPC requests processed by the agent
- **Labels**:
  - `method`: RPC method name (e.g., "create_kernel", "destroy_kernel", "execute_code")
- Tracks processing frequency by Manager command type

**`backendai_rpc_failure_requests`** (Counter)
- **Description**: Failed RPC request count
- **Labels**:
  - `method`: RPC method name that failed
  - `exception`: Exception class name (e.g., "ContainerNotFound", "ResourceExhausted")
- Analyzes RPC command processing failure causes

**`backendai_rpc_request_duration_seconds`** (Histogram)
- **Description**: RPC request processing time in seconds
- **Labels**:
  - `method`: RPC method name
- **Buckets**: [0.001, 0.01, 0.1, 0.5, 1, 2, 5, 10, 30, 60] seconds
- Measures performance of RPC commands like kernel creation/deletion

#### Resource Utilization Metrics
Metrics related to container and hardware resource usage.

**`backendai_container_utilization`** (Gauge)
- **Description**: Container resource utilization per kernel
- **Labels**:
  - `container_metric_name`: Metric type (cpu_used, cpu_util, mem, cuda_mem, cuda_util, net_rx, net_tx)
  - `agent_id`: Agent identifier
  - `kernel_id`: Kernel identifier (Backend.AI's container wrapper, not the actual container ID)
  - `session_id`: Session identifier
  - `user_id`: User UUID who owns the session
  - `project_id`: Project UUID that the session belongs to
  - `value_type`: Value interpretation (current, capacity, pct)
- Tracks resource usage per kernel (CPU, memory, GPU, network, etc.)

**`backendai_device_utilization`** (Gauge)
- **Description**: Physical device utilization on the agent node
- **Labels**:
  - `device_metric_name`: Metric type (cpu_util, mem, cuda_mem, cuda_util, cuda_temp, cuda_power)
  - `agent_id`: Agent identifier
  - `device_id`: Hardware device identifier (e.g., GPU index)
  - `value_type`: Value interpretation (current, capacity, pct)
- Tracks hardware device usage (GPU, CPU, memory) on the agent node

#### Container Lifecycle Synchronization Metrics
Metrics related to container state synchronization operations between Manager and Agent.

**`backendai_sync_container_lifecycle_trigger_count`** (Counter)
- **Description**: Number of times container lifecycle sync was triggered
- **Labels**:
  - `agent_id`: Agent identifier
- Frequency of container state synchronization between Manager and Agent

**`backendai_sync_container_lifecycle_success_count`** (Counter)
- **Description**: Number of successfully synchronized containers
- **Labels**:
  - `agent_id`: Agent identifier
- Number of successfully synchronized kernels

**`backendai_sync_container_lifecycle_failure_count`** (Counter)
- **Description**: Number of failed container synchronization attempts
- **Labels**:
  - `agent_id`: Agent identifier
  - `exception`: Exception class name causing the failure
- Tracks synchronization failure causes

#### Statistics Collection Task Metrics
Metrics related to node, container, and process level statistics collection.

**`backendai_stat_task_trigger_count`** (Counter)
- **Description**: Number of times statistics collection was triggered
- **Labels**:
  - `agent_id`: Agent identifier
  - `stat_scope`: Statistics scope (node, container, process)
- Collection frequency by Node/Container/Process level

**`backendai_stat_task_success_count`** (Counter)
- **Description**: Number of successful statistics collection operations
- **Labels**:
  - `agent_id`: Agent identifier
  - `stat_scope`: Statistics scope (node, container, process)
- Tracks statistics collection success rate

**`backendai_stat_task_failure_count`** (Counter)
- **Description**: Number of failed statistics collection operations
- **Labels**:
  - `agent_id`: Agent identifier
  - `stat_scope`: Statistics scope (node, container, process)
  - `exception`: Exception class name causing the failure
- Analyzes statistics collection failure causes

### Prometheus Query Examples

The following examples demonstrate common Prometheus queries for Agent metrics. Note that Counter metrics use the `_total` suffix and Histogram metrics use `_bucket`, `_sum`, `_count` suffixes in actual queries.

**Important Notes:**
- When using `increase()` or `rate()` functions, the time range must be at least 2-4x longer than your Prometheus scrape interval to get reliable data. If the time range is too short, metrics may not appear or show incomplete data.
- Default Prometheus scrape interval is typically 15s-30s
- **Time range selection trade-offs**:
  - Shorter ranges (e.g., `[1m]`): Detect changes faster with more granular data, but more sensitive to noise and short-term fluctuations
  - Longer ranges (e.g., `[5m]`): Smoother graphs with reduced noise, better for identifying trends, but slower to detect sudden changes
  - For real-time alerting: Use shorter ranges like `[1m]` or `[2m]`
  - For dashboards and trend analysis: Use longer ranges like `[5m]` or `[10m]`

#### RPC Request Monitoring

**RPC Request Rate by Method**

Monitor RPC request rate by method. This shows how frequently the Manager sends commands to agents. Use this to understand agent workload and command distribution.

```promql
sum(rate(backendai_rpc_requests_total{service_group="$service_groups"}[5m])) by (method)
```

**RPC Failure Rate by Method**

Track RPC failure rate by method and exception type. This helps identify which RPC operations are failing and why.

```promql
sum(rate(backendai_rpc_failure_requests_total{service_group="$service_groups"}[5m])) by (method, exception)
```

**P95 RPC Request Duration**

Calculate P95 RPC request duration by method. This shows how long critical operations like kernel creation take. Use this to identify performance bottlenecks in agent operations.

```promql
histogram_quantile(0.95,
  sum(rate(backendai_rpc_request_duration_seconds_bucket{service_group="$service_groups"}[5m])) by (le, method)
)
```

#### Container Resource Utilization

**Top 5 GPU Memory Usage by Kernel**

Monitor GPU memory usage by kernel (top 5). This identifies kernels consuming the most GPU memory. Note: `topk` returns the top N items at each evaluation time. Over multiple time points, you may see more than 5 unique kernels if different kernels appear in the top 5 at different times.

```promql
topk(5, sum(backendai_container_utilization{container_metric_name="cuda_mem", value_type="current"}) by (kernel_id))
```

**Top 5 GPU Utilization by Kernel**

Monitor GPU utilization by kernel (top 5). This shows which kernels are actively using GPU compute. Note: `topk` returns the top N items at each evaluation time. Over multiple time points, you may see more than 5 unique kernels if different kernels appear in the top 5 at different times.

```promql
topk(5, sum(backendai_container_utilization{container_metric_name="cuda_util", value_type="current"}) by (kernel_id))
```

**Top 5 CPU Utilization Increase by Kernel**

Track CPU utilization increase rate by kernel (top 5). This shows CPU consumption trends over time. Note: Divided by 1000 to convert from per-mille to percentage. `topk` returns the top N items at each evaluation time. Over multiple time points, you may see more than 5 unique kernels if different kernels appear in the top 5 at different times.

```promql
topk(5, sum(increase(backendai_container_utilization{container_metric_name="cpu_util", value_type="current"}[5m])) by (kernel_id)) / 1000
```

**Network RX Traffic by Kernel**

Monitor network RX (receive) traffic by kernel. This tracks inbound network bandwidth usage per container.

```promql
sum(increase(backendai_container_utilization{container_metric_name="net_rx", value_type="current"}[5m])) by (kernel_id)
```

**Network TX Traffic by Kernel**

Monitor network TX (transmit) traffic by kernel. This tracks outbound network bandwidth usage per container.

```promql
sum(increase(backendai_container_utilization{container_metric_name="net_tx", value_type="current"}[5m])) by (kernel_id)
```

#### Device Utilization (Agent-level)

**GPU Memory Usage by Agent**

Monitor GPU memory usage at the agent level. This shows total GPU memory consumption on each agent node.

```promql
sum(backendai_device_utilization{device_metric_name="cuda_mem", value_type="current"}) by (agent_id)
```

**GPU Memory Capacity by Agent**

Monitor GPU memory capacity. This shows the maximum available GPU memory on each agent.

```promql
sum(backendai_device_utilization{device_metric_name="cuda_mem", value_type="capacity"}) by (agent_id)
```

**GPU Utilization Percentage by Agent**

Monitor GPU utilization percentage at the agent level. This shows overall GPU usage across all kernels on the agent.

```promql
sum(backendai_device_utilization{device_metric_name="cuda_util", value_type="current"}) by (agent_id)
```

**CPU Utilization Increase by Agent**

Track CPU utilization increase rate at the agent level. This shows agent-wide CPU consumption trends.

```promql
sum(increase(backendai_device_utilization{device_metric_name="cpu_util", value_type="current"}[5m])) by (agent_id) / 1000
```

**Network RX Traffic by Agent**

Monitor network RX (receive) traffic at the agent level. This tracks total inbound network bandwidth usage on each agent node.

```promql
sum(increase(backendai_device_utilization{device_metric_name="net_rx", value_type="current"}[5m])) by (agent_id)
```

**Network TX Traffic by Agent**

Monitor network TX (transmit) traffic at the agent level. This tracks total outbound network bandwidth usage on each agent node.

```promql
sum(increase(backendai_device_utilization{device_metric_name="net_tx", value_type="current"}[5m])) by (agent_id)
```

#### Container Lifecycle Synchronization

**Container Sync Trigger Frequency**

Monitor container sync trigger frequency. This shows how often the agent synchronizes container state with the Manager.

```promql
sum(rate(backendai_sync_container_lifecycle_trigger_count_total[5m])) by (agent_id)
```

**Container Sync Success Rate**

Track container sync success rate. This helps ensure containers are properly synchronized.

```promql
sum(rate(backendai_sync_container_lifecycle_success_count_total[5m])) by (agent_id)
```

**Container Sync Failure Rate**

Monitor container sync failures. This identifies issues with container state synchronization.

```promql
sum(rate(backendai_sync_container_lifecycle_failure_count_total[5m])) by (agent_id, exception)
```

#### Statistics Collection

**Statistics Collection Trigger Rate**

Monitor statistics collection trigger rate by scope. This shows how frequently statistics are collected at different levels.

```promql
sum(rate(backendai_stat_task_trigger_count_total[5m])) by (agent_id, stat_scope)
```

**Statistics Collection Success Rate**

Track statistics collection success rate. This ensures statistics collection is working properly.

```promql
sum(rate(backendai_stat_task_success_count_total[5m])) by (agent_id, stat_scope)
```

**Statistics Collection Failure Rate**

Monitor statistics collection failures. This identifies issues with resource monitoring.

```promql
sum(rate(backendai_stat_task_failure_count_total[5m])) by (agent_id, stat_scope, exception)
```

### Logs
Agent logs include:
- Kernel creation and termination events
- Resource allocation and release
- Error situations and recovery operations
- RPC request/response tracking

## Development

See [README.md](./README.md) for development setup instructions.

## Related Documentation

- [Manager Component](../manager/README.md) - Session orchestration
- [Storage Proxy Component](../storage/README.md) - Virtual folder management
- [Overall Architecture](../README.md) - System-wide architecture
""",
    'long_description_content_type': 'text/markdown',
    'name': 'backend.ai-agent',
    'namespace_packages': (
    ),
    'package_data': {
        'ai.backend.agent': (
            'VERSION',
            'py.typed',
        ),
        'ai.backend.agent.docker': (
            'backendai-socket-relay.img.aarch64.tar.gz',
            'backendai-socket-relay.img.x86_64.tar.gz',
            'backendai-socket-relay.version.txt',
            'linuxkit-metadata-proxy-worker.aarch64.bin',
            'linuxkit-metadata-proxy-worker.x86_64.bin',
            'linuxkit-nsenter.img.aarch64.tar.gz',
            'linuxkit-nsenter.img.x86_64.tar.gz',
            'linuxkit-nsenter.version.txt',
        ),
    },
    'packages': (
        'ai.backend.agent',
        'ai.backend.agent.cli',
        'ai.backend.agent.config',
        'ai.backend.agent.docker',
        'ai.backend.agent.docker.metadata',
        'ai.backend.agent.dummy',
        'ai.backend.agent.kernel_registry',
        'ai.backend.agent.kubernetes',
        'ai.backend.agent.metrics',
        'ai.backend.agent.observer',
        'ai.backend.agent.plugin',
        'ai.backend.agent.stage',
        'ai.backend.agent.stage.kernel_lifecycle',
        'ai.backend.agent.stage.kernel_lifecycle.docker',
        'ai.backend.agent.stage.kernel_lifecycle.docker.mount',
        'ai.backend.agent.vendor',
        'ai.backend.agent.watcher',
    ),
    'project_urls': {
        'Documentation': 'https://docs.backend.ai/',
        'Source': 'https://github.com/lablup/backend.ai',
    },
    'python_requires': '>=3.13,<3.14',
    'url': 'https://github.com/lablup/backend.ai',
    'version': """25.16.0
""",
    'zip_safe': False,
})
