## Quick Summary
The `utils` directory provides essential utility functions and tools for the Solace Agent Mesh system. It contains both direct utility files for common operations (MIME type handling, caching, message validation, authentication) and a sophisticated `embeds` subdirectory that implements a dynamic expression evaluation system. The utilities work together to provide platform compatibility, security features, data processing capabilities, and dynamic content generation for agent workflows.

## Files and Subdirectories Overview

### Direct Files:
- **`__init__.py`** - Main entry point exporting commonly used utilities like MIME type checking
- **`artifact_utils.py`** - Utilities for working with ADK artifacts, including version resolution
- **`asyncio_macos_fix.py`** - Automatic fix for asyncio subprocess issues on macOS
- **`in_memory_cache.py`** - Thread-safe singleton cache with TTL support
- **`initializer.py`** - Enterprise feature initialization and configuration loading
- **`log_formatters.py`** - Custom logging formatters for platforms like Datadog
- **`message_utils.py`** - Message size calculation and validation utilities
- **`mime_helpers.py`** - MIME type classification and file extension utilities
- **`push_notification_auth.py`** - JWT-based authentication for push notifications

### Subdirectories:
- **`embeds/`** - Dynamic expression evaluation system using `«...»` syntax for mathematical calculations, datetime formatting, UUID generation, and artifact content processing

## Developer API Reference

### Direct Files

#### __init__.py
**Purpose:** Main entry point for the utils package, exporting the most commonly used utility functions
**Import:** `from solace_agent_mesh.common.utils import is_text_based_mime_type`

**Functions:**
- `is_text_based_mime_type(mime_type: Optional[str]) -> bool` - Checks if a MIME type represents text-based content

#### artifact_utils.py
**Purpose:** Common utility functions for working with ADK artifacts
**Import:** `from solace_agent_mesh.common.utils.artifact_utils import get_latest_artifact_version`

**Functions:**
- `get_latest_artifact_version(artifact_service: BaseArtifactService, app_name: str, user_id: str, session_id: str, filename: str) -> Optional[int]` - Resolves the latest version number for a given artifact

#### asyncio_macos_fix.py
**Purpose:** Automatic fix for asyncio subprocess creation issues on macOS (imported for side effects)
**Import:** `from solace_agent_mesh.common.utils import asyncio_macos_fix`

**Functions:**
- `apply_macos_asyncio_fix() -> bool` - Applies the asyncio fix for macOS subprocess support
- `ensure_asyncio_compatibility() -> bool` - Ensures asyncio compatibility for subprocess creation

#### in_memory_cache.py
**Purpose:** Thread-safe singleton in-memory cache with TTL support
**Import:** `from solace_agent_mesh.common.utils.in_memory_cache import InMemoryCache`

**Classes:**
- **`InMemoryCache`** - Singleton cache class
  - `set(key: str, value: Any, ttl: Optional[int] = None) -> None` - Store value with optional TTL
  - `get(key: str, default: Any = None) -> Any` - Retrieve value or default
  - `delete(key: str) -> bool` - Delete specific key
  - `clear() -> bool` - Clear all cached data

#### initializer.py
**Purpose:** Handles initialization of enterprise features if available
**Import:** `from solace_agent_mesh.common.utils.initializer import initialize`

**Functions:**
- `initialize() -> None` - Initializes enterprise features using SAM_AUTHORIZATION_CONFIG environment variable

#### log_formatters.py
**Purpose:** Custom logging formatters for structured output
**Import:** `from solace_agent_mesh.common.utils.log_formatters import DatadogJsonFormatter`

**Classes:**
- **`DatadogJsonFormatter(logging.Formatter)`** - JSON formatter with Datadog-compatible attributes including trace IDs

#### message_utils.py
**Purpose:** Message size calculation and validation utilities
**Import:** `from solace_agent_mesh.common.utils.message_utils import calculate_message_size, validate_message_size`

**Functions:**
- `calculate_message_size(payload: Dict[str, Any]) -> int` - Calculate exact message size using JSON + UTF-8 encoding
- `validate_message_size(payload: Dict[str, Any], max_size_bytes: int, component_identifier: str = "Unknown") -> Tuple[bool, int]` - Validate message doesn't exceed size limits

**Constants:**
- `MAX_UTF8_BYTES_PER_CHARACTER: int` - Maximum UTF-8 bytes per character (4)

#### mime_helpers.py
**Purpose:** MIME type classification and file extension utilities
**Import:** `from solace_agent_mesh.common.utils.mime_helpers import is_text_based_mime_type, get_extension_for_mime_type, is_text_based_file`

**Functions:**
- `is_text_based_mime_type(mime_type: Optional[str]) -> bool` - Check if MIME type is text-based
- `is_text_based_file(mime_type: Optional[str], content_bytes: Optional[bytes] = None) -> bool` - Determine if file is text-based using MIME type and content analysis
- `get_extension_for_mime_type(mime_type: Optional[str], default_extension: str = ".dat") -> str` - Get file extension for MIME type

**Constants:**
- `TEXT_CONTAINER_MIME_TYPES: Set[str]` - Set of non-text/* MIME types that contain text

#### push_notification_auth.py
**Purpose:** JWT-based authentication for push notifications with request integrity verification
**Import:** `from solace_agent_mesh.common.utils.push_notification_auth import PushNotificationSenderAuth, PushNotificationReceiverAuth`

**Classes:**
- **`PushNotificationSenderAuth`** - Handles sending authenticated notifications
  - `generate_jwk() -> None` - Generate RSA key pair for signing
  - `handle_jwks_endpoint(request: Request) -> JSONResponse` - Serve public keys endpoint
  - `send_push_notification(url: str, data: dict[str, Any]) -> None` - Send authenticated notification
  - `verify_push_notification_url(url: str) -> bool` - Verify notification URL
- **`PushNotificationReceiverAuth`** - Handles receiving and verifying notifications
  - `load_jwks(jwks_url: str) -> None` - Load public keys from JWKS endpoint
  - `verify_push_notification(request: Request) -> bool` - Verify notification authenticity

### Subdirectory APIs

#### embeds/
**Purpose:** Comprehensive dynamic expression evaluation system using `«...»` syntax for mathematical calculations, datetime formatting, UUID generation, and artifact content processing with transformation pipelines
**Key Exports:** Main resolution functions, evaluator registry, modifier system, and type constants
**Import Examples:**
```python
from solace_agent_mesh.common.utils.embeds import resolve_embeds_recursively_in_string, evaluate_embed, EMBED_REGEX
from solace_agent_mesh.common.utils.embeds.constants import EARLY_EMBED_TYPES, LATE_EMBED_TYPES
from solace_agent_mesh.common.utils.embeds.types import DataFormat
```

## Complete Usage Guide

### 1. Basic Utility Operations

```python
# Import commonly used utilities
from solace_agent_mesh.common.utils import is_text_based_mime_type
from solace_agent_mesh.common.utils.in_memory_cache import InMemoryCache
from solace_agent_mesh.common.utils.message_utils import validate_message_size, calculate_message_size
from solace_agent_mesh.common.utils.mime_helpers import get_extension_for_mime_type, is_text_based_file

# MIME type checking
if is_text_based_mime_type("application/json"):
    print("JSON is text-based")

# File analysis with content
with open("data.bin", "rb") as f:
    content = f.read()
if is_text_based_file("application/octet-stream", content):
    print("File contains text despite binary MIME type")

# Singleton cache usage
cache = InMemoryCache()
cache.set("user_session", {"user_id": "123", "role": "admin"}, ttl=3600)  # 1 hour TTL
session_data = cache.get("user_session", {})

# Message size validation
payload = {"message": "Hello world", "data": [1, 2, 3], "metadata": {"timestamp": "2024-01-15"}}
is_valid, size = validate_message_size(payload, max_size_bytes=1024, component_identifier="MessageProcessor")
if not is_valid:
    print(f"Message too large: {size} bytes exceeds 1024 byte limit")

# Get appropriate file extension
extension = get_extension_for_mime_type("image/png")  # Returns ".png"
filename = f"image_{uuid.uuid4()}{extension}"
```

### 2. Platform Compatibility and System Initialization

```python
# Early in application startup - import for side effects
from solace_agent_mesh.common.utils import asyncio_macos_fix  # Auto-applies macOS fix
from solace_agent_mesh.common.utils.initializer import initialize

# Initialize enterprise features if available
try:
    initialize()
    print("Enterprise features initialized")
except Exception as e:
    print(f"Running in community mode: {e}")

# Now asyncio subprocess creation works reliably on macOS
import asyncio

async def run_command(cmd: str):
    process = await asyncio.create_subprocess_exec(
        *cmd.split(),
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE
    )
    stdout, stderr = await process.communicate()
    return stdout.decode(), stderr.decode(), process.returncode

# This will work on macOS without NotImplementedError
result = await run_command("echo Hello World")
```

### 3. Structured Logging Setup

```python
import logging
import os
from solace_agent_mesh.common.utils.log_formatters import DatadogJsonFormatter

# Configure structured JSON logging
logger = logging.getLogger("my_application")
handler = logging.StreamHandler()
handler.setFormatter(DatadogJsonFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# Set service name for Datadog
os.environ["SERVICE_NAME"] = "my_agent_service"

# Log with structured data - automatically includes trace IDs if available
logger.info("User action completed", extra={
    "user_id": "user123",
    "action": "file_upload",
    "file_size": 1024,
    "dd.trace_id": "abc123"  # Will be included in JSON output
})

# Output will be JSON with timestamp, level, service, code location, etc.
```

### 4. Secure Push Notification System

```python
from solace_agent_mesh.common.utils.push_notification_auth import (
    PushNotificationSenderAuth, 
    PushNotificationReceiverAuth
)
from starlette.applications import Starlette
from starlette.requests import Request
from starlette.responses import Response, JSONResponse

# Sender setup and usage
sender_auth = PushNotificationSenderAuth()
sender_auth.generate_jwk()  # Generate RSA key pair

async def notify_clients(event_data: dict):
    client_urls = ["https://client1.example.com/webhook", "https://client2.example.com/webhook"]
    
    for url in client_urls:
        # Verify URL accepts notifications
        if await sender_auth.verify_push_notification_url(url):
            # Send authenticated notification
            await sender_auth.send_push_notification(url, {
                "event": "data_updated",
                "timestamp": "2024-01-15T10:30:00Z",
                "data": event_data
            })
        else:
            print(f"Failed to verify URL: {url}")

# Receiver setup
app = Starlette()
receiver_auth = PushNotificationReceiverAuth()

# Load sender's public keys
await receiver_auth.load_jwks("https://sender.example.com/.well-known/jwks.json")

@app.route("/webhook", methods=["POST"])
async def webhook_handler(request: Request):
    try:
        # Verify JWT signature and request integrity
        if await receiver_auth.verify_push_notification(request):
            data = await request.json()
            # Process authenticated notification
            print(f"Received verified notification: {data}")
            return Response("OK")
        else:
            return Response("Unauthorized", status_code=401)
    except Exception as e:
        print(f"Verification failed: {e}")
        return Response("Bad Request", status_code=400)

# JWKS endpoint for sender
@app.route("/.well-known/jwks.json")
async def jwks_endpoint(request: Request):
    return sender_auth.handle_jwks_endpoint(request)
```

### 5. Advanced Embed Processing

```python
from solace_agent_mesh.common.utils.embeds import (
    resolve_embeds_recursively_in_string, 
    evaluate_embed,
    EARLY_EMBED_TYPES,
    LATE_EMBED_TYPES
)

# Set up context for embed resolution
context = {
    "artifact_service": my_artifact_service,
    "session_context": {
        "app_name": "sales_analyzer",
        "user_id": "user123", 
        "session_id": "sess456"
    }
}

# Simple embed resolution for dynamic content
template = """
Sales Report Generated: «datetime:%Y-%m-%d %H:%M:%S»
Total Revenue: $«math:«artifact_content:sales.csv >>> jsonpath:$.total_revenue» * 1.08 | .2f»
Report ID: «uuid:new»
Tax Rate Applied: 8%
"""

# Resolve embeds recursively with safety limits
resolved_content = await resolve_embeds_recursively_in_string(
    text=template,
    context=context,
    resolver_func=evaluate_embed,
    types_to_resolve=EARLY_EMBED_TYPES | LATE_EMBED_TYPES,
    log_identifier="[SalesReport]",
    config={"max_artifact_size": 1024*1024},  # 1MB limit
    max_depth=5
)

# Complex data transformation pipeline
data_analysis = """
Product Analysis:
«artifact_content:products.csv:latest >>> select_cols:name,category,price,sales >>> filter_rows_eq:category:electronics >>> slice_rows:0:10 >>> format:json_pretty»

Summary Statistics:
«artifact_content:summary.txt >>> head:5»

Top Performers:
«artifact_content:sales.json >>> jsonpath:$.top_products[*].name >>> format:text»
"""

analysis_result = await resolve_embeds_recursively_in_string(
    text=data_analysis,
    context=context,
    resolver_func=evaluate_embed,
    types_to_resolve=LATE_EMBED_TYPES,
    log_identifier="[Analysis]",
    config={},
    max_depth=3
)
```

### 6. Integrated Workflow Example

```python
from solace_agent_mesh.common.utils import is_text_based_mime_type
from solace_agent_mesh.common.utils.in_memory_cache import InMemoryCache
from solace_agent_mesh.common.utils.message_utils import validate_message_size
from solace_agent_mesh.common.utils.embeds import resolve_embeds_recursively_in_string, evaluate_embed
from solace_agent_mesh.common.utils.push_notification_auth import PushNotificationSenderAuth
import hashlib

async def process_and_notify_report(user_id: str, template: str, context: dict, notification_urls: list):
    """Complete workflow integrating

# content_hash: 0f48341a035bc4fbdb720885186fe04075f8fbca7ddeee6a151b00b4c667c47a
