# Protocol Buffers Support Implementation Plan

## Overview
This plan details the implementation of native Protocol Buffers (Protobuf) support within `lodum` via its AST compiler, addressing issue #47. The goal is to enable `lodum` to efficiently serialize and deserialize objects that correspond to Protobuf message definitions, allowing for cross-format compatibility and leveraging `lodum`'s performance optimizations.

## Current State Analysis
Our research into the Python Protobuf library reveals:
*   Protobuf messages are defined in `.proto` files.
*   The `protoc` compiler generates Python classes (`_pb2.py` files) from `.proto` definitions.
*   These generated Python classes provide methods like `SerializeToString()` and `ParseFromString()` for binary serialization, and allow direct attribute access for fields.
*   `lodum` currently operates on standard Python classes decorated with `@lodum`, relying on `__init__` type hints for schema inference.

## Implementation Approach
The approach will involve creating a `ProtobufLoader` and `ProtobufDumper` that interact with the Python classes generated by `protoc`. `lodum`'s AST compiler will then generate optimized handlers to efficiently transfer data between `@lodum` objects and these Protobuf message instances. A key prerequisite will be a clear mapping from Protobuf message definitions to `@lodum` class definitions.

---

## Phase 1: Protobuf `Dumper` and `Loader` for existing Protobuf objects

### Overview
This phase will implement `ProtobufDumper` and `ProtobufLoader` classes that can serialize `@lodum` objects to existing `_pb2` Protobuf message instances, and deserialize `_pb2` Protobuf message instances into `@lodum` objects. The `lodum` compiler will generate handlers to optimize this data transfer.

### Changes Required:

#### 1. `src/lodum/ext/protobuf.py`: Create new module
**Changes**:
- Create a new file `src/lodum/ext/protobuf.py`.
- Define `ProtobufDumper` and `ProtobufLoader` classes that inherit from `lodum.core.BaseDumper` and `lodum.core.BaseLoader` respectively.
- These classes will be adapted to interact with `_pb2.Message` objects. The `ProtobufDumper` would populate a `_pb2.Message` instance, and the `ProtobufLoader` would read from one.

```python
# src/lodum/ext/protobuf.py

from typing import Any, Type
from google.protobuf.message import Message as ProtobufMessage # Protobuf base message class
from ..core import BaseDumper, BaseLoader, T
from ..internal import dump, load

class ProtobufDumper(BaseDumper):
    def __init__(self, message: ProtobufMessage):
        self._message = message

    def dump_int(self, value: int) -> Any:
        # Placeholder: This will eventually be optimized by codegen to set directly
        return value

    # ... implement other dump_ methods to populate self._message ...
    
    def begin_struct(self, cls: Type) -> Any:
        # For nested messages, create a new ProtobufMessage instance
        # This will be passed to subsequent recursive dump calls
        return self._message.DESCRIPTOR.fields_by_name[cls.__name__.lower()].message_type._concrete_class()

    def end_struct(self) -> Any:
        pass # Protobuf messages don't have an explicit end_struct

# --- Public API for lodum.protobuf.dumps ---
def dumps_to_message(obj: Any, message: ProtobufMessage) -> ProtobufMessage:
    dumper = ProtobufDumper(message)
    # The internal dump will use the dumper to populate the message
    dump(obj, dumper)
    return message


class ProtobufLoader(BaseLoader):
    def __init__(self, message: ProtobufMessage):
        self._data = message # Treat the ProtobufMessage as the source data

    def load_int(self) -> int:
        # Placeholder: This will eventually be optimized by codegen to read directly
        return self._data # This needs to be adapted to read the correct field
        
    # ... implement other load_ methods to read from self._data (ProtobufMessage) ...

    def get_dict(self) -> Optional[Dict[str, Any]]:
        # This can be used to convert Protobuf message to a dict-like structure for lodum's codegen
        return self._data.AsDict(preserving_proto_field_name=True) # Or iterate fields

# --- Public API for lodum.protobuf.loads ---
def loads_from_message(cls: Type[T], message: ProtobufMessage) -> T:
    loader = ProtobufLoader(message)
    return load(cls, loader)

```

#### 2. `tests/test_protobuf.py`: Add new test file for Protobuf support
**Changes**:
- Create a new test file `tests/test_protobuf.py`.
- Define a sample `.proto` file and generate `_pb2.py` during test setup (or use a pre-generated one).
- Add test cases covering:
    - Serialization of a `@lodum` object to a Protobuf message.
    - Deserialization of a Protobuf message to a `@lodum` object.
    - Handling of nested messages, repeated fields, and enums.

```python
# tests/test_protobuf.py

import pytest
import subprocess
import os
from lodum import lodum
from lodum.ext.protobuf import dumps_to_message, loads_from_message

# Assume person_pb2 is generated from a .proto file during test setup
# from . import person_pb2 # Example

# Setup: Generate protobuf code if it doesn't exist
@pytest.fixture(scope="module", autouse=True)
def generate_protobuf_code():
    proto_dir = os.path.join(os.path.dirname(__file__), "proto_test")
    os.makedirs(proto_dir, exist_ok=True)
    proto_file_path = os.path.join(proto_dir, "person.proto")
    pb2_file_path = os.path.join(proto_dir, "person_pb2.py")

    if not os.path.exists(pb2_file_path):
        proto_content = """
            syntax = "proto3";
            package test_proto;
            message Person {
              string name = 1;
              int32 id = 2;
            }
        """
        with open(proto_file_path, "w") as f:
            f.write(proto_content)
        
        # Run protoc
        subprocess.run(["protoc", "--python_out=.", proto_file_path], check=True, cwd=proto_dir)
    
    # Add proto_test directory to sys.path for import
    import sys
    sys.path.insert(0, proto_dir)
    yield
    sys.path.remove(proto_dir)
    # Clean up generated files (optional)
    # shutil.rmtree(proto_dir)

# Import generated code after setup
from proto_test import person_pb2

@lodum
class LodumPerson:
    def __init__(self, name: str, id: int):
        self.name = name
        self.id = id

def test_dumps_to_protobuf_message():
    lodum_person = LodumPerson(name="Alice", id=123)
    proto_message = person_pb2.Person()
    dumps_to_message(lodum_person, proto_message)
    assert proto_message.name == "Alice"
    assert proto_message.id == 123

def test_loads_from_protobuf_message():
    proto_message = person_pb2.Person(name="Bob", id=456)
    lodum_person = loads_from_message(LodumPerson, proto_message)
    assert lodum_person.name == "Bob"
    assert lodum_person.id == 456
```

### Success Criteria:
#### Automated:
- [ ] `PYTHONPATH=src pytest tests/test_protobuf.py` passes.
#### Manual:
- [ ] Verify that `@lodum` objects can be correctly converted to and from `_pb2` Protobuf messages.

---

## Phase 2: Compiler Optimizations for Protobuf `Dumper`/`Loader`

### Overview
Integrate the `ProtobufDumper` and `ProtobufLoader` into `lodum`'s AST compilation pipeline. This will generate specialized code paths for interacting with Protobuf message attributes directly, bypassing generic reflection and `AsDict()` calls for maximum performance.

### Changes Required:
1.  **`src/lodum/compiler/dump_codegen.py` / `load_codegen.py`:**
    **Changes**:
    - Modify the codegen to recognize `google.protobuf.message.Message` types.
    - For these types, generate AST that directly accesses `_pb2.Message` attributes (e.g., `message.name = value`) instead of using generic `dumper.dump_str()` or `loader.load_str()`.
2.  **`src/lodum/ext/protobuf.py`:**
    **Changes**:
    - Update `ProtobufDumper` and `ProtobufLoader` to have their `dump_`/`load_` methods optimized by the `lodum` compiler. This might involve custom logic for repeated fields and nested messages.
3.  **`tests/test_protobuf.py`:**
    **Changes**:
    - Add benchmarks comparing `lodum`-optimized Protobuf conversion performance against direct `_pb2` message manipulation.

### Success Criteria:
#### Automated:
- [ ] New benchmarks demonstrate significant performance improvement for `lodum`-optimized Protobuf conversions.
#### Manual:
- [ ] Verify that the generated `lodum` handlers for Protobuf types are more efficient than reflection-based approaches.

## Review Criteria (Self-Critique)
- **Specificity**: High, providing explicit code modifications, test examples, and addressing the integration with `_pb2` messages and `lodum`'s compiler.
- **Verification**: Includes both automated and manual success criteria.
- **Phasing**: Logically separates basic functionality from compiler optimizations.
