# Creating HSHM Data Structures: A Complete Guide

## Overview

This guide explains how to create custom shared memory data structures using the Hermes Shared Memory (HSHM) framework. HSHM data structures are designed to work seamlessly in shared memory environments while maintaining compatibility with standard C++ containers.

## Core Concepts

### 1. ShmContainer Base Class

All HSHM data structures inherit from `ShmContainer`, which marks them as shared memory-compatible:

```cpp
namespace hshm::ipc {
    class ShmContainer {};
}
```

### 2. Template System Architecture

HSHM uses an advanced macro-based template system with several key components:

#### Template Parameters
```cpp
#define HSHM_CLASS_TEMPL typename AllocT, hipc::ShmFlagField HSHM_FLAGS
#define HSHM_CLASS_TEMPL_ARGS AllocT, HSHM_FLAGS
```

**AllocT**: The allocator type (e.g., `StackAllocator`, `ScalablePageAllocator`)  
**HSHM_FLAGS**: Bitfield controlling behavior (private vs shared memory, destructible vs undestructible)

#### Flag System
```cpp
struct ShmFlag {
    CLS_CONST ShmFlagField kIsPrivate = BIT_OPT(ShmFlagField, 0);        // Use process-local memory
    CLS_CONST ShmFlagField kIsUndestructable = BIT_OPT(ShmFlagField, 1); // Skip destructor calls
    CLS_CONST ShmFlagField kIsThreadLocal = kIsPrivate | kIsUndestructable;
};
```

### 3. The IS_SHM_ARCHIVEABLE Macro

This critical macro determines if a type is shared memory compatible:

```cpp
#define IS_SHM_ARCHIVEABLE(T) \
  std::is_base_of<hshm::ipc::ShmContainer, TYPE_UNWRAP(T)>::value
```

**Usage**: Enables compile-time decisions about memory operations:
- **true**: Type inherits from `ShmContainer` - use shared memory serialization
- **false**: Regular POD type - use simple `memcpy`

### 4. The TYPE_UNWRAP and __TU Macros

These macros handle template parameters with complex types:

```cpp
#define TYPE_UNWRAP(X) ESC(ISH X)
#define __TU(X) TYPE_UNWRAP(X)
```

**Purpose**: Remove parentheses from macro parameters containing commas
**Example**: `__TU((std::vector<int, MyAlloc>))` → `std::vector<int, MyAlloc>`

### 5. The HIPC_CONTAINER_TEMPLATE Macro

This is the core macro that generates the shared memory infrastructure:

```cpp
#define HIPC_CONTAINER_TEMPLATE(CLASS_NAME, CLASS_NEW_ARGS)                   \
  HIPC_CONTAINER_TEMPLATE_BASE(                                               \
      CLASS_NAME,                                                             \
      (__TU(CLASS_NAME) < __TU(CLASS_NEW_ARGS), HSHM_CLASS_TEMPL_ARGS >),     \
      (__TU(CLASS_NAME) < __TU(CLASS_NEW_ARGS), HSHM_CLASS_TEMPL_TLS_ARGS >), \
      (__TU(CLASS_NAME) < __TU(CLASS_NEW_ARGS), AllocT, OTHER_FLAGS >))
```

**Generated functionality**:
- Allocator management
- Constructor/destructor handling
- Thread-local storage support
- Memory context management
- Pointer conversion utilities

## Step-by-Step Guide: Creating a Custom Data Structure

### Step 1: Basic Structure Setup

```cpp
#include "hermes_shm/data_structures/internal/shm_internal.h"

namespace hshm::ipc {

// Forward declaration
template <typename T, HSHM_CLASS_TEMPL_WITH_DEFAULTS>
class my_container;

// Define macros for container template
#define CLASS_NAME my_container
#define CLASS_NEW_ARGS T

template <typename T, HSHM_CLASS_TEMPL>
class my_container : public ShmContainer {
 public:
  HIPC_CONTAINER_TEMPLATE((CLASS_NAME), (CLASS_NEW_ARGS))

 private:
  // Your data members here
  OffsetPointer data_ptr_;
  size_t size_;
  size_t capacity_;

 public:
  // Your implementation here...
};

// Cleanup macros
#undef CLASS_NAME
#undef CLASS_NEW_ARGS

} // namespace hshm::ipc
```

### Step 2: Implement Required Interface Methods

Every HSHM data structure must implement:

#### IsNull() and SetNull()
```cpp
HSHM_INLINE_CROSS_FUN bool IsNull() const { 
    return data_ptr_.IsNull(); 
}

HSHM_INLINE_CROSS_FUN void SetNull() {
    data_ptr_.SetNull();
    size_ = 0;
    capacity_ = 0;
}
```

#### shm_destroy_main()
```cpp
HSHM_INLINE_CROSS_FUN void shm_destroy_main() {
    // Clean up all allocated elements
    clear_all_elements();
    
    // Free the main data buffer
    if (!data_ptr_.IsNull()) {
        CtxAllocator<AllocT> alloc = GetCtxAllocator();
        FullPtr<void, OffsetPointer> full_ptr(
            alloc->template Convert<void>(data_ptr_), data_ptr_);
        alloc->template Free<void>(alloc.ctx_, full_ptr);
    }
}
```

### Step 3: Implement Constructors

#### Default Constructor
```cpp
HSHM_CROSS_FUN explicit my_container() {
    init_shm_container(HSHM_MEMORY_MANAGER->GetDefaultAllocator<AllocT>());
    SetNull();
}

HSHM_CROSS_FUN explicit my_container(const hipc::CtxAllocator<AllocT> &alloc) {
    init_shm_container(alloc);
    SetNull();
}
```

#### Parameterized Constructor
```cpp
template <typename... Args>
HSHM_CROSS_FUN explicit my_container(size_t initial_capacity, Args &&...args) {
    shm_init(HSHM_MEMORY_MANAGER->GetDefaultAllocator<AllocT>(), 
             initial_capacity, std::forward<Args>(args)...);
}

template <typename... Args>
HSHM_CROSS_FUN void shm_init(const CtxAllocator<AllocT> &alloc, 
                             size_t initial_capacity, Args &&...args) {
    init_shm_container(alloc);
    SetNull();
    reserve(initial_capacity);
    // Initialize with args...
}
```

### Step 4: Implement Copy Semantics

The HSHM framework provides patterns for handling different copy scenarios:

```cpp
// Copy constructor
HSHM_CROSS_FUN explicit my_container(const my_container &other) {
    init_shm_container(other.GetCtxAllocator());
    SetNull();
    shm_strong_copy_main(other);
}

// Main copy implementation
template <typename ContainerT>
HSHM_CROSS_FUN void shm_strong_copy_main(const ContainerT &other) {
    reserve(other.size());
    
    if constexpr (std::is_pod<T>() && !IS_SHM_ARCHIVEABLE(T)) {
        // Fast path: Plain old data types
        memcpy(data(), other.data(), other.size() * sizeof(T));
        size_ = other.size();
    } else {
        // Slow path: Complex types requiring construction
        for (const auto &item : other) {
            emplace_back(item);
        }
    }
}
```

### Step 5: Implement Core Operations

#### Memory Management
```cpp
template <typename... Args>
HSHM_CROSS_FUN void reserve(size_t new_capacity, Args &&...args) {
    if (new_capacity <= capacity_) return;
    
    CtxAllocator<AllocT> alloc = GetCtxAllocator();
    
    if constexpr (std::is_pod<T>() && !IS_SHM_ARCHIVEABLE(T)) {
        // Use reallocation for POD types
        if (!data_ptr_.IsNull()) {
            FullPtr<T, OffsetPointer> old_ptr(
                alloc->template Convert<T>(data_ptr_), data_ptr_);
            auto new_ptr = alloc->template ReallocateObjs<T>(
                alloc.ctx_, old_ptr, new_capacity);
            data_ptr_ = new_ptr.shm_;
        } else {
            auto new_ptr = alloc->template AllocateObjs<T, OffsetPointer>(
                alloc.ctx_, new_capacity);
            data_ptr_ = new_ptr.shm_;
        }
    } else {
        // Manual copy for complex types
        auto new_ptr = alloc->template AllocateObjs<T, OffsetPointer>(
            alloc.ctx_, new_capacity);
        T* new_data = new_ptr.ptr_;
        OffsetPointer new_data_ptr = new_ptr.shm_;
        
        // Move existing elements
        if (!data_ptr_.IsNull()) {
            T* old_data = alloc->template Convert<T>(data_ptr_);
            for (size_t i = 0; i < size_; ++i) {
                new (new_data + i) T(std::move(old_data[i]));
                old_data[i].~T();
            }
            
            FullPtr<void, OffsetPointer> old_ptr(
                alloc->template Convert<void>(data_ptr_), data_ptr_);
            alloc->template Free<void>(alloc.ctx_, old_ptr);
        }
        
        data_ptr_ = new_data_ptr;
    }
    
    capacity_ = new_capacity;
}
```

#### Element Access
```cpp
HSHM_INLINE_CROSS_FUN T& operator[](size_t index) {
    return data()[index];
}

HSHM_INLINE_CROSS_FUN const T& operator[](size_t index) const {
    return data()[index];
}

HSHM_INLINE_CROSS_FUN T* data() {
    return GetAllocator()->template Convert<T>(data_ptr_);
}

HSHM_INLINE_CROSS_FUN const T* data() const {
    return GetAllocator()->template Convert<T>(data_ptr_);
}
```

### Step 6: Advanced Features

#### Iterator Support
```cpp
// Define iterator types
typedef T* iterator_t;
typedef const T* const_iterator_t;

// Implement iterator methods
HSHM_INLINE_CROSS_FUN iterator_t begin() { return data(); }
HSHM_INLINE_CROSS_FUN iterator_t end() { return data() + size_; }
HSHM_INLINE_CROSS_FUN const_iterator_t cbegin() const { return data(); }
HSHM_INLINE_CROSS_FUN const_iterator_t cend() const { return data() + size_; }
```

#### Serialization Support
```cpp
template <typename Ar>
HSHM_CROSS_FUN void save(Ar &ar) const {
    ar << size_;
    for (size_t i = 0; i < size_; ++i) {
        ar << (*this)[i];
    }
}

template <typename Ar>
HSHM_CROSS_FUN void load(Ar &ar) {
    size_t loaded_size;
    ar >> loaded_size;
    reserve(loaded_size);
    size_ = loaded_size;
    for (size_t i = 0; i < size_; ++i) {
        ar >> (*this)[i];
    }
}
```

## Complete Example: Custom Stack Container

Here's a complete implementation of a stack data structure:

```cpp
#include "hermes_shm/data_structures/internal/shm_internal.h"
#include "hermes_shm/data_structures/serialization/serialize_common.h"

namespace hshm::ipc {

// Forward declaration
template <typename T, HSHM_CLASS_TEMPL_WITH_DEFAULTS>
class stack;

// Template helper macros
#define CLASS_NAME stack
#define CLASS_NEW_ARGS T

/**
 * A stack data structure optimized for shared memory
 */
template <typename T, HSHM_CLASS_TEMPL>
class stack : public ShmContainer {
 public:
  HIPC_CONTAINER_TEMPLATE((CLASS_NAME), (CLASS_NEW_ARGS))

 private:
  OffsetPointer data_ptr_;
  size_t size_;
  size_t capacity_;

 public:
  /**====================================
   * Default Constructors
   * ===================================*/
   
  HSHM_CROSS_FUN explicit stack() {
    init_shm_container(HSHM_MEMORY_MANAGER->GetDefaultAllocator<AllocT>());
    SetNull();
  }

  HSHM_CROSS_FUN explicit stack(const hipc::CtxAllocator<AllocT> &alloc) {
    init_shm_container(alloc);
    SetNull();
  }

  HSHM_CROSS_FUN explicit stack(size_t initial_capacity) {
    shm_init(HSHM_MEMORY_MANAGER->GetDefaultAllocator<AllocT>(), initial_capacity);
  }

  HSHM_CROSS_FUN explicit stack(const hipc::CtxAllocator<AllocT> &alloc, 
                               size_t initial_capacity) {
    shm_init(alloc, initial_capacity);
  }

  HSHM_CROSS_FUN void shm_init(const CtxAllocator<AllocT> &alloc, 
                               size_t initial_capacity) {
    init_shm_container(alloc);
    SetNull();
    reserve(initial_capacity);
  }

  /**====================================
   * Copy Constructors
   * ===================================*/
   
  HSHM_CROSS_FUN explicit stack(const stack &other) {
    init_shm_container(other.GetCtxAllocator());
    SetNull();
    shm_strong_copy_main(other);
  }

  HSHM_CROSS_FUN explicit stack(const hipc::CtxAllocator<AllocT> &alloc,
                               const stack &other) {
    init_shm_container(alloc);
    SetNull();
    shm_strong_copy_main(other);
  }

  HSHM_CROSS_FUN stack &operator=(const stack &other) {
    if (this != &other) {
      shm_destroy();
      shm_strong_copy_main(other);
    }
    return *this;
  }

  template <typename ContainerT>
  HSHM_CROSS_FUN void shm_strong_copy_main(const ContainerT &other) {
    reserve(other.size());
    if constexpr (std::is_pod<T>() && !IS_SHM_ARCHIVEABLE(T)) {
      memcpy(data(), other.data(), other.size() * sizeof(T));
      size_ = other.size();
    } else {
      for (size_t i = 0; i < other.size(); ++i) {
        push(other[i]);
      }
    }
  }

  /**====================================
   * Move Constructors
   * ===================================*/
   
  HSHM_CROSS_FUN stack(stack &&other) {
    shm_move_op<false>(HSHM_MEMORY_MANAGER->GetDefaultAllocator<AllocT>(), 
                       std::move(other));
  }

  HSHM_CROSS_FUN stack(const hipc::CtxAllocator<AllocT> &alloc, stack &&other) {
    shm_move_op<false>(alloc, std::move(other));
  }

  HSHM_CROSS_FUN stack &operator=(stack &&other) noexcept {
    if (this != &other) {
      shm_move_op<true>(other.GetCtxAllocator(), std::move(other));
    }
    return *this;
  }

  template <bool IS_ASSIGN>
  HSHM_CROSS_FUN void shm_move_op(const hipc::CtxAllocator<AllocT> &alloc,
                                  stack &&other) noexcept {
    if constexpr (!IS_ASSIGN) {
      init_shm_container(alloc);
    }
    if (GetAllocator() == other.GetAllocator()) {
      // Same allocator: simple move
      memcpy((void *)this, (void *)&other, sizeof(*this));
      other.SetNull();
    } else {
      // Different allocator: copy and destroy
      shm_strong_copy_main(other);
      other.shm_destroy();
    }
  }

  /**====================================
   * Destructors
   * ===================================*/
   
  HSHM_INLINE_CROSS_FUN bool IsNull() const { 
    return data_ptr_.IsNull(); 
  }

  HSHM_INLINE_CROSS_FUN void SetNull() {
    size_ = 0;
    capacity_ = 0;
    data_ptr_.SetNull();
  }

  HSHM_INLINE_CROSS_FUN void shm_destroy_main() {
    clear();
    if (!data_ptr_.IsNull()) {
      CtxAllocator<AllocT> alloc = GetCtxAllocator();
      FullPtr<void, OffsetPointer> full_ptr(
        alloc->template Convert<void>(data_ptr_), data_ptr_);
      alloc->template Free<void>(alloc.ctx_, full_ptr);
    }
  }

  /**====================================
   * Stack Operations
   * ===================================*/
   
  template <typename... Args>
  HSHM_CROSS_FUN void emplace(Args &&...args) {
    if (size_ == capacity_) {
      size_t new_capacity = capacity_ == 0 ? 8 : capacity_ * 2;
      reserve(new_capacity);
    }
    T* data = this->data();
    new (data + size_) T(std::forward<Args>(args)...);
    ++size_;
  }

  HSHM_CROSS_FUN void push(const T &item) {
    emplace(item);
  }

  HSHM_CROSS_FUN void push(T &&item) {
    emplace(std::move(item));
  }

  HSHM_CROSS_FUN void pop() {
    if (size_ > 0) {
      --size_;
      data()[size_].~T();
    }
  }

  HSHM_INLINE_CROSS_FUN T &top() { 
    return data()[size_ - 1]; 
  }

  HSHM_INLINE_CROSS_FUN const T &top() const { 
    return data()[size_ - 1]; 
  }

  HSHM_INLINE_CROSS_FUN bool empty() const { 
    return size_ == 0; 
  }

  HSHM_INLINE_CROSS_FUN size_t size() const { 
    return size_; 
  }

  HSHM_INLINE_CROSS_FUN size_t capacity() const { 
    return capacity_; 
  }

  HSHM_CROSS_FUN void clear() {
    T* data = this->data();
    for (size_t i = 0; i < size_; ++i) {
      data[i].~T();
    }
    size_ = 0;
  }

  HSHM_CROSS_FUN void reserve(size_t new_capacity) {
    if (new_capacity <= capacity_) return;

    CtxAllocator<AllocT> alloc = GetCtxAllocator();
    
    if constexpr (std::is_pod<T>() && !IS_SHM_ARCHIVEABLE(T)) {
      // Use reallocation for POD types
      if (!data_ptr_.IsNull()) {
        FullPtr<T, OffsetPointer> old_ptr(
          alloc->template Convert<T>(data_ptr_), data_ptr_);
        auto new_ptr = alloc->template ReallocateObjs<T>(
          alloc.ctx_, old_ptr, new_capacity);
        data_ptr_ = new_ptr.shm_;
      } else {
        auto new_ptr = alloc->template AllocateObjs<T, OffsetPointer>(
          alloc.ctx_, new_capacity);
        data_ptr_ = new_ptr.shm_;
      }
    } else {
      // Manual move for complex types
      auto new_ptr = alloc->template AllocateObjs<T, OffsetPointer>(
        alloc.ctx_, new_capacity);
      T* new_data = new_ptr.ptr_;
      OffsetPointer new_data_ptr = new_ptr.shm_;
      
      if (!data_ptr_.IsNull()) {
        T* old_data = alloc->template Convert<T>(data_ptr_);
        for (size_t i = 0; i < size_; ++i) {
          new (new_data + i) T(std::move(old_data[i]));
          old_data[i].~T();
        }
        
        FullPtr<void, OffsetPointer> old_ptr(
          alloc->template Convert<void>(data_ptr_), data_ptr_);
        alloc->template Free<void>(alloc.ctx_, old_ptr);
      }
      
      data_ptr_ = new_data_ptr;
    }
    
    capacity_ = new_capacity;
  }

  /**====================================
   * Utility Methods
   * ===================================*/
   
  HSHM_INLINE_CROSS_FUN T* data() {
    return GetAllocator()->template Convert<T>(data_ptr_);
  }

  HSHM_INLINE_CROSS_FUN const T* data() const {
    return GetAllocator()->template Convert<T>(data_ptr_);
  }

  HSHM_INLINE_CROSS_FUN T& operator[](size_t index) {
    return data()[index];
  }

  HSHM_INLINE_CROSS_FUN const T& operator[](size_t index) const {
    return data()[index];
  }

  /**====================================
   * Serialization
   * ===================================*/
   
  template <typename Ar>
  HSHM_CROSS_FUN void save(Ar &ar) const {
    ar << size_;
    for (size_t i = 0; i < size_; ++i) {
      ar << (*this)[i];
    }
  }

  template <typename Ar>
  HSHM_CROSS_FUN void load(Ar &ar) {
    size_t loaded_size;
    ar >> loaded_size;
    clear();
    reserve(loaded_size);
    size_ = loaded_size;
    for (size_t i = 0; i < size_; ++i) {
      ar >> (*this)[i];
    }
  }
};

#undef CLASS_NAME
#undef CLASS_NEW_ARGS

} // namespace hshm::ipc

// Namespace alias for convenience
namespace hshm {
  template <typename T, HSHM_CLASS_TEMPL_WITH_PRIV_DEFAULTS>
  using stack = hipc::stack<T, HSHM_CLASS_TEMPL_ARGS>;
}
```

## Usage Example

```cpp
#include "your_stack.h"

int main() {
    // Create shared memory allocator
    auto alloc = HSHM_MEMORY_MANAGER->GetDefaultAllocator<hipc::ScalablePageAllocator>();
    
    // Create stack in shared memory
    hipc::stack<int> shared_stack(alloc, 100);  // Initial capacity of 100
    
    // Use the stack
    shared_stack.push(42);
    shared_stack.push(24);
    shared_stack.emplace(99);
    
    printf("Top element: %d\n", shared_stack.top());  // Prints: 99
    printf("Stack size: %zu\n", shared_stack.size());  // Prints: 3
    
    // Pop elements
    shared_stack.pop();
    printf("New top: %d\n", shared_stack.top());  // Prints: 24
    
    // Create a local stack (for comparison)
    hshm::stack<int> local_stack;  // Uses private memory allocator
    local_stack.push(1);
    local_stack.push(2);
    
    return 0;
}
```

## Best Practices

### 1. Memory Safety
- Always check `IsNull()` before operations
- Implement proper `shm_destroy_main()` cleanup
- Use RAII principles with proper constructors/destructors

### 2. Performance Optimization
- Use `IS_SHM_ARCHIVEABLE` to optimize for POD vs complex types
- Prefer `emplace` over `push` when possible
- Batch allocations to reduce fragmentation

### 3. Template Design
- Use `HSHM_CLASS_TEMPL_WITH_DEFAULTS` for public interfaces
- Always provide both shared and private memory versions
- Clean up macros with `#undef` at the end of headers

### 4. Error Handling
- Throw appropriate HSHM exceptions for error conditions
- Validate inputs in public methods
- Handle out-of-memory conditions gracefully

### 5. Serialization
- Implement `save()` and `load()` methods for network compatibility
- Consider endianness for cross-platform deployments
- Handle version compatibility in serialization formats

## Advanced Topics

### Custom Allocators
```cpp
// Custom allocator-aware container
template <typename T, typename AllocT = hipc::ScalablePageAllocator>
class my_custom_container : public ShmContainer {
    // Implementation using specific allocator constraints...
};
```

### Thread-Local Containers
```cpp
// Thread-local version using flags
hipc::stack<int, hipc::MallocAllocator, hipc::ShmFlag::kIsThreadLocal> tls_stack;
```

### Heterogeneous Data Structures
```cpp
// Container holding other HSHM containers
hipc::vector<hipc::string> string_vector;
hipc::stack<hipc::vector<int>> nested_container;
```

This comprehensive guide provides everything needed to create sophisticated shared memory data structures that integrate seamlessly with the HSHM ecosystem while maintaining high performance and memory safety.