# Hera2 Python SDK

A modern, fluent Python SDK for OpenMetadata that provides an intuitive API for all operations. Authentication is routed through the Heimdall authorization service for DataOS integration.

Installing `hera2-sdk` pulls in `hera2-ingestion` (HERA-modified fork of openmetadata-ingestion) automatically and shares the `metadata` namespace, so you get both `metadata.sdk` (Heimdall auth, fluent API) and the full ingestion stack: `metadata.ingestion`, `metadata.generated`, `metadata.clients`, `metadata.profiler`, `metadata.utils`, `metadata.workflow`, etc.

## Installation

**Python 3.9+ required.** We recommend Python 3.10 or 3.11 for best compatibility.

**macOS note:** On macOS, `python3.10` or `python3.11` in your shell may be a symlink to the system Python 3.9. Always use the **full path** to the Homebrew-installed interpreter when creating the venv:

```bash
# Install Python 3.11 via Homebrew if not already available
brew install python@3.11

# Verify the Homebrew interpreter (M-series Mac uses /opt/homebrew, Intel uses /usr/local)
/opt/homebrew/bin/python3.11 --version   # should print 3.11.x

# Create the venv using the full path
/opt/homebrew/bin/python3.11 -m venv .venv
source .venv/bin/activate

# Always use python3 -m pip (not bare pip) to avoid shell alias interference
python3 -m pip install hera2-sdk
```

After activation, confirm the interpreter is correct before installing:

```bash
python3 --version   # must show 3.11.x (or 3.10.x)
python3 -m pip --version   # must show python 3.11 (or 3.10), not 3.9
```

For an editable install from the repo:

```bash
pip install -e /path/to/hera2/ingestion/
pip install -e /path/to/hera2/hera2-sdk/
```

### Data Quality SDK Installation

For running data quality tests, additional dependencies may be required:

**DataFrame Validation:**
```bash
pip install 'hera2-sdk[pandas]'
```

**Table-Based Testing:**
```bash
pip install 'hera2-sdk[mysql]'        # For MySQL
pip install 'hera2-sdk[postgres]'     # For PostgreSQL
pip install 'hera2-sdk[snowflake]'    # For Snowflake
pip install 'hera2-sdk[clickhouse]'   # For ClickHouse
```

### Troubleshooting

- **`python --version` shows 3.9 inside the venv** — Your shell has an alias (`alias python=...`) that overrides venv activation. Check with:
  ```bash
  type python   # shows "python is an alias for ..."
  grep -n "python" ~/.zshrc ~/.zprofile 2>/dev/null
  ```
  Remove the alias from `~/.zshrc` / `~/.zprofile` and reload (`source ~/.zshrc`). Until then, use `python3` instead of `python` — the venv’s `python3` is not affected by the alias.

- **`pip --version` shows python 3.9** — Similarly, `pip` may be aliased to the system pip (`alias pip=/usr/bin/pip3`). Always use `python3 -m pip` instead of bare `pip` to drive pip through the venv interpreter:
  ```bash
  python3 -m pip install hera2-sdk==<version>
  ```

- **"Defaulting to user installation because normal site-packages is not writeable"** — Pip is installing to your user directory instead of the active venv.
  1. Use the venv’s Python explicitly: `python3 -m pip install hera2-sdk`.
  2. If it still happens, make the venv writable and reinstall: `chmod -R u+w .venv` (or your venv dir), then `python3 -m pip install --force-reinstall hera2-sdk`.
  3. Confirm the venv is the one being used: `python3 -c "import sys; print(sys.prefix)"` should print a path inside your venv. If it prints a system path, the venv wasn’t activated or `python3` in your shell isn’t from the venv.

- **`TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'`** when importing `metadata.ingestion` — The interpreter is loading the **local** `metadata` package from the hera2 ingestion source (`.../hera2/ingestion/src/metadata/`) instead of the installed `hera2-ingestion`. Something is adding that path to `sys.path`.
  1. Unset `PYTHONPATH` before running Python: `unset PYTHONPATH`, then run your script again.
  2. See what adds the repo to the path: `python3 -c "import sys; print([p for p in sys.path if 'hera2' in p or 'ingestion' in p])"`. If you see a path like `.../hera2/ingestion/src`, it was added by `PYTHONPATH` or a `.pth` file in site-packages.
  3. Run Python from a directory that is **outside** the hera2 repo.

- **`UserWarning: pkg_resources is deprecated as an API`** on import — Harmless deprecation warning. Suppress with:
  ```python
  import warnings
  warnings.filterwarnings("ignore", category=UserWarning, module="pkg_resources")
  ```
  Or via environment variable: `PYTHONWARNINGS="ignore::UserWarning:pkg_resources" python3 your_script.py`

## Quick Start

### Configure the SDK (Heimdall Auth — Recommended)

Use **heimdallConfiguration** (same structure as `hera/config/config.yaml`
`authenticationConfiguration.heimdallConfiguration`):

```python
from metadata.sdk import configure

configure(
    host="http://localhost:8585/api",
    api_key="your-dataos-api-key",
    heimdall_configuration={
        "enabled": True,
        "baseUrl": "https://your-instance.dataos.cloud/heimdall",
        "timeout": 10,
        "fallbackOnBasic": True,
    },
)
```

Or use the legacy `heimdall_url`:

```python
configure(
    host="http://localhost:8585/api",
    api_key="your-dataos-api-key",
    heimdall_url="https://your-instance.dataos.cloud/heimdall",
)
```

Or set environment variables and call `configure()` with no arguments:

```bash
export OPENMETADATA_HOST="http://localhost:8585/api"
export OPENMETADATA_API_KEY="your-dataos-api-key"
export HEIMDALL_BASE_URL="https://your-instance.dataos.cloud/heimdall"
```

```python
from metadata.sdk import configure
configure()
```

### Configure Parameters

The `configure()` function supports:
- **`host`** or **`server_url`**: OpenMetadata server URL
- **`api_key`** or **`jwt_token`**: DataOS API key or JWT token
- **`heimdall_configuration`**: Dict matching `hera/config/config.yaml` `heimdallConfiguration` (enabled, baseUrl, timeout, fallbackOnBasic, trustAll)
- **`heimdall_url`**: Heimdall base URL (legacy; use heimdall_configuration when possible)
- Falls back to environment variables:
  - `OPENMETADATA_HOST` or `OPENMETADATA_SERVER_URL` for the server URL
  - `OPENMETADATA_API_KEY` or `OPENMETADATA_JWT_TOKEN` for authentication
  - `HEIMDALL_BASE_URL`: Heimdall service URL (enables Heimdall auth)
  - `HEIMDALL_TIMEOUT`: Heimdall request timeout in seconds (default: 10)
  - `HEIMDALL_TRUST_ALL`: Trust all SSL certs for Heimdall (default: true)
  - `OPENMETADATA_VERIFY_SSL`: Enable SSL verification (default: false)
  - `OPENMETADATA_CA_BUNDLE`: Path to CA bundle
  - `OPENMETADATA_CLIENT_TIMEOUT`: Client timeout in seconds (default: 30)

### Alternative: Builder Pattern

```python
from metadata.sdk.config import OpenMetadataConfig

config = (
    OpenMetadataConfig.builder()
    .server_url("http://localhost:8585/api")
    .api_key("your-dataos-api-key")
    .heimdall_configuration({
        "enabled": True,
        "baseUrl": "https://your-instance.dataos.cloud/heimdall",
        "timeout": 15,
        "fallbackOnBasic": True,
    })
    .build()
)
```

Or with flat params: `.heimdall_url("...").heimdall_timeout(15)`.

### Alternative: Direct JWT (Legacy)

If Heimdall is not available, the SDK falls back to direct JWT authentication:

```python
from metadata.sdk import configure
configure(host="http://localhost:8585/api", jwt_token="your-om-jwt-token")
```

### Using like the [OpenMetadata Python SDK](https://docs.open-metadata.org/v1.11.x/sdk/python#1-initialize-openmetadata)

hera2-sdk depends on `hera2-ingestion`, so you can use the **same low-level API** as in the [official OpenMetadata SDK docs](https://docs.open-metadata.org/v1.11.x/sdk/python#1-initialize-openmetadata): `OpenMetadataConnection` + `OpenMetadata(server_config)`, then `metadata.create_or_update()`, `metadata.get_by_name()`, `metadata.delete()`, etc.

**Option 1 — Standard OpenMetadata style (same as the docs)**

```python
from metadata.ingestion.ometa.ometa_api import OpenMetadata
from metadata.generated.schema.entity.services.connections.metadata.openMetadataConnection import (
    OpenMetadataConnection,
    AuthProvider,
)
from metadata.generated.schema.security.client.openMetadataJWTClientConfig import (
    OpenMetadataJWTClientConfig,
)
from metadata.generated.schema.entity.data.table import Table

server_config = OpenMetadataConnection(
    hostPort="http://localhost:8585/api",
    authProvider=AuthProvider.openmetadata,
    securityConfig=OpenMetadataJWTClientConfig(
        jwtToken="<YOUR-INGESTION-BOT-JWT-TOKEN>",
    ),
)
metadata = OpenMetadata(server_config)

# Same API as in the docs
metadata.health_check()
service_entity = metadata.create_or_update(data=create_service)
my_table = metadata.get_by_name(entity=Table, fqn="test-service-table.test-db.test-schema.test")
metadata.delete(entity=Table, entity_id=my_table.id)
```

**Option 2 — hera2-sdk wrapper (same API + optional Heimdall)**

Use `configure()` or `OpenMetadataConfig`, then get the underlying client via `.ometa` and call the same methods:

```python
from metadata.sdk import configure, client
from metadata.generated.schema.entity.data.table import Table

configure(
    host="http://localhost:8585/api",
    api_key="your-dataos-api-key",
    heimdall_configuration={
        "enabled": True,
        "baseUrl": "https://your-instance.dataos.cloud/heimdall",
        "timeout": 10,
        "fallbackOnBasic": True,
    },
)

metadata = client().ometa   # same interface as OpenMetadata(server_config)

metadata.health_check()
service_entity = metadata.create_or_update(data=create_service)
my_table = metadata.get_by_name(entity=Table, fqn="test-service-table.test-db.test-schema.test")
metadata.delete(entity=Table, entity_id=my_table.id)
```

So you can follow the [OpenMetadata SDK walkthrough](https://docs.open-metadata.org/v1.11.x/sdk/python) (create DatabaseService, Database, Schema, Table, etc.) with either the raw `OpenMetadata` from `metadata.ingestion.ometa.ometa_api` or with `client().ometa` after configuring hera2-sdk.

### Manual Initialization

For more control, you can manually initialize the SDK:

```python
from metadata.sdk import OpenMetadata, OpenMetadataConfig
from metadata.sdk.entities import Table, User
from metadata.sdk.api import Search, Lineage, Bulk

config = OpenMetadataConfig(
    server_url="http://localhost:8585/api",
    api_key="your-dataos-api-key",
    heimdall_configuration={
        "enabled": True,
        "baseUrl": "https://your-instance.dataos.cloud/heimdall",
        "timeout": 10,
        "fallbackOnBasic": True,
    },
)

client = OpenMetadata.initialize(config)

Table.set_default_client(client)
User.set_default_client(client)
Search.set_default_client(client)
Lineage.set_default_client(client)
Bulk.set_default_client(client)
```

### Configuration from Environment Variables Only

```python
from metadata.sdk.config import OpenMetadataConfig

# Reads from OPENMETADATA_HOST, OPENMETADATA_API_KEY, HEIMDALL_BASE_URL, etc.
config = OpenMetadataConfig.from_env()
```

## Entity Operations

### Tables

```python
from metadata.generated.schema.api.data.createTable import CreateTableRequest
from metadata.sdk.entities.table import TableListParams

# Create a table
request = CreateTableRequest(
    name="my_table",
    databaseSchema="my_schema",
    columns=[...]
)
table = Table.create(request)

# Retrieve a table by ID
table = Table.retrieve("table-id")

# Retrieve by fully qualified name with specific fields
table = Table.retrieve_by_name(
    "service.database.schema.table",
    fields=["owners", "tags", "columns"]
)

# List tables with pagination
for table in Table.list().auto_paging_iterable():
    print(table.name)

# List with filters
params = TableListParams.builder() \
    .limit(50) \
    .database("my_database") \
    .fields(["owners", "tags"]) \
    .build()

tables = Table.list(params)

# Update a table
table.description = "Updated description"
updated = Table.update(table.id, table)

# Delete a table
Table.delete("table-id")

# Delete with options
Table.delete("table-id", recursive=True, hard_delete=True)

# Export/Import CSV
csv_data = Table.export_csv("table-name")
Table.import_csv(csv_data, dry_run=False)
```

## Supported Entity Types

The SDK provides the same fluent API for all OpenMetadata entity types:

- **Data Assets**: Table, Database, DatabaseSchema, Dashboard, Pipeline, Topic, Container, Query, StoredProcedure, DashboardDataModel, SearchIndex, MlModel, Report
- **Services**: DatabaseService, MessagingService, DashboardService, PipelineService, MlModelService, StorageService, SearchService, MetadataService, ApiService
- **Teams & Users**: User, Team, Role, Policy
- **Governance**: Glossary, GlossaryTerm, Classification, Tag, DataProduct, Domain
- **Quality**: TestCase, TestSuite, TestDefinition, DataQualityDashboard
- **Ingestion**: Ingestion, Workflow, Connection
- **Other**: Type, Webhook, Kpi, Application, Persona, DocStore, Page, SearchQuery

## Testing

Run the SDK tests:

```bash
# Run all SDK tests
pytest tests/unit/sdk/

# Run specific test
pytest tests/unit/sdk/test_sdk_entities.py
```

## License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
