# Changelog

All notable changes to the `entr-adapter-core` package are documented here.
This project follows [Semantic Versioning](https://semver.org/) and the
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format.

## How to Maintain This Changelog

Every PR that changes package behavior must include a changelog entry under the
`[Unreleased]` section. When a version is tagged, the `[Unreleased]` section is
renamed to `[vX.Y.Z] - YYYY-MM-DD` and a new empty `[Unreleased]` section is added.

**Rules:**

- Use imperative mood (e.g., "Add flight recorder module" not "Added flight recorder module")
- If in doubt whether a change is breaking, consult the breaking change table below;
  if still uncertain, document it as potentially breaking
- Deployment file changes must **always** be called out explicitly in
  `Deployment File Changes`, even if the change is non-breaking
- Schema changes must be documented in `Schema Changes` with clear breaking/non-breaking labels
- Dependency version bumps must be documented in `Dependency Changes` with impact assessment

**Section categories per version entry:**

- `Added` — new features and modules
- `Changed` — changes to existing functionality
- `Fixed` — bug fixes
- `Deprecated` — features that will be removed in a future version
- `Removed` — features removed in this version
- `Breaking Changes` — only present in major version bumps
- `Deployment File Changes` — present whenever default deployment files change
- `Schema Changes` — present whenever contract schemas change
- `Dependency Changes` — present whenever package dependencies change

<!-- RELEASE PROCESS:
     1. Update version in pyproject.toml (e.g., "1.1.0")
     2. Add a new section below [Unreleased] with the version and date
     3. Move items from [Unreleased] to the new section
     4. Commit: "release: v1.1.0"
     5. Tag:   git tag v1.1.0
     6. Push:  git push origin main --tags

     TAG FORMAT: v{major}.{minor}.{patch} (v prefix is mandatory)
     The pyproject.toml version field omits the v prefix.
-->

## Breaking Change Policy

The following table (from the architecture document) is the authoritative reference
for version bump decisions. See `VERSIONING.md` for full details.

| Change Type | Breaking? | Version Bump |
|-------------|-----------|-------------|
| New required parameter on public function | **Yes** | Major |
| `BaseHandler.process()` signature change | **Yes** | Major |
| Removing or renaming a public export | **Yes** | Major |
| Schema field removal or type change | **Yes** | Major |
| Override directory path change | **Yes** | Major |
| New optional parameter with default | No | Minor |
| New optional schema field | No | Minor |
| New public function or class | No | Minor |
| Bug fix in dispatch/resolution logic | No | Patch |
| Internal refactoring (no API change) | No | Patch |
| Dependency minimum version bump | Potentially | Minor or Major |
| Default deployment file changes | No* | Minor |

\* Deployment file changes are non-breaking because adapters can override any file.
However, these changes are **always explicitly documented** below.

Major version bumps always correspond to breaking changes per semver. Migration
paths are found in the `Breaking Changes` section of each version entry.

## Format Guide

### Breaking Changes Entry Template

```
### Breaking Changes

- **`BaseHandler.process()` signature change**: Added required `context` parameter.
  - **Change type**: BaseHandler.process() signature change
  - **Why**: Enables handlers to access request metadata (tenant info, correlation ID)
  - **Migration path**: Update all handler `process()` methods:
    ```python
    # Before (v1.x)
    async def process(self, payload: DocumentPayload) -> AdapterResponse:
    # After (v2.0)
    async def process(self, payload: DocumentPayload, context: ProcessContext) -> AdapterResponse:
    ```
```

### Deployment File Changes Entry Template

```
### Deployment File Changes

- **`nginx.conf`**: Increased `client_max_body_size` from `100M` to `200M`
  - **Default before**: `client_max_body_size 100M`
  - **Default after**: `client_max_body_size 200M`
  - **Adapter action**: If you have an `nginx.conf` override in `deploy/`, review
    whether your override should also increase this limit. If you do not override
    `nginx.conf`, no action needed — the new default applies automatically.
```

### Schema Changes Entry Template

Breaking schema change:
```
### Schema Changes

- **`AdapterResponse.notifications` field removed** (BREAKING)
  - **Change type**: Schema field removal
  - **Why**: Replaced by event-based notification system
  - **Migration path**: Replace `notifications` list with `emit_event()` calls:
    ```python
    # Before (v1.x)
    return AdapterResponse(status="success", notifications=["Processed"])
    # After (v2.0)
    emit_event("processed", payload)
    return AdapterResponse(status="success")
    ```
```

Non-breaking schema change:
```
### Schema Changes

- **`AdapterResponse.warnings` field added** (non-breaking)
  - Optional `list[str]` field, defaults to `[]`
  - No adapter action required — existing adapters continue to work unchanged
```

### Dependency Changes Entry Template

```
### Dependency Changes

- **FastAPI**: Minimum version bumped from `>=0.100.0` to `>=0.115.0`
  - **Potentially breaking**: If your adapter pins FastAPI below `0.115.0`, you
    will need to upgrade
  - **Reason**: Required for new middleware features used by app factory
```

---

## [v1.5.5] - 2026-03-24
### Fixed
- Fixed an issue where the logs were not streamed but only captured.

## [v1.5.4] - 2026-03-24

### Removed
- **Docker healthchecks from `docker-compose.prod.yml`**: Healthchecks used `curl` which is not installed in the slim Python or Node base images, causing all services to report `unhealthy` despite running correctly. Removed until a proper server-level health monitoring solution is in place

## [v1.5.2] 19-03-2025

### Fixed
- **FlightRecorder captured logs from concurrent requests**: When multiple `/process` requests ran concurrently, each FlightRecorderHandler (attached to the root logger) captured logs from ALL concurrent handlers, causing file 1501's `execution_logs` to contain logs from files 1500, 1499, 1498, etc. Now uses a `ContextVar`-based token to scope each recorder to its own async request context, so each response only contains its own logs

## [v1.5.0] 19-03-2026

### Added
- **`package_version` field in `/version` endpoint**: The `GET /version` response now includes `package_version` showing the installed `entr-adapter-core` version. Displayed as "Adapter-Core" on the admin page

## [v1.4.0] 19-03-2026

### Fixed
- **`/download_document` silently returned empty response on error**: When the dispatcher raised `ValueError` (unknown handler or missing retrieval handler), the endpoint returned an empty `DocumentDownloadResponse` with no error indication. Now raises `HTTPException(404)` with a descriptive error message so Core can distinguish failures from empty documents
- **FlightRecorder missed INFO/DEBUG logs in production**: The FlightRecorder set the handler level to DEBUG but did not adjust the logger level. When the root logger was at WARNING (the production default), handler INFO/DEBUG messages were filtered before reaching the recorder, resulting in nearly empty `execution_logs`. Now temporarily lowers the logger level to DEBUG during recording and restores it on exit
- **`/version` endpoint used relative file paths**: `pyproject.toml` and `core-version.txt` were resolved relative to the current working directory at request time, which could silently return "unknown" if CWD changed after startup (e.g., in tests or background tasks). Now captures CWD at router creation time and resolves against that fixed path

### Changed
- **Concurrent stateful handler test now meaningful**: The `test_concurrent_requests_stateful_handler_leaks` test previously had `assert mismatches >= 0` which always passed. Now uses `pytest.skip()` when no leakage is detected (timing-dependent) instead of a no-op assertion, and sends 50 concurrent requests (up from 20) to increase interleaving probability
- **`resolve_compose_files()` now accepts a `mode` parameter**: Previously returned all 3 compose files at once (unusable for `docker compose -f`). Now accepts `mode="dev"` or `mode="prod"` and returns only the appropriate compose file plus optional override. Default mode is `"dev"` for backward compatibility
- **`nginx.conf` upload limit increased to 50M**: Was `5M`, which caused uploads to fail for documents larger than 5MB. Increased to 50M to cover all typical business documents (invoices, scanned multi-page PDFs, email attachments)

### Removed
- **`docker-compose.yml` (base compose file)**: Neither the dev nor prod compose files extended this base — both are self-contained after the v1.3.x production rewrites. Removed dead code to avoid confusion

### Deployment File Changes
- **`nginx.conf`**: `client_max_body_size` changed from `5M` to `50M`
  - **Adapter action**: If you override `nginx.conf` in `deploy/`, review whether your override should also increase this limit. If you do not override `nginx.conf`, no action needed
- **`docker-compose.prod.yml`**: Added `healthcheck` sections to `nextapp`, `backend`, and `adapter` services
  - **Adapter action**: If you override `docker-compose.prod.yml`, consider adding healthchecks to your override. If using package defaults, no action needed
- **`docker-compose.dev.yml`**: Added documentation for `DEFAULTS_DIR` env var with manual setup instructions for running without the CLI
- **`publish.yml`**: Added version tag validation step — workflow now fails if git tag version doesn't match `pyproject.toml` version, preventing mismatched PyPI publishes
- **`VERSIONING.md`**: Release process now documents GitHub Release creation (step 6) required to trigger PyPI publish
- **`CHANGELOG.md`**: Fixed stale v1.0.0 entry — corrected `DocumentPayload` fields (removed `mappings`/`mapping_rules`/`meta`, added `applied_mappings`), fixed `execution_log` → `execution_logs`, removed references to deleted `docker-compose.yml` base file, added notes on subsequently rewritten deployment files
- **`defaults.md`**: Updated package defaults table, per-file descriptions, and resolution log example to reflect current state — removed `docker-compose.yml` references, corrected nginx description (routes Core traffic with 50M limit, not adapter:8096 with 100M), updated fluent-bit description (forward input, not tail), corrected compose override location (adapter root, not `deploy/`), added `prefix.lua` to file list
- **`new-adapter-guide.md`**: Fixed overridable files appendix — corrected nginx description and removed deleted `docker-compose.yml` entry

### Schema Changes
- **`AppliedMapping` fields aligned with Core** (BREAKING for adapters using old field names)
  - `source_value` renamed to `original_value`
  - `target_value` renamed to `mapped_value`
  - Added `field_path: str` (required) — JSON path where the mapping was applied
  - Added `context: Optional[Dict]` — context fields used for strict matching
  - **Why**: Package schema was out of sync with Core's `AppliedMapping` (Core is single source of truth). Mismatched field names caused 422 validation errors when Core sent payloads with applied mappings
  - **Migration**: Update any adapter code referencing `AppliedMapping` fields:
    ```python
    # Before
    mapping.source_value, mapping.target_value
    # After
    mapping.original_value, mapping.mapped_value, mapping.field_path
    ```
- **FTP/SFTP `upload_file` only supported text files**: Both connectors opened files in text mode (`'r'`), making binary uploads (PDFs, images) fail with `UnicodeDecodeError`. FTP `upload_file` now uses binary mode via new `upload_file_binary` method. SFTP `upload_file` now uses paramiko's `sftp.put()` which handles binary natively
- **Missing extras gave confusing `TypeError` instead of `ImportError`**: Importing connectors with uninstalled extras via `connectors/__init__.py` returned `None`, causing `TypeError: 'NoneType' object is not callable` on use. Now returns a placeholder class that raises a clear `ImportError` with install instructions when instantiated
- **FTP/SFTP used deprecated `asyncio.get_event_loop()`**: Replaced with `asyncio.get_running_loop()` (recommended since Python 3.10, required target is 3.12+)
- **Coding standards default scan path was `app/` instead of `src/`**: Workflow, checker script, caller template, and ENTR-008 rule all defaulted to the old `app/` directory structure. Updated to `src/` matching the current adapter convention from Story 5.3
- **Coding standards workflow used outdated GitHub Actions**: Updated `actions/checkout` to `@v5` and `actions/setup-python` to `@v6`. Fixed repository reference from `ENTR-Solutions` to `GitGoShaka`

## [v1.3.3] - 2026-03-19

### Fixed
- **`fluent-bit.conf` used wrong input method**: Config used `tail` input (reading Docker log files directly) which requires a `docker` parser and `tail-db` directory that don't exist in the `fluent/fluent-bit` image. Replaced with `forward` input on port 24224, matching the fluentd Docker logging driver used by all services in `docker-compose.prod.yml`
- **`fluent-bit.conf` wrong AWS region**: Was `eu-west-1`, corrected to `eu-central-1`
- **`fluent-bit.conf` wrong log group structure**: Used per-tenant log groups (`/entr/${TENANT_SLUG}/adapter`), corrected to shared log group `/entr/production` with per-tenant stream names (`${TENANT_SLUG}-production`) matching existing tenant deployments

### Deployment File Changes
- **`fluent-bit.conf`**: Rewritten to match proven production config — `forward` input, `lua` filter with `prefix.lua`, CloudWatch output to `/entr/production` log group

## [v1.3.2] - 2026-03-19

### Fixed
- **`nginx.conf` was adapter-only**: The default `nginx.conf` only routed to the adapter upstream, but production mounts it into Core's nginx container which needs routes to nextapp (frontend), backend (API), and SSE. Replaced with the full production nginx config matching all existing tenant deployments
- **`fluent-bit` missing `prefix.lua`**: The fluent-bit config references a Lua script for log prefixing (`prefix.lua`), but the package didn't include it and the volume mount only mapped the single conf file. Fluent-bit crashed on startup with exit code 255
- **`resolve.py` didn't know about `prefix.lua`**: Added `prefix.lua` to `SHADOW_FILES` so the file resolution system picks it up

### Deployment File Changes
- **`docker-compose.prod.yml`**: Rewritten to be fully self-contained — matches the proven production pattern used by all existing tenants:
  - All services include their `image` field (no base+overlay split)
  - `nginx` uses `${DOCKER_USER}/entr-nginx:${CORE_TAG}` (Core's nginx, not `nginx:alpine`)
  - `fluent-bit` uses `fluent/fluent-bit:latest` with custom command and whole-directory mount (`./fluent-bit:/config:ro`)
  - `backend` env_file corrected to `./backend/.env.client_specific`
  - `adapter` env_file corrected to `../.env.prod`
  - Added fluentd logging (`x-logging`), resource limits, and `entr` network
  - **Adapter action**: If you override `docker-compose.prod.yml` in `deploy/`, no action needed. If using package defaults, this fix is required for production deploys to work
- **`nginx.conf`**: Now the full production config with routes to nextapp, backend, SSE, static asset caching, ELB health check handling, and gzip compression
- **`prefix.lua`**: New file — Lua script for fluent-bit log line prefixing with container name

## [v1.3.0] - 2026-03-18

### Added
- **Adapter porting guide**: New Section 12 in `new-adapter-guide.md` documenting how to efficiently port logic from an existing adapter. Includes file copy rules for AI agents (use `cp` for unchanged files, Edit for modifications), example workflow, and a reference table of what typically changes vs what can be copied as-is

## [v1.2.0]

### Added

- **Gmail connector config model** (`GmailConnectorConfig`): Pydantic model for tenant-specific Gmail OAuth2 settings with `Field(description=...)` for LLM readability. Supports `label_override` and `max_results_per_page` (1-500)
- **FTP connector config model** (`FTPConnectorConfig`): Pydantic model for FTP connection settings. Backward-compatible — constructors accept config objects or direct keyword arguments
- **SFTP connector config model** (`SFTPConnectorConfig`): Pydantic model for SFTP connection settings with port validation (1-65535)
- **Microsoft connector config model** (`MicrosoftConnectorConfig`): Pydantic model for Microsoft Exchange Graph API settings including `mailbox`, `folder_path`, `delay_days`, `move_to_folder_path`, and `require_pdf_attachment`
- **Microsoft connector enhancements**: Merged features from 5 production adapters — email delay (`delay_days`), post-download archival (`move_to_folder_path`), PDF attachment pre-filtering (server-side + client-side), `sentDateTime` filtering (immutable), pagination warnings, 30s request timeout
- **Fuzzy matching engine** (`utils/matching/`): Data-source-agnostic matching engine with configurable field names, thresholds, synonym/stopword normalization, and weighted rapidfuzz scoring. Includes `MatchingEngine`, `MatchConfig`, `MatchResult`, and `normalize_text`
- **Lazy import guards**: All connectors with optional dependencies (SFTP, Gmail, Microsoft) now gracefully handle missing extras with clear `ImportError` messages directing to the correct `uv add` command
- **Component SETUP guides**: Co-located setup guides for Gmail (`SETUP_gmail.md`), FTP/SFTP (`SETUP_ftp.md`), Microsoft (`SETUP_microsoft.md`), and Matching engine (`utils/matching/SETUP.md`) with prerequisites, configuration, registration, verification, pitfalls, and client onboarding checklists
- **Automated coding standards**: Reusable GitHub Actions workflow (`coding-standards.yml`) with 10 rules enforced via AST, grep, and ruff checks. Central `checks/rules.yaml` applies to all adapter repos. Caller workflow template for adapter repos included
- **Migration guide**: Comprehensive 6-step migration checklist (`docs/MIGRATION_GUIDE.md`) for converting existing adapters from template-fork to package-dependency model

### Dependency Changes

- **New optional dependency group**: `matching = ["rapidfuzz>=3.0.0"]` — for the fuzzy matching engine
- **New dev dependencies**: `pyyaml>=6.0.3` (coding standards rules), `ruff>=0.15.6` (PEP 8 checks), `rapidfuzz>=3.0.0` (matching engine tests)

## [1.1.0] - 2026-03-17

### Added
- `GET /adapters` endpoint — returns sorted list of registered adapter identifiers (used by Core for adapter selection dropdowns)
- `GET /version` endpoint — returns adapter version (from `pyproject.toml`), Core platform version (from `core-version.txt`), and tenant name
- `nginx.dev.conf` — development nginx config that routes `/api` to Core backend, `/` to Next.js frontend, and SSE streams correctly (separate from production `nginx.conf`)

### Changed
- Distribution documentation updated from `git+https` to PyPI format across all docs (VERSIONING.md, new-adapter-guide.md, connectors.md)
- `adapter-version-policy.yml` template simplified — removed `adapter_core` key (PyPI version specifiers in `pyproject.toml` handle this natively)
- Version policy docs in VERSIONING.md updated to show standard Python version specifiers instead of custom auto-update mechanism

### Deployment File Changes

- **`docker-compose.dev.yml`**: Rewritten to match proven local dev setup
  - **Removed**: `fluent-bit` service (production only — never needed in local dev)
  - **Removed**: `adapter-nginx` service (core nginx handles both Core + adapter routing)
  - **Added**: Backend source volume mounts for live reload (`../Entr-Core/backend:/app/backend`)
  - **Added**: Backend explicit `uvicorn --reload` command with `--reload-dir src`
  - **Added**: Backend `.env.dev.override` in env_file list (for local database URL etc.)
  - **Added**: Nextapp source volumes and `next.dev.dockerfile` for dev hot reload
  - **Added**: `networks: entr-dev` named network
  - **Changed**: Adapter port exposed directly (`8096`) instead of through separate adapter-nginx
  - **Adapter action**: If you have a `docker-compose.override.yml`, review for compatibility. If using package defaults, no action needed — the new defaults apply automatically. Run `docker compose down --remove-orphans` to clean up old `adapter-fluent-bit` and `adapter-nginx` containers.

## [1.0.4] - 2026-03-16

### Added
- PyPI publishing: package now available via `pip install entr-adapter-core` (no GitHub credentials needed)
- GitHub Actions publish workflow: automatically publishes to PyPI on GitHub Release creation
- Proprietary license (ENTR Solutions B.V.) replacing MIT
- PyPI metadata: classifiers, keywords, project URLs (Repository, Documentation, Changelog)

### Changed
- GitHub Actions updated to Node.js 24-compatible versions (checkout@v5, setup-python@v6)
- Publish workflow triggers on GitHub Release (not tag push) for consistency with Entr-Core
- README installation instructions updated from `git+https` to PyPI

### Fixed
- Fixed `pyproject.toml` TOML table ordering that caused build failure (`dependencies` was parsed inside `[project.urls]`)

## [1.0.1] - 2026-03-16

Test autorelease to PyPi

## [1.0.0] - 2026-03-13

### Added

**Epic 1 — Installable Handler Framework:**
- Package initialization with `uv init --lib` project structure
- BaseHandler abstract base class with `process()` contract for document processing handlers
- RetrievalHandler abstract base class with `list_documents()` and `download_document()` for document retrieval handlers
- Contract schemas: `DocumentPayload`, `AdapterResponse`, `RetrievalPayload`, `RetrievalResponse` (Pydantic v2 models)
- Handler registration system: `@register_handler` decorator and `HandlerRegistry` for adapter identifier routing
- Request dispatch: `dispatch_request()` resolves handler by adapter identifier, invokes `process()`, returns `AdapterResponse`
- FlightRecorder: context-managed log capture that records handler execution logs and attaches them to responses
- FastAPI app factory: `create_adapter_app()` produces a configured FastAPI application with health check and process endpoints
- Composable router: `create_adapter_router()` for adapters that need custom route composition
- CoreSettings: Pydantic BaseSettings class for tenant configuration (adapter name, logging level, feature flags)
- Smoke and contract tests: pytest suite validating handler contracts, dispatch behavior, schema serialization, and app factory output
- Pre-built connectors: Gmail (OAuth2 service account), Microsoft Exchange (Graph API), SFTP, FTP

**Epic 2 — Deployment Infrastructure:**
- Default deployment files shipped inside the installed package at `src/entr_adapter_core/defaults/`
- Override resolution system with two mechanisms:
  - File-system shadow: `resolve_file()` checks `deploy/` override directory first, falls back to package default (for Dockerfile, nginx.conf, fluent-bit.conf)
  - Compose-native merge: `resolve_compose_files()` returns ordered compose file list for `-f base -f override` stacking
- Bulk resolution: `resolve_all_files()` resolves all deployment files at once, returns dict of `ResolvedFile` results
- Resolution logging: `log_resolution_summary()` produces a formatted table at INFO level showing source of each file

**Epic 3 — Versioning & Distribution:**
- Semantic versioning with single exact git tags (e.g., `v1.0.0`)
- `__version__` exposed via `importlib.metadata` — always reflects installed version from `pyproject.toml`
- `VERSIONING.md` documenting breaking change contract, migration path requirements, and release process
- `adapter-version-policy.yml` reference template for adapter version pinning (three specificity levels)
- Nightly auto-update GitHub Actions workflow template (`.github/workflows/auto-update.yml`)

### Deployment File Changes

Initial default deployment files shipped with this release:

- **`Dockerfile`**: uv-based Python 3.12 slim build, `EXPOSE 8096`, uvicorn CMD targeting `src.main:app`
- **`nginx.conf`**: Production nginx config routing to Core services (nextapp, backend), SSE support, gzip compression. Note: subsequently rewritten in v1.3.2 to match proven production patterns
- **`docker-compose.dev.yml`**: Core services (nextapp, backend, core-nginx) + adapter with live reload (`./src:/app/src` mount, `--reload` flag)
- **`docker-compose.prod.yml`**: Variable substitution (`${DOCKER_USER}`, `${TENANT_SLUG}`, `${ADAPTER_TAG}`, `${CORE_TAG}`), restart policies, fluentd logging. Note: subsequently rewritten in v1.3.2 as self-contained compose
- **`fluent-bit.conf`**: CloudWatch Logs output with `${TENANT_SLUG}` in log stream naming. Note: subsequently rewritten in v1.3.3 to use fluentd forward input

### Schema Changes

Initial contract schemas shipped with this release (all non-breaking — baseline):

- **`DocumentPayload`**: Inbound schema from Core → Adapter. Fields: `adapter_identifier`, `document_id`, `data`, `original_data`, `applied_mappings`, `original_file_content_b64`, `original_file_name`
- **`AdapterResponse`**: Outbound schema from Adapter → Core. Fields: `status` (success/failed/review_required), `review_suggestions`, `notifications`, `execution_logs`
- **`RetrievalPayload`**: Inbound schema for document retrieval requests
- **`RetrievalResponse`**: Outbound schema for document retrieval results

### Dependency Changes

Initial dependencies shipped with this release:

- **FastAPI** `>=0.100.0` — web framework for adapter HTTP endpoints
- **Pydantic** `>=2.0.0` — data validation for contract schemas and settings
- **pydantic-settings** `>=2.13.1` — environment-based configuration for CoreSettings
- **uvicorn** `>=0.20.0` — ASGI server for running adapters

Optional dependency groups:
- **sftp**: `paramiko>=3.0.0` — for SFTP/FTP connector
- **gmail**: `httpx>=0.24.0` — for Gmail OAuth2 connector
- **microsoft**: `httpx>=0.24.0` — for Microsoft Exchange Graph API connector
- **connectors**: all of the above

## [0.1.0] - 2026-03-12

### Added
- Initial project structure with `uv init --lib`
- Package skeleton with subdirectories: app, handlers, schemas, logging, connectors, utils, defaults
- Core dependencies: FastAPI, Pydantic, uvicorn
- PEP 561 py.typed marker for type checking support
