# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.7.4] - 2026-03-26

### Fixed

- **Cumulative Status Filter in Frontend Email List**: Fixed `filterEmails()` in `app.js` to use the `STATUS_IMPLIES` hierarchy instead of an exact `current_status` comparison. Emails with status "opened" or "clicked" are now correctly shown when filtering by "Delivered", mirroring the same hierarchy already applied in `update_stats()`.

## [0.7.3] - 2026-03-26

### Fixed

- **Cumulative Status Filter**: Fixed status filter in `message_emails_api` so that clicking a status counter (e.g. "Delivered") now returns all emails at that status or higher in the event hierarchy. Previously an exact match was used, causing the counter and the filtered list to disagree. 5 new tests added.

## [0.7.2] - 2026-03-17

### Fixed

- **Dark Mode Support**: Added comprehensive dark mode support for Django admin interface
  - CSS now uses Django admin CSS custom properties (--body-fg, --body-bg, --hairline-color, etc.)
  - Automatic theme adaptation based on user's Django admin theme preference
  - Improved contrast and readability in both light and dark modes
- **KPI Card Styling**: Fixed missing border on clickable KPI cards (bounced/blocked email cards)
  - Added proper 2px solid transparent border to `.kpi-card.clickable` class
  - Ensures consistent card appearance and hover effects across all KPI cards

## [0.7.1] - 2026-03-17

### Fixed

- **Tag Subject Extraction**: Fixed bug in tag-based grouping where full tag string was stored instead of just the subject
  - Webhook handler now correctly extracts subject from `digest:42:Subject` format (stores "Subject" instead of full tag)
  - CSV import command now correctly extracts subject from tag format (stores "Subject" instead of full tag)
  - Updated existing tests to reflect corrected subject extraction behavior
  - Added 2 new test cases for edge cases: subjects containing colons and tags without ID field
  - Affects tag-based grouping mode only (MESSAGE_GROUP_BY = 'tag')
  - No migration required - fix applies to new data processing only

## [0.7.0] - 2026-03-16

### Added

- **Domain-Based Sender Matching**: Enhanced `ALLOWED_SENDERS` configuration to support domain patterns in addition to exact email addresses
  - Domain patterns use `@domain.com` format to match any sender from that domain
  - Example: `'@yourcompany.com'` matches `info@yourcompany.com`, `noreply@yourcompany.com`, etc.
  - Exact email addresses (`'info@example.com'`) continue to work as before
  - Case-insensitive matching for both exact addresses and domain patterns
  - Fully backward compatible with existing configurations
- **Centralized Sender Matching Logic**: New `sender_utils` module consolidates sender validation logic
  - Shared utilities used consistently across webhook processing, ORM queries, and API responses
  - Function `matches_allowed_senders()` handles both exact and domain-based matching
  - Improved code maintainability and consistency across the package

### Changed

- **Sender Filtering Architecture**: Refactored sender validation to use centralized `sender_utils` module
  - Webhook handler now uses `matches_allowed_senders()` for sender validation
  - `BrevoEmailManager` ORM queries now use `matches_allowed_senders()` for database filtering
  - Blacklist API endpoints now use `matches_allowed_senders()` for multi-client filtering
  - Eliminated code duplication and hardcoded default senders across codebase
- **Multi-Tenant Security**: Enhanced sender filtering applies to all data access points
  - Domain patterns improve security for organizations with multiple sending addresses
  - More flexible configuration for shared Brevo accounts with multiple sender addresses

### Technical Details

- Added 28 new tests covering domain pattern matching in webhook, ORM, and API contexts
- Removed hardcoded `['info@infoparlamento.it']` default from webhook handler
- Removed hardcoded `'info@infoparlamento.it'` default from ORM manager
- Centralized configuration ensures consistent behavior across all package components
- All tests pass, no breaking changes to existing functionality

## [0.6.0] - 2026-03-16

### Added

- **Tag-Based Message Grouping**: Optional grouping strategy that uses Brevo tags instead of email subjects
  - New setting `MESSAGE_GROUP_BY`: `'subject'` (default, unchanged) or `'tag'`
  - New setting `MESSAGE_TAG_PREFIX`: prefix to match grouping tags (default: `'digest'`)
  - Tag format convention: `{prefix}:{id}:{display_title}`
  - New `tags` JSONField on `BrevoEmail` model — stores all tags from webhook/CSV regardless of grouping mode
  - New `display_subject` computed field in API serializers — strips prefix and ID for human-readable display
  - Webhook: extracts tags from payload, applies tag-based grouping when configured
  - CSV import: reads `tag` column, stores on BrevoEmail, applies tag-based grouping when configured
  - Graceful fallback: when no matching tag found, uses email subject (same as default mode)
  - Full backward compatibility: no changes to BrevoMessage model, no migration on that table
  - New migration 0007: adds `tags` JSONField to BrevoEmail (default=list)
- **Standalone Test Infrastructure**: Added `test_settings.py` and `runtests.py` for running tests without external Django project
  - Enables `python runtests.py` for local development and CI/CD
  - Self-contained test configuration with SQLite database
  - Simplifies contributor onboarding and automated testing

### Fixed

- **Django 5.2 compatibility**: Fixed `timezone.utc` deprecation in webhook handler (use `datetime.timezone.utc`)

## [0.5.1] - 2026-03-05

### Changed

- **Date formatting improvements** in email list views (thanks to @htrex):
  - `formatShortDate` now includes the year (e.g. "5 dic 2025" instead of "5 dic")
  - `formatDateTime` now includes the year and uses a "date - time" separator (e.g. "5 dic 2025 - 12:15:32")
  - Column header renamed from "Data" to "Data - Ora Invio" with a dedicated i18n key `sent_datetime`

## [0.5.0] - 2026-03-05

### Added

- **Complete Brevo webhook event mapping** (thanks to @htrex):
  - Added `invalid_email` → `bounced` and `error` → `bounced` event types
  - Added `unique_opened` → `opened` and `unique_proxy_open` → `opened` (first-open tracking)
  - Added explicit `proxy_open` → ignored (repeated proxy opens suppressed)

### Changed

- **Ignore repeated open events**: `opened` and `proxy_open` events are now ignored; only `unique_opened` and `unique_proxy_open` (first opening) are tracked to avoid inflating open counts
- **CSV import**: `Aperta` events (repeated opens) are now ignored; only `Prima apertura` is tracked, consistent with webhook behaviour
- **Fix `unsubscribe` key**: corrected event mapping key from `unsubscribe` to `unsubscribed` to match Brevo API documentation
- **Fix ip/user_agent capture**: open metadata (IP, user agent) now correctly captured for `unique_opened` and `unique_proxy_open` events

## [0.4.0] - 2026-02-20

### Changed

- **Django 5.x Compatibility** (thanks to @htrex):
  - Widened Django version constraint from `>=4.2,<5.0` to `>=4.2,<6.0`
  - Replaced deprecated `unique_together` with `UniqueConstraint` in both `BrevoMessage` and `BrevoEmail` models
  - Added migration 0006 to convert existing unique_together constraints to named UniqueConstraints
  - Added Django 5.0 and 5.1 framework classifiers to pyproject.toml
  - Fully backward compatible with Django 4.2 (UniqueConstraint available since Django 2.2)

### Fixed

- **Documentation**: Corrected URL paths in README.md to align with frontend API implementation
  - Changed URL mounting from `/admin/brevo_analytics/` to `/brevo-analytics/` to match Vue.js SPA expectations
  - Updated webhook URL documentation from `/admin/brevo_analytics/webhook/` to `/brevo-analytics/webhook/`
  - Added clarifying note that API endpoints are mounted at `/brevo-analytics/` while dashboard access remains at Django admin URL `/admin/brevo_analytics/brevomessage/`

## [0.3.0] - 2026-02-09

### Added

- **Blacklist-Only Mode**: New configuration flag for deployments that only need blacklist management without full email analytics
  - **Configuration**: Set `BLACKLIST_ONLY_MODE = True` in `BREVO_ANALYTICS` settings
  - **Use Case**: Ideal for scenarios where email subjects vary per client (e.g., Comin with customizable subjects), making subject-based message grouping impractical
  - **Behavior**:
    - Admin interface shows only "Blacklist Management" menu (hides "Message Analysis")
    - Webhook endpoint returns 404 with informative message
    - Import and statistics commands exit with clear error messages
    - Database tables are created but remain unused
    - Blacklist SPA works normally via direct Brevo API queries
  - **Commands disabled**: `import_brevo_logs`, `clean_internal_emails`, `recalculate_stats`, `verify_brevo_stats`
  - **Backwards compatible**: Default is `False`, existing installations continue working normally

### Technical Details

- **Admin Registration** (`admin.py`):
  - Refactored from `@admin.register()` decorators to conditional registration
  - `BrevoMessageAdmin` registered only when `BLACKLIST_ONLY_MODE = False`
  - `BrevoEmailAdmin` always registered for blacklist management

- **Webhook Handler** (`webhooks.py`):
  - Added early mode check at function start
  - Returns JSON response with `status: "disabled"` and HTTP 404 when mode enabled
  - Includes logging for monitoring webhook calls in blacklist-only mode

- **Management Commands**:
  - `import_brevo_logs.py`: Added mode check, exits with styled error message
  - `clean_internal_emails.py`: Added mode check to prevent database cleanup operations
  - `recalculate_stats.py`: Added mode check to prevent statistics recalculation
  - `verify_brevo_stats.py`: Added mode check to prevent API verification
  - All commands show consistent error message directing users to Blacklist Management

- **Test Coverage** (`tests.py`):
  - Added `BlacklistOnlyModeTestCase` with 3 test methods
  - Tests webhook returns 404 in blacklist-only mode
  - Tests import command exits with error in blacklist-only mode
  - Tests webhook works normally when mode disabled
  - All 8 tests pass successfully

- **Documentation**:
  - Added comprehensive "Blacklist-Only Mode" section to `docs/README.md`
  - Added detailed configuration guide to `CLAUDE.md` with verification steps
  - Documented all disabled commands and expected behavior
  - Included example configurations for blacklist-only deployments

## [0.2.6] - 2026-02-03

### Fixed

- **Webhook Event Recovery**: Webhook now creates email records from 'delivered' events when 'sent' event was missed or delayed
  - **Problem**: Brevo doesn't guarantee event order - 'delivered', 'opened', or 'clicked' events can arrive before 'sent' event
  - **Previous behavior**: Webhook ignored all events for unknown emails, waiting for 'sent' event that might never arrive
  - **New behavior**:
    - 'delivered' event is treated as proof of sending and triggers record creation
    - When creating from 'delivered': adds inferred 'sent' event to maintain statistics consistency
    - If real 'sent' event arrives later: replaces inferred 'sent' with real one (correct timestamp)
  - **Impact**: Improves delivery rate accuracy by recovering ~6% of emails where 'sent' event arrives late or is lost
  - **Benefit**: Reduces false error reports in Brevo webhook dashboard (from ~250 errors to near zero)

### Technical Details

- **Webhook changes** (`webhooks.py`):
  - Added `is_delivered_event` check alongside `is_sent_event` for record creation
  - When creating from 'delivered' event:
    - Sets initial status to 'delivered' instead of 'sent'
    - Adds inferred 'sent' event with `inferred: True` flag to events array
    - Uses delivered timestamp as sent_at (close approximation)
  - Updated log messages to distinguish between normal creation and recovery from delivered event
  - Other events (opened, clicked) still require prior sent/delivered event

- **Model changes** (`models.py`):
  - Modified `BrevoEmail.add_event()` to handle inferred 'sent' events
  - When adding real 'sent' event: automatically removes any inferred 'sent' events first
  - Prevents duplicate 'sent' events in statistics and timeline
  - Ensures correct sent_count in message statistics

- **Event ordering scenarios**:
  - Normal: sent → delivered → opened → clicked (no change in behavior)
  - Delayed: delivered → opened → sent (now handled: creates from delivered, updates when sent arrives)
  - Missing: delivered → opened → clicked (no sent ever) (now handled: uses inferred sent)

## [0.2.5] - 2026-01-29

### Removed

- **Blacklist Management UI**: Removed "Arricchisci DB" (Enrich Database) button from blacklist management interface
  - Button allowed enriching local blocked emails database with information from Brevo blacklist
  - Functionality removed as it's no longer needed in the current workflow
  - Simplified UI by removing unnecessary operation button
  - Removed `enrichDatabase()` function and related state management from Vue.js component

### Technical Details

- Removed enrichment button from ListAllTab component template
- Removed `enriching` reactive state reference
- Removed `enrichDatabase` async function
- Cleaned up component return statement to remove unused references

## [0.2.4] - 2026-01-28

### Fixed

- **Internationalization**: Fixed untranslated KPI filter labels in Message Emails view
  - Status filter buttons (Sent, Delivered, Opened, Clicked, Bounced, Blocked) now properly use i18n translation keys
  - Labels were previously hardcoded in English, preventing proper localization
  - All KPI filter labels now use `$t()` function for dynamic translation based on user locale
  - Improves user experience for non-English speakers

### Technical Details

- Updated Vue.js SPA template to replace hardcoded English labels with i18n keys
- Modified KPI filter button rendering in Message Emails view component
- Maintains existing functionality while enabling proper multilingual support

## [0.2.3] - 2026-01-27

### Fixed

- **Webhook Authentication**: Fixed critical bug where webhook validation was looking for non-existent `X-Brevo-Signature` HMAC header
  - Brevo actually sends authentication as `Authorization: Bearer <token>` header
  - Changed webhook authentication from HMAC signature validation to Bearer token validation
  - This bug caused all webhook requests to fail with "Invalid webhook signature" warnings since v0.1.0
  - Webhook now correctly validates the Bearer token from `Authorization` header against `WEBHOOK_SECRET` setting
- **Sender Email Extraction**: Enhanced sender field extraction to support `sender_email` field in webhook payloads
  - Webhook now checks `sender`, `from`, and `sender_email` fields (in that order) to extract sender information
  - Eliminates "no sender information" warnings for events that include `sender_email` field

### Technical Details

- Removed unused `hmac` and `hashlib` imports from webhook handler
- Simplified authentication logic: direct string comparison of Bearer token instead of HMAC computation
- Maintains backward compatibility with existing `WEBHOOK_SECRET` configuration

## [0.2.2] - 2026-01-27

### Fixed

- **Missing Migration File**: Included Django migration file (`0005_brevoemail_sender_email.py`) that was inadvertently omitted from v0.2.1 package
  - Users who installed v0.2.1 encountered errors when running `python manage.py migrate brevo_analytics`
  - Migration creates the `sender_email` field required for sender validation introduced in v0.2.1
  - This hotfix completes the security patch from v0.2.1 by providing the required database schema changes

### Migration Note

**For users who installed v0.2.1:**
- Upgrade immediately to v0.2.2 to get the missing migration file
- Run `python manage.py migrate brevo_analytics` after upgrading
- No other changes required - all configuration from v0.2.1 remains valid

**For new installations:**
- This version includes all migrations needed for the sender validation security feature
- Follow the standard installation and configuration steps from the README

### Technical Details

- Added migration file: `brevo_analytics/migrations/0005_brevoemail_sender_email.py`
- Creates `sender_email` field on `BrevoEmail` model (nullable CharField, indexed)
- No code changes - purely a packaging fix to include the migration in the distribution

## [0.2.1] - 2026-01-27

### Security

**CRITICAL SECURITY PATCH - IMMEDIATE UPDATE REQUIRED**

- **Multi-Tenant Data Contamination Vulnerability**: Fixed critical security flaw where webhook accepted events from ANY sender on shared Brevo account
  - **Impact**: Before this fix, webhook processed events from all clients sharing the same Brevo account, mixing data from different organizations into the same database
  - **Severity**: CRITICAL - Potential for unauthorized access to analytics data from other tenants
  - **Resolution**: Webhook now validates sender email against `ALLOWED_SENDERS` configuration before processing any events

### Added

- **Sender Email Tracking**: New `sender_email` field in `BrevoEmail` model
  - Captures sender address from every Brevo event
  - Enables sender-based filtering and data isolation
  - Indexed for fast queries
- **Sender Validation**: Automatic sender verification in webhook processing
  - Only events from senders in `BREVO_ANALYTICS['ALLOWED_SENDERS']` are processed
  - Unauthorized sender events are logged and rejected
  - Prevents data contamination from other tenants on shared Brevo accounts
- **ORM-Level Sender Filtering**: Enhanced `BrevoEmailManager` with automatic sender filtering
  - All database queries automatically exclude unauthorized senders
  - Transparent filtering - no code changes required in existing applications
  - Works with all Django ORM methods (filter, exclude, annotate, etc.)
- **Management Command**: `verify_senders` - Identify potentially contaminated data
  - Scans database for emails from unauthorized senders
  - Reports statistics on data contamination
  - Supports dry-run mode to preview issues before cleanup
  - Helps assess impact of vulnerability on existing installations

### Changed

- **Database Schema**: Added `sender_email` field to `BrevoEmail` model
  - **Migration Required**: Run `python manage.py migrate brevo_analytics` after update
  - Nullable field for backward compatibility with existing data
  - Future webhook events will populate this field automatically
- **Configuration Requirement**: `ALLOWED_SENDERS` setting is now MANDATORY
  - Must be configured in `BREVO_ANALYTICS['ALLOWED_SENDERS']` setting
  - List of authorized sender email addresses for your organization
  - Events from unlisted senders will be rejected
  - Example: `ALLOWED_SENDERS = ['noreply@example.com', 'alerts@example.com']`
- **Webhook Behavior**: Enhanced event processing with sender validation
  - Extracts sender email from `from` or `sender` webhook fields
  - Validates against ALLOWED_SENDERS before database operations
  - Logs rejection of unauthorized sender events for audit trail

### Migration Guide

**CRITICAL - Action Required for All Installations:**

1. **Update Package**:
   ```bash
   pip install --upgrade django-brevo-analytics==0.2.1
   ```

2. **Configure ALLOWED_SENDERS** in Django settings:
   ```python
   BREVO_ANALYTICS = {
       'WEBHOOK_SECRET': 'your-webhook-secret',
       'CLIENT_UID': 'your-client-uuid',
       'ALLOWED_SENDERS': [
           'noreply@yourcompany.com',
           'alerts@yourcompany.com',
           # Add all legitimate sender addresses for your organization
       ],
   }
   ```

3. **Run Database Migration**:
   ```bash
   python manage.py migrate brevo_analytics
   ```

4. **Verify Existing Data** (optional but recommended):
   ```bash
   # Check for potentially contaminated data
   python manage.py verify_senders
   ```

5. **Re-import Historical Data** (recommended):
   ```bash
   # Clear and reimport to populate sender_email field
   python manage.py import_brevo_logs /path/to/logs.csv --clear
   ```

### Impact Assessment

**Before Fix:**
- Webhook accepted events from ANY sender on shared Brevo account
- Analytics data could include emails from other organizations
- No sender validation or isolation between tenants
- Potential for data leakage in multi-tenant Brevo accounts

**After Fix:**
- Only authorized senders (configured in ALLOWED_SENDERS) are processed
- Automatic sender validation on all webhook events
- ORM-level filtering ensures unauthorized data never appears in queries
- Complete data isolation for multi-tenant environments

**Affected Versions**: All versions prior to 0.2.1

**Recommended Action**: Immediate update for all installations, especially those on shared Brevo accounts

## [0.2.0] - 2026-01-27

### Added
- **Internal Domain Filtering System**: Comprehensive three-level filtering to exclude internal/test emails from analytics
  - Configure excluded domains via `BREVO_ANALYTICS['EXCLUDED_RECIPIENT_DOMAINS']` setting
  - Automatic filtering during CSV import prevents internal emails from entering the database
  - Real-time webhook filtering blocks internal domain events before processing
  - Model-level query filtering ensures internal emails never appear in analytics views or API responses
- **Management Command**: `clean_internal_emails` - Remove existing internal emails from database
  - Supports dry-run mode to preview deletions before applying
  - Automatically recalculates message statistics after cleanup
  - Useful for cleaning up data imported before domain filtering was configured
- **Management Command**: `recalculate_stats` - Recalculate statistics for all messages
  - Rebuild denormalized statistics from event data
  - Useful after data cleanup or manual database changes
  - Ensures dashboard metrics remain accurate

### Fixed
- **Statistics Accuracy**: Fixed critical bug in `BrevoMessage.update_stats()` that was counting all emails in the database instead of only emails with 'sent' events for the specific message
  - Delivery rate, open rate, and click rate calculations now correctly reflect actual sent emails
  - Prevents inflated or incorrect percentage metrics in dashboard
- **Webhook Event Processing**: Webhook now correctly ignores events that arrive without a prior 'sent' event in the database
  - Prevents orphaned events from creating incomplete email records
  - Ensures all tracked emails have complete event history starting from 'sent'

### Changed
- **Email Model**: Added custom `BrevoEmailQuerySet` and `BrevoEmailManager` for automatic domain filtering at the ORM level
  - All queries automatically exclude internal domains without manual filtering
  - Transparent to existing code - filtering happens automatically
- **Import Command**: Enhanced `import_brevo_logs` to filter internal domains during CSV processing
  - Reduces database size by excluding test/internal emails from the start
  - Improves import performance by skipping unnecessary records
- **Webhook Processing**: Updated webhook handler to filter internal domains in real-time
  - Prevents test emails from affecting production analytics
  - Reduces database writes for non-production events

### Technical Details
- All changes are backward compatible with existing configurations
- Domain filtering is optional - package works without `EXCLUDED_RECIPIENT_DOMAINS` configuration
- Custom manager ensures filtering works with all Django ORM query methods (filter, exclude, annotate, etc.)
- Statistics recalculation is automatically triggered after cleanup operations

## [0.1.1] - 2026-01-22

### Changed
- Updated README.md to remove all Supabase references
- Updated documentation to reflect Django-native architecture
- Added comprehensive setup instructions for DRF and CORS
- Added management commands documentation
- Updated troubleshooting section for current architecture

### Fixed
- Multi-client blacklist filtering: now correctly filters by ALLOWED_SENDERS and local database
- Emails with empty senderEmail (hard bounces) now properly included when in local DB
- Prevents showing blacklisted emails from other clients on shared Brevo accounts

## [0.1.0] - 2026-01-22

### Added
- Initial release of django-brevo-analytics
- Django-native architecture with models stored in PostgreSQL
- Django REST Framework API endpoints for analytics data
- Vue.js Single Page Application (SPA) for interactive analytics viewing
- Real-time webhook integration for Brevo events
- Dashboard with KPI metrics:
  - Total emails sent, delivery rate, open rate, click rate
  - Bounced and blocked emails count
  - Recent messages list
- Message-level email tracking with status filtering
- Email detail modal with complete event timeline
- Blacklist management interface:
  - Check individual emails for blacklist status
  - View and manage all blacklisted emails
  - Integration with Brevo API for real-time blacklist data
  - Remove emails from blacklist directly from UI
- Internationalization (i18n) support:
  - English and Italian translations
  - JavaScript-based UI localization
  - Django model verbose names localization
- Historical data import from raw Brevo logs (CSV)
- Automatic bounce reason enrichment via Brevo API
- Statistics verification command against Brevo API
- DuckDB-based CSV import for efficient data processing
- JSONField-based event storage for optimal performance

### Technical Details
- Python 3.8+ support
- Django 4.2+ support
- Django REST Framework integration
- Vue.js 3 with Composition API
- Hash-based routing (Vue Router in-memory)
- HMAC signature validation for webhooks
- Denormalized statistics for fast queries
- Cached status fields for efficient filtering
- Multi-client filtering via ALLOWED_SENDERS configuration
- Modal-based UI for seamless navigation
- Comprehensive management commands:
  - `import_brevo_logs`: Import historical data from CSV
  - `verify_brevo_stats`: Verify statistics against Brevo API
