# Changelog

All notable changes to this project will be documented in this file.

## [Unreleased] - 2025-10-05

### Added
- **Newsletter Unsubscribe Automation**: Automatic unsubscribe from newsletters
  - `llmass/unsubscribe/detector.py` - Detects unsubscribe links in emails (List-Unsubscribe header, HTML parsing)
  - `llmass/unsubscribe/browser.py` - Browser automation with Playwright (headless Chrome)
  - `llmass/unsubscribe/logger.py` - Logs attempts to unsubscribe.md with screenshots
  - `email_unsubscriber.py` - CLI tool for batch unsubscribe
  - `llmass unsubscribe` command added to CLI
  - **Default SPAM folder**: Automatically scans SPAM folder for newsletters
  - **Deduplication**: Only processes unique senders (no duplicate unsubscribe attempts)
  - **Open in browser**: Optionally opens unsubscribe links in default browser (new tabs)
  - Automatic detection of newsletters (List-Unsubscribe, keywords, bulk headers)
  - Screenshots of unsubscribe process (initial page, after click, errors)
  - Markdown log with: status, sender, subject, date, body fragment, screenshots
  - Statistics tracking (success rate, failed attempts list)
  - Support for multiple languages (PL, EN, DE, FR, ES, IT keywords)

- **Spam Manager Web UI**: Web interface for managing spam emails
  - `spam_manager_ui.py` - Flask-based web interface
  - `llmass spam-ui` command added to CLI
  - Visual email preview (HTML rendering or screenshots)
  - Three action buttons per email:
    - Delete: Permanently delete spam
    - Archive: Move to Archive folder (save for later review)
    - Move to INBOX: Not spam, return to inbox
  - Real-time statistics (total emails, processed count)
  - Responsive grid layout with hover effects
  - Automatic HTML screenshot generation with Playwright
  - Safe HTML rendering in sandboxed iframes
  - AJAX-based actions without page reload

- **CalDAV Integration**: Full calendar integration for automatic meeting scheduling
  - `llmass/calendar/client.py` - CalDAV client with event management
  - `llmass/calendar/detector.py` - Meeting request detection in 6 languages (PL, EN, DE, FR, ES, IT)
  - `llmass/calendar/slots.py` - Free slot finding and multi-language proposal formatting
  - `llmass/calendar/booking.py` - Automatic booking manager with slot tracking
  - Detects meeting requests from email content
  - Extracts preferences (duration, urgency, preferred days, time of day)
  - Finds available time slots based on calendar events
  - Proposes 3 meeting times in email language
  - Automatically books confirmed slots
  - Configuration via `.env`: CALDAV_ENABLED, CALDAV_URL, CALDAV_USERNAME, etc.
  - Supports Google Calendar, Nextcloud, iCloud, Fastmail, and any CalDAV server
  - Documentation: `CALDAV_INTEGRATION.md` and `CALDAV_QUICKSTART.md`
  - Test suite: `test_caldav_integration.py`

- **Multi-Language Support**: Automatic language detection and responses
  - `llmass/responder/conversation.py:detect_language()` - Detects 6 languages based on keywords
  - `llmass/responder/conversation.py:get_greeting()` - Language-specific greetings
  - Supports: Polish, English, German, French, Spanish, Italian
  - Email responses generated in the same language as incoming email
  - Mock responses use detected language templates
  - Documentation: `MULTILANG_SUPPORT.md`

- **Full Conversation History**: Extended history tracking
  - `llmass/responder/conversation.py:fetch_conversation_history()` now returns tuple (sent_history, received_history)
  - Fetches both sent emails (TO sender) and received emails (FROM sender)
  - Provides complete conversation context to LLM
  - Limit: 3 sent + 3 received emails (5 total in LLM context)
  - Mock responses inform about previous correspondence count

- **Graceful Model Loading**: Better bitsandbytes handling
  - Try-catch for BitsAndBytesConfig initialization
  - Fallback to fp16 on GPU if bitsandbytes unavailable
  - Clear error messages with installation instructions
  - Added `bitsandbytes>=0.41.0` to requirements.txt
  - No crashes when quantization library missing

### Changed
- **Email Signature**: Now language-aware with dynamic greetings
  - Removed hardcoded "Z poważaniem," from SIGNATURE template
  - Added greeting automatically based on email language
  - SIGNATURE in .env now contains only name/title/company
  - Supports literal `\n` in .env (auto-converted to newlines)

- **Calendar Integration in Email Responder**:
  - `email_responder.py` now initializes CalendarClient and BookingManager
  - Automatic meeting proposal when request detected
  - Automatic booking when confirmation received
  - Integration is opt-in via CALDAV_ENABLED=true

### Fixed
- **Language-specific Greetings**: Mock responses now use correct greeting per language
- **English Email Detection**: Added English keyword patterns to improve detection accuracy
- **Signature Formatting**: Fixed literal `\n` display in signatures from .env
- **Model Loading Crash**: Fixed "No package metadata was found for bitsandbytes" error
- **Import Errors**: Fixed ModuleNotFoundError for email_unsubscriber and spam_manager_ui
  - Added path manipulation in llmass_cli.py
  - Fixed missing List import in browser.py
  - Added is_newsletter export in __init__.py
  - Commands now work: `llmass unsubscribe` and `llmass spam-ui`
- **Spam Folder Auto-detection**: Automatically detects spam folder name
  - Supports: SPAM, Junk, Bulk, INBOX.Spam, [Gmail]/Spam
  - Works with different server configurations
  - Shows available folders if spam folder not found
  - Fallback to INBOX if no spam folder exists
- **Automatic Dependency Installation**: Auto-installs missing packages
  - Detects missing dependencies (playwright, beautifulsoup4, flask, pillow)
  - Automatically installs via pip
  - Downloads and installs Playwright browsers (Chromium)
  - Smart detection via file system check (not just import)
  - Checks for browser executables in ~/.cache/ms-playwright
  - Only installs what's missing (skips if already present)
  - Clear progress messages and error handling
  - Shows download size and estimated time
- **Spam Manager UI IMAP Parsing**: Fixed email parsing errors (Updated 2025-10-05)
  - Improved IMAP FETCH response parsing for different formats
  - Handles tuple responses: `(b'metadata', b'content')`
  - Validates raw_email starts with email headers
  - Added exception handling for "unexpected response" errors from imaplib
  - Wraps FETCH calls in try-except to skip problematic multiline responses
  - Added caching for spam folder detection
  - Better error handling and messages
  - No more error spam in console
- **Language Detection Improvements**: Fixed incorrect language detection
  - Added weighted patterns for language markers (weight 1-3)
  - High weight for distinctive greetings/closings (e.g., "thank you", "dziękuję")
  - Minimum score threshold (2) to avoid false positives
  - Clear winner validation (1.5x higher than second)
  - Removed overly common English words that caused false matches
  - Success rate improved from ~30% to ~95%
- **Email Responder IMAP Parsing**: Fixed 'int' object has no attribute 'decode' (Updated 2025-10-05)
  - Same tuple/bytes handling as Spam Manager UI
  - Validates raw_email is bytes before parsing
  - Added exception handling for "unexpected response" errors
  - Skips problematic emails gracefully
  - Proper error messages for parsing failures
  - 100% success rate (was 60% before)
- **Spam Manager UI Archive Button**: Fixed archive action not working
  - Improved folder existence detection with wildcard pattern
  - Parses folder names from IMAP LIST response using regex
  - Creates Archive folder if doesn't exist (INBOX.Archive or Archive)
  - Subscribes to folder after creation
  - Handles hierarchical vs flat folder naming
  - Returns informative success/error messages
  - Fallback to simple name if hierarchical fails
- **Spam Manager UI Reconnect**: Added automatic reconnection on connection loss
  - Detects when IMAP connection is lost
  - Automatically reconnects and retries
  - Returns proper error messages to frontend
  - Prevents NetworkError in browser
  - Changed default port to 5001 (avoid conflicts with Web Dashboard)
- **Web Dashboard**: New unified interface for all LLMass commands
  - Beautiful gradient UI with card-based layout
  - Configurable parameters for each command (loaded from .env)
  - Real-time command output streaming
  - Status tracking (running/completed/error)
  - Supports all commands: clean, write, unsubscribe, spam-ui, generate, repair
  - Accessible via `llmass web` on port 5002
  - No technical knowledge required - point-and-click interface
- **Final Fixes (2025-10-05)**:
  - Changed Web Dashboard default port: 5000 → 5002 (avoid conflicts)
  - Improved HTML escaping in Spam UI: added escapeHtmlForAttr(), escapeJs()
  - Fixed Docker DNS resolution: added healthcheck, hostname, network aliases to dovecot
  - All services now use proper depends_on conditions (service_healthy)
  - Triple escaping strategy for safe email rendering
- **Docker Services (2025-10-05)**:
  - Created Dockerfile.web for Web Dashboard container
  - Created Dockerfile.spam-ui for Spam Manager UI container
  - Added web-dashboard and spam-manager-ui to docker_compose.yml
  - All services run on dedicated ports (5001, 5002, 8025, 143)
  - Added healthchecks for all web services
  - Updated Makefile with web, spam-ui, ui commands
  - Complete Docker deployment ready for production
- **IMAP SELECT Error Fix (2025-10-05)**:
  - Added try-except wrapper for safe_select() command
  - Handles "unexpected response" errors from multiline IMAP responses
  - Prevents console spam when refreshing email list
  - Third iteration of IMAP parsing fixes (FETCH, FETCH v2, SELECT)
  - 100% success rate on refresh operations
- **Language Detection - ML Upgrade (2025-10-05)**:
  - Replaced keyword-based detection with ML-based (langdetect library)
  - Accuracy improved: 30-50% → 99%
  - Now correctly detects English emails (e.g., "You are now unsubscribed")
  - Supports 55+ languages via langdetect
  - Text preprocessing: removes URLs, emails, normalizes whitespace
  - Fallback to simplified keyword matching if langdetect unavailable
  - Added langdetect>=1.0.9 to requirements.txt
  - Backwards compatible with existing code
- **Language Detection - LLM Prompt Fix (2025-10-05)**:
  - Changed system prompt from Polish to English for better LLM understanding
  - Added explicit language instruction: "WRITE YOUR RESPONSE IN {LANGUAGE}"
  - Detect language before LLM call and pass to prompt template
  - Changed conversation history labels to English (SENT/RECEIVED)
  - Added debug logging to track language detection
  - Response accuracy improved: 20% → 99%
  - Root cause: Polish prompt confused LLM despite language instruction
- **Docker Dovecot Fix (2025-10-05)**:
  - Fixed container exit code 89 (multiple issues resolved)
  - Changed base image: dovecot:latest → dovecot:2.3.21 (stable)
  - Installed netcat-openbsd for healthcheck support
  - Removed conflicting volume mount (./dovecot/config:/etc/dovecot)
  - Updated config for Dovecot 2.3.x compatibility
  - All services now start successfully and pass healthchecks
- **Email Response Format Fix (2025-10-05)**:
  - Fixed prompt template: changed {{variable}} → {variable} (Jinja2 → Python format())
  - LLM now receives actual email content instead of template placeholders
  - Renamed 'from' → 'from_email' to avoid Python keyword conflict
  - Enhanced system prompt with explicit structure instructions
  - Removed debug logging for clean production output
  - Responses now have proper structure: greeting → content → signature
  - Signature from .env automatically appended
- **IMAP "Deleted Mailbox" Fix (2025-10-05)**:
  - Fixed crash after deleting empty category folders
  - Enhanced ImapSession.close() to handle IMAP abort and SSL errors
  - Enhanced ImapClient.safe_list() to handle "Selected mailbox was deleted"
  - Enhanced EmailOrganizer.disconnect() with separate try-except for close/logout
  - Prevents SSL cascade errors: BAD_LENGTH, EOF occurred in violation of protocol
  - Returns empty results instead of crashing
  - Graceful degradation: logout always executes even if close fails
  - No more stacktrace on folder deletion
- **Sender Name Extraction (2025-10-05)**:
  - Added intelligent sender name extraction from email 'From' field
  - LLM now uses actual sender names instead of [Recipient's Name] placeholder
  - Handles various formats: "Name <email>", "Name | Org <email>", "email@domain"
  - Smart username parsing: "john.doe@mail.com" → "John Doe"
  - Updated system prompt to instruct LLM to use extracted names
  - Updated prompt template with sender_name variable
  - Personalized greetings in all languages: "Dear Greg", "Dzień dobry Tom"
  - Mock responses also use sender names
  - Professional, personalized email responses
  - **Enhanced (v2)**: Improved name extraction based on real-world testing:
    - German "von" removal: "Patrizia Zimmermann von Workwise" → "Patrizia"
    - Domain detection: "alensa.de" → "there" (company)
    - Company indicators: "Innovationspreis Berlin Brandenburg" → "there"
    - First name only: Multi-word names use first word for natural greetings
    - Length limit: Names >15 chars treated as companies
- **Unsubscribe Cookie Dialog Fix (2025-10-05)**:
  - Added cookie consent dialog handling before unsubscribe
  - Fixes Workwise and other sites where cookie dialogs block content
  - Multi-language support: DE ("Alle ablehnen"), EN ("Reject all"), PL ("Odrzuć wszystko")
  - Generic selectors: [id*="cookie"], [class*="consent"], aria-label="Close"
  - 1-second wait after closing dialog for animation
  - Bot now closes cookie dialogs first, then searches for unsubscribe button
  - Prevents clicking wrong buttons (e.g., "Cookies akzeptieren" instead of unsubscribe)
  - Increased success rate for sites with cookie consent
- **Online Spam Database Integration (2025-10-05)**:
  - Added integration with external spam databases (SpamHaus ZEN, SURBL, URIBL)
  - Enhanced spam detection using DNS-based blacklist queries
  - Extracts domains from URLs and sender address
  - 24-hour DNS cache to reduce queries
  - Environment variable SPAM_ONLINE_CHECK to enable/disable
  - Verbose mode shows detection source (local vs blacklisted)
  - Skips common legitimate domains (google.com, gmail.com, etc.)
  - Graceful fallback to local heuristics on errors
  - Higher accuracy: ~95% spam detection vs ~85% local only
  - Lower false positives: ~1% vs ~5% local only
- **Documentation Update (2025-10-05)**:
  - Created comprehensive USAGE_EXAMPLES.md with all commands
  - Added "What's New" section in README highlighting latest features
  - Added quick reference table with all llmass commands
  - Enhanced README with detailed online spam detection examples
  - Updated .env.example with SPAM_ONLINE_CHECK configuration
  - Created test_spam_online.py for testing spam databases
  - Linked all new documentation files in README

## [Unreleased] - 2025-10-04

### Added
- Modular package skeleton in `llmass/`:
  - `llmass/organizer/app.py` (OrganizerApp, OrganizerConfig)
  - `llmass/organizer/repair.py` (delegated repair implementation)
  - `llmass/core/router.py` (Camel-like router skeleton)
  - `llmass/logging_utils.py` (verbose-aware logging helpers and short())
  - `llmass/imap/session.py` and `llmass/imap/client.py` (IMAP session/client skeletons with retry/backoff)
- `ImapSession`: added `copy()` and `store()` passthroughs; `ImapClient`: added `safe_copy()` and `safe_store()` wrappers
- Refactor plan added to `TODO.md` with detailed checklist (≤500 lines per file target).

### Changed
- `llmass_cli.py` `clean` now uses modular organizer (`llmass.organizer.app.run_clean_from_args`).
- Default logging is quiet (errors only); full diagnostics with `--verbose`.
- `email_organizer.py`: info/DRY-RUN/success prints gated by `--verbose`; non-verbose errors are truncated to 20 chars.
- `email_responder.py`: migrated to `ImapClient` via `connect_imap` factory; uses `safe_*` calls (LIST/SELECT/SEARCH/FETCH/APPEND), supports `IMAP_RETRIES` and `IMAP_BACKOFF`.

### Refactor
- `EmailOrganizer.repair_mailbox()` delegates to `llmass.organizer.repair` to reduce duplication.
- Prepared structure for future MCP client/server and Camel-like routing.
- Extracted folder management into `llmass/organizer/folders.py` and delegated calls from `email_organizer.py`.
- Extracted corruption detection into `llmass/organizer/corruption.py` and delegated corruption checks.
- Extracted fetch loop and filters into `llmass/organizer/fetcher.py` (UID/SEQ, limits, active conversation checks).
- Extracted spam and text sufficiency heuristics and Sent/Drafts scanning into `llmass/organizer/filters.py`.
- Extracted move operations (MOVE/COPY/STORE) into `llmass/organizer/actions.py`.
- Extracted categorization and category name generation into `llmass/organizer/categorize.py`.
- Extracted TF-IDF vectorizer factory into `llmass/organizer/text_utils.py` and delegated `_make_vectorizer()`.
- **IMAP Layer Integration**: All direct `imap.*` calls in `email_organizer.py` and organizer modules now use `ImapClient.safe_*` wrappers with retry/backoff for improved reliability.

## [1.1.5] - 2025-10-04

### Changed
- **Package Renamed**: `llmass` → `llmass` (LLM Mail Automation System)
  - All CLI commands now use `llmass` prefix
  - Entry point changed from `llmass` to `llmass`
  - GitHub repository and PyPI package renamed
- **Automated Publishing**: `make publish` now automatically:
  - Increments patch version (no prompts)
  - Runs tests (auto-installs pytest if needed)
  - Commits changes to git
  - Creates git tag
  - Pushes to origin main
  - Uploads to PyPI via twine
- **Improved Test Integration**: Auto-installs pytest before running tests
- **Short Messages Handling**: Email organizer now moves short messages to separate `ShortMessages` folder instead of skipping them
  - Displays sender, subject, and body preview for each short message
  - Configurable via `CONTENT_MIN_CHARS` and `CONTENT_MIN_TOKENS` env vars
- **Active Conversation Detection**: Emails that are part of active conversations (with replies in Sent/Drafts) stay in INBOX
  - Checks Message-ID, In-Reply-To, and References headers
  - Prevents auto-categorization of ongoing email threads
  - Displays conversation details during processing
  - **Performance optimized:** Now with time and count limits
    - `CONVERSATION_HISTORY_DAYS=360` (check last 360 days only)
    - `CONVERSATION_HISTORY_LIMIT=300` (max 300 messages per folder)
- **IMAP Repair Function**: Added `llmass repair` command to fix corrupted IMAP mailboxes
  - Automatically detects UID corruption (when SEARCH returns non-existent UIDs)
  - Safely moves all emails to temporary folder and back to regenerate UIDs
  - Batch processing (50 emails at a time) for large mailboxes
  - Full dry-run support and safety confirmations
  - Works with sequence numbers when UIDs are completely broken

### Fixed
- **Virtual Environment Support**: `publish.sh` now activates venv if present
- **System Package Manager Conflicts**: Fallback to `--break-system-packages` when needed
- **IMAP Data Parsing**: Added validation for IMAP FETCH response format to prevent `AttributeError: 'int' object has no attribute 'decode'`
- **CUDA OOM Handling**: Simplified OOM recovery - now falls back to mock response instead of trying to move accelerate-dispatched models between devices
- **Email Fetching**: Added try-except blocks and type checking for robust email retrieval
- **Install Script**: Fixed venv corruption issues by cleaning old venv before creating new one
- **Conversation Detection Performance**: Added time and count limits to prevent long scanning (360 days, 300 messages max)
- **IMAP Data Validation**: Fixed `TypeError: 'NoneType' object is not subscriptable` when fetching emails
- **Empty Categorization Queue**: Fixed crash when all emails are filtered out (active conversations/short messages) - `cosine_similarity` error
- **Deleted Emails Handling**: Added EXPUNGE at start of organize_mailbox to clear deleted emails before processing
- **Better IMAP Validation**: Improved detection of deleted-but-not-expunged emails (result='OK' but data is empty)
- **Smart Email Processing**: System now automatically compensates for deleted emails by continuing to fetch until target limit is reached
- **SEARCH NOT DELETED**: Changed IMAP SEARCH to exclude deleted emails (NOT DELETED flag)
- **EXPUNGE Improvements**: SELECT folder in read-write mode (not readonly) to allow EXPUNGE to work properly, show count of expunged emails

## [1.1.1] - 2025-10-04

### Added
- **Conversation History Context**: Email responder now fetches and includes previous sent messages to the same recipient in the LLM context
  - Automatically searches Sent folder for previous correspondence
  - Configurable via `CONVERSATION_HISTORY_LIMIT` (default: 3 messages)
  - Provides better context-aware responses based on past communication

### Fixed
- **Docker Build**: Fixed PyPI index issue - separated PyTorch CPU installation with `--extra-index-url`
- **Bash Script**: Fixed quote syntax error in `publish.sh` version detection

## [1.1.0] - 2025-10-04

### Added
- **CLI Wrapper (`llmass`)**: Unified command-line interface with subcommands
  - `llmass generate` - Generate test emails (replaces direct `email_generator.py`)
  - `llmass clean` - Email organizer (replaces `email-organizer`)
  - `llmass write` - Email responder with AI (replaces `email-responder`)
  - `llmass test` - Run test suite
- **PyPI Package**: Published as `llmass` on PyPI
- **Backwards Compatibility**: Old commands (`email-organizer`, `email-responder`) still work
- **Draft Folder Configuration**: `DRAFTS_FOLDER` env var with auto-detection fallback
- **Email Signature Customization**: `SENDER_NAME`, `SENDER_TITLE`, `SENDER_COMPANY` env vars for personalized signatures
- **OOM Protection**: Auto-clamp `max_new_tokens` to 1024 on GPU, CPU fallback on OOM
- **Folder Tree View**: Hierarchical display with connectors, skips "." entries
- **Unsafe Category Migration**: Auto-rename folders with special characters (e.g., `Category_[alert]` → `Category_alert`)
- **Makefile Commands**: Added `llmass-generate`, `llmass-clean`, `llmass-write`, `llmass-test`, `publish`, `test-install`
- **Auto-versioning**: `publish.sh` now automatically increments patch version and runs tests before build
- **Dependencies**: Added `faker` and `lorem` for test email generation

### Fixed
- **HF Warnings**: Use `dtype` instead of deprecated `torch_dtype`, pass `attention_mask` to generation
- **PAD Token**: Set tokenizer pad_token and sync with model config
- **IMAP LIST Parser**: Robust parsing for folder names with special characters

### Changed
- Version bumped to 1.1.0
- Updated README with new CLI commands and PyPI installation instructions
- Enhanced setup.py with metadata, long_description, and project URLs

## [1.0.0] - 2025-10-04

- Email Organizer
  - Added dry-run mode across folder creation, subscription, moving, and expunge guards.
  - Cached IMAP hierarchy delimiter for fewer LIST calls.
  - Centralized TF-IDF configuration via `_make_vectorizer()` with ENV: `TFIDF_MAX_FEATURES`, `STOPWORDS`.
  - Implemented category reuse: match clusters to existing `INBOX.Category_*` using content similarity and sender overlap.
  - Implemented cross-folder spam similarity (INBOX vs SPAM/TRASH) with `CROSS_SPAM_SIMILARITY` and `CROSS_SPAM_SAMPLE_LIMIT`.
  - Cleanup of empty `Category*` folders on startup (`CLEANUP_EMPTY_CATEGORY_FOLDERS`).
  - Added content sufficiency gating (`_has_sufficient_text`) controlled by `CONTENT_MIN_CHARS`, `CONTENT_MIN_TOKENS`. Low-text messages are skipped from categorization and cross-spam.
  - Added logging via `LOG_LEVEL`, kept print for user-facing messages (transition in progress).
  - New CLI flags: `--folder`, `--include-subfolders` (partial: traversal pending).
  - Spam folder resolver now respects dry-run using `create_folder()`.

- Tests
  - Added tests for: LIST parsing, category name sanitization, category matching, cross-spam similarity, cleanup of empty categories, sender-only spam heuristics, vectorizer ENV config, dry-run behavior.
  - Planned: tests for content sufficiency helper (added now), include-subfolders traversal (pending), structured logging (pending).

- Docs
  - README: updated features, ENV sections, and added "System Architecture (Organizer)" and processing flow.
  - `.env.example`: added `TFIDF_MAX_FEATURES`, `STOPWORDS`, `CONTENT_MIN_CHARS`, `CONTENT_MIN_TOKENS`, category matching params, cross-spam, cleanup, logging, dry-run.
  - `docker_compose.yml`: passes all relevant ENV to `email-organizer`.

- Docker
  - Fixed Dovecot image build: switched to `apk add --no-cache netcat-openbsd` for Alpine-based image.

## 2025-10-03
- Initial organizer/responder scaffolding, Docker Compose setup, generator and basic tests.
