# Product Requirements Document: Selfspy

**Document Version**: 1.0
**Last Updated**: 2025-10-22
**Status**: Implemented & Production-Ready
**Product Type**: Privacy-focused activity monitoring and analytics tool

---

## 1. Context & Why Now

- **Personal productivity analytics is a growing category** but most solutions require cloud services and sacrifice user privacy by sending sensitive activity data to external servers
- **Developer workflow optimization is critical** as knowledge workers spend significant time context-switching between applications, terminals, and projects without clear visibility into their patterns
- **Privacy regulations (GDPR, CCPA) make cloud-based tracking risky** for organizations and individuals who want to monitor activity without exposing sensitive data to third parties
- **Modern async Python patterns enable efficient local monitoring** with SQLAlchemy 2.0 and async/await making it feasible to build high-performance local-first applications
- **macOS provides robust accessibility APIs** that enable comprehensive activity tracking when users grant explicit permissions, creating an opportunity for native platform integration

---

## 2. Users & Jobs to Be Done (JTBD)

### Primary User Segments

**Individual Knowledge Workers**
- As a software developer, I need to understand where my time goes during coding sessions so that I can identify productivity patterns and reduce context-switching
- As a remote worker, I need to track my activity privately without employer surveillance so that I can self-optimize my workflow while maintaining autonomy
- As a productivity enthusiast, I need detailed analytics about my computer usage so that I can make data-driven decisions about tool usage and work habits

**Privacy-Conscious Professionals**
- As a security professional, I need activity monitoring that stores all data locally with encryption so that I can analyze work patterns without data exposure risk
- As a healthcare/legal professional, I need compliance-friendly monitoring so that I can track productivity while meeting HIPAA/attorney-client privilege requirements
- As an open-source contributor, I need transparent activity tracking so that I can verify what data is collected and how it's stored

**Technical Researchers & Analysts**
- As a UX researcher, I need granular application usage data so that I can study real-world interaction patterns for software design
- As a developer tools maker, I need terminal command analytics so that I can understand how developers actually use CLI tools in their workflow
- As a productivity researcher, I need long-term activity datasets so that I can identify seasonal patterns and behavioral trends

### Secondary User Segments

**Development Teams (Future)**
- As a team lead, I need aggregated anonymous activity patterns so that I can optimize team workflows without individual surveillance
- As a DevOps engineer, I need to understand terminal command patterns so that I can create better automation and improve developer experience

---

## 3. Business Goals & Success Metrics

### Business Goals

1. **Establish Selfspy as the privacy-first standard for personal activity monitoring** by providing comprehensive tracking without cloud dependencies or privacy compromises
2. **Enable deep productivity insights for technical users** through rich terminal analytics, development workflow tracking, and git integration
3. **Build a sustainable open-source project** with clear architecture, comprehensive documentation, and extensibility for community contributions
4. **Demonstrate technical excellence** through modern Python patterns (async/await, SQLAlchemy 2.0, Pydantic settings) that serve as reference implementation
5. **Support multiple platforms** with macOS as primary target and cross-platform fallback for broader adoption

### Leading Indicators (Early Signals)

- **Installation success rate**: Percentage of users who successfully complete setup with permissions granted
- **Daily active monitoring sessions**: Number of days per week users run the monitoring daemon
- **Terminal analytics adoption**: Percentage of users who enable and use terminal command tracking
- **Widget engagement**: Number of users who install and actively use desktop widgets
- **Configuration customization rate**: Percentage of users who modify default settings

### Lagging Indicators (Ultimate Success)

- **Monthly active users**: Target 1,000+ active installations within first year
- **Data retention**: Average user maintains monitoring for 90+ consecutive days
- **Community contributions**: 10+ external contributors submitting features or fixes
- **Documentation quality**: Support questions answered by existing docs (80%+ self-service rate)
- **Platform coverage**: Successful deployments on macOS (primary), Linux, Windows (fallback)

**Targets & Timeframes**

- **Q1**: Core monitoring stable with 100+ weekly active users
- **Q2**: Terminal analytics and widgets adopted by 50%+ of user base
- **Q3**: First community-contributed integrations or visualization tools
- **Q4**: Cross-platform support with Linux and Windows installations

---

## 4. Functional Requirements

### FR1: Core Activity Monitoring
**Requirement**: System shall continuously monitor keyboard, mouse, and window activity with configurable intervals

**Acceptance Criteria**:
- Monitor captures keystroke events with millisecond precision
- Mouse clicks (left, middle, right) and scroll events are tracked with coordinates
- Active window changes detected within 100ms of switch
- Process name, window title, and bundle ID captured for each window
- Monitoring runs continuously without user intervention after start
- Graceful shutdown on SIGTERM/SIGINT with data flush

### FR2: Encrypted Keystroke Storage
**Requirement**: All keystroke data shall be encrypted using AES-256 with user-provided password stored in system keychain

**Acceptance Criteria**:
- User prompted for password on first run
- Password stored securely in macOS Keychain (or platform equivalent)
- Keystroke text encrypted before database write using Fernet (PBKDF2 + AES-256)
- Password verification using magic string prevents incorrect password usage
- Encryption can be disabled via configuration for non-sensitive usage
- No plaintext keystroke data written to disk at any time

### FR3: Async Database Persistence
**Requirement**: Activity data shall be persisted to SQLite database using async SQLAlchemy 2.0 with proper buffering

**Acceptance Criteria**:
- Database operations are non-blocking using aiosqlite
- Keystroke buffer flushes after 1 second of inactivity or 50 keystrokes
- Click and scroll events stored with button type and coordinates
- Window changes persisted with geometry, screen info, and timestamps
- Database schema includes proper indexes for query performance
- Foreign key relationships maintained for data integrity
- Database size managed with configurable cleanup thresholds

### FR4: Platform-Specific Integration
**Requirement**: System shall use native macOS APIs for optimal tracking with cross-platform fallback

**Acceptance Criteria**:
- macOS: Uses PyObjC with Quartz and ApplicationServices frameworks
- macOS: Checks for Accessibility permissions and prompts if missing
- macOS: Opens System Settings directly to permission page
- Fallback: Uses pynput for keyboard/mouse on Linux/Windows
- Platform detection automatic based on `platform.system()`
- Bundle ID capture on macOS for application identification
- Cross-platform operation without macOS-specific features

### FR5: Rich CLI Visualizations
**Requirement**: System shall provide multiple CLI commands for viewing statistics with rich terminal output

**Acceptance Criteria**:
- `selfviz enhanced`: Shows productivity dashboard with charts and heatmaps
- `selfviz timeline`: Displays chronological activity timeline
- `selfviz live`: Real-time updating dashboard (5-second refresh)
- Bar charts rendered using Rich library with colored blocks
- Time-based heatmap shows activity by 6-hour blocks (Night/Morning/Day/Evening)
- Productivity score calculated from keystrokes, windows, process count
- Top applications ranked by keystroke count and click count
- All visualizations support date range filtering (--days parameter)

### FR6: Terminal Command Tracking
**Requirement**: System shall track terminal commands from shell history with project context

**Acceptance Criteria**:
- Monitors bash, zsh, and fish history files
- Commands parsed from different shell history formats
- Working directory and git branch captured for each command
- Project type detected (Python, Node, Rust, Go, Java, C/C++)
- Commands categorized (git, npm, python, build, editor, file, system, network)
- Dangerous commands flagged (rm, sudo, chmod, etc.)
- Common commands filtered (ls, cd, pwd, clear)
- Duplicate commands within same session deduplicated by hash
- Background polling every 2 seconds for new commands

### FR7: Desktop Widgets
**Requirement**: System shall provide native macOS desktop widgets for always-visible activity display

**Acceptance Criteria**:
- Python/PyObjC implementation for rapid development
- Always-on-top windows that don't interfere with workflow
- Drag-and-drop repositioning anywhere on screen
- Auto-refresh data every 5-10 seconds
- Display real-time keystroke, click, and application statistics
- Terminal activity integration showing recent commands
- Productivity metrics calculated and displayed
- Multiple widget types (simple, advanced, terminal-focused)
- Configuration management for widget preferences

### FR8: Privacy Controls
**Requirement**: System shall provide comprehensive privacy controls via configuration

**Acceptance Criteria**:
- Configurable excluded applications (default: 1Password, Keychain Access, System Settings)
- Process bundle exclusion list for macOS
- Privacy mode enables stricter filtering
- Optional screenshot capture (disabled by default)
- Screenshot exclusion list separate from activity tracking
- Database size limits with auto-cleanup
- Backup management with rotation
- No data transmission outside local machine
- All settings via Pydantic Settings with environment variables

### FR9: Configuration Management
**Requirement**: System shall support flexible configuration via environment variables, .env files, and code defaults

**Acceptance Criteria**:
- Settings class using Pydantic Settings with validation
- Environment variables prefixed with `SELFSPY_`
- .env file support in data directory or project root
- Database path, encryption, intervals, exclusions all configurable
- Field validators prevent invalid configurations
- Type-safe settings with Python type hints
- Computed properties for derived values (database_path)
- Settings documentation in code and external docs

### FR10: Statistics and Analytics
**Requirement**: System shall provide comprehensive analytics across time periods with multiple output formats

**Acceptance Criteria**:
- Daily, weekly, monthly aggregations
- Per-application keystroke and click counts
- Window visit frequency and duration
- Hourly activity heatmaps for pattern detection
- Productivity scoring algorithm with level classification
- Daily averages calculated across time period
- Timeline view showing chronological activity
- Support for text, JSON, CSV output formats
- Date range filtering with ISO 8601 date format

---

## 5. Non-Functional Requirements

### Performance
- **Response Time**: Window change detection within 100ms of actual switch
- **Throughput**: Support 1000+ keystrokes per minute without data loss
- **CPU Usage**: Less than 2% CPU usage during normal monitoring on modern hardware
- **Memory**: Under 100MB RAM footprint for monitoring daemon
- **Database Write Latency**: Async writes complete within 50ms

### Scale
- **Data Volume**: Support years of continuous monitoring (100GB+ database)
- **Concurrent Sessions**: Handle multiple terminal sessions simultaneously
- **Window Tracking**: Track 100+ application switches per hour
- **History Processing**: Parse shell history files with 10,000+ commands
- **Widget Updates**: Refresh widgets without UI lag or database lock contention

### SLOs/SLAs
- **Uptime**: Monitoring daemon runs 99.5% of time without crashes
- **Data Integrity**: Zero data loss during graceful shutdown
- **Query Performance**: Statistics generation completes in under 3 seconds
- **Startup Time**: Monitoring starts within 2 seconds of command
- **Permission Check**: Accessibility permission verification in under 1 second

### Privacy
- **Data Locality**: All data stored locally in `~/.selfspy/` by default
- **Encryption At Rest**: AES-256 encryption for all keystroke text
- **Password Security**: Passwords stored in system keychain, never in plaintext
- **No Telemetry**: Zero data transmission to external servers
- **Audit Trail**: All database operations logged in structured logs
- **Compliance Ready**: Architecture supports GDPR, HIPAA, SOC2 data requirements

### Security
- **Authentication**: Password required for encrypted keystroke access
- **Authorization**: Only process owner can access database files (filesystem permissions)
- **Encryption**: PBKDF2 key derivation with 100,000 iterations
- **Key Management**: Fernet symmetric encryption with URL-safe base64
- **Dangerous Command Detection**: Flags commands like `rm -rf`, `sudo`, `chmod`
- **Code Security**: No eval, no arbitrary code execution, no network sockets

### Observability
- **Structured Logging**: All logs use structlog with context
- **Debug Mode**: Configurable debug logging for troubleshooting
- **Error Handling**: All exceptions caught and logged with context
- **Live Monitoring**: Real-time display shows current monitoring state
- **Database Inspection**: Direct SQLite access for advanced users
- **Metrics**: Built-in productivity scoring and activity metrics

---

## 6. Scope Definition

### In Scope

- **Activity Monitoring**: Keystrokes, mouse clicks, scrolls, window changes
- **Encryption**: AES-256 for keystroke data with password protection
- **Platform Support**: macOS (full), Linux/Windows (fallback)
- **Terminal Analytics**: Command tracking for bash/zsh/fish
- **Visualizations**: Rich CLI dashboards, charts, timelines, heatmaps
- **Desktop Widgets**: Python/PyObjC implementation for macOS
- **Configuration**: Environment variables, .env files, Pydantic settings
- **Privacy Controls**: Application exclusions, privacy mode, optional features
- **Documentation**: Installation, usage, configuration, architecture, troubleshooting
- **Database**: SQLite with async SQLAlchemy 2.0, proper indexes
- **CLI Commands**: `selfspy start`, `selfviz`, `selfstats`, `selfterminal`

### Out of Scope

- **Cloud Sync**: No cloud storage or backup features
- **Multi-User**: Single-user per installation only
- **Web Interface**: No web-based dashboard (CLI and widgets only)
- **Mobile Apps**: No iOS/Android clients
- **Team Features**: No multi-user analytics or team dashboards
- **Real-time Collaboration**: No shared monitoring or data sync
- **Screenshot OCR**: No text extraction from screenshots
- **Video Recording**: No screen recording capabilities
- **Network Activity**: No network traffic monitoring
- **File System Monitoring**: No file change tracking
- **API Endpoints**: No REST API or webhook integrations (FastAPI imported but not exposed)

### Future Considerations

- **Web Dashboard**: Browser-based visualization for richer charts (Plotly already imported)
- **Plugin System**: Extension points for custom trackers or visualizations
- **Export Formats**: Additional export options (Excel, PDF reports)
- **AI Insights**: Machine learning for pattern detection and recommendations
- **Team Analytics**: Anonymous aggregation for team productivity insights
- **Cross-Device**: Sync activity across multiple machines (privacy-preserving)
- **Integration APIs**: Hooks for calendar, task managers, time tracking tools
- **Mobile Companion**: View stats on phone without full tracking
- **Voice Commands**: Optional voice activity tracking
- **Advanced Filtering**: SQL-like queries or GUI filter builder

---

## 7. Rollout Plan

### Phase 1: Core Monitoring (Implemented)
**Status**: Complete and stable

**Milestones**:
- ActivityMonitor orchestration with async/await patterns
- Platform-specific trackers (macOS with PyObjC, cross-platform fallback)
- SQLite database with SQLAlchemy 2.0 async
- Encryption with Fernet and keychain integration
- Basic CLI commands (`selfspy start`)

**Success Gates**:
- Monitoring runs continuously without crashes for 24+ hours
- Database grows predictably with activity
- CPU/memory usage within acceptable thresholds
- Permissions check works on fresh macOS install

### Phase 2: Analytics & Visualization (Implemented)
**Status**: Complete and stable

**Milestones**:
- Rich CLI visualizations (`selfviz enhanced`, timeline, live)
- Statistical aggregations with proper SQL queries
- Productivity scoring algorithm
- Time-based heatmaps and bar charts
- Terminal command tracking integration

**Success Gates**:
- Stats generation completes in under 3 seconds for 7-day periods
- Visualizations render correctly in various terminal emulators
- No cartesian product issues in SQL queries
- Terminal tracking captures 95%+ of executed commands

### Phase 3: Desktop Widgets (Implemented)
**Status**: Complete with Python/PyObjC implementation

**Milestones**:
- Simple widget for development and testing
- Advanced widget system with multiple types
- Widget configuration management
- Data integration with real-time updates
- Drag-and-drop positioning

**Success Gates**:
- Widgets update without blocking main monitoring
- No database lock contention between widgets and daemon
- Widgets maintain position across restarts
- Memory usage acceptable with multiple widgets

### Phase 4: Documentation & Polish (Implemented)
**Status**: Comprehensive documentation in place

**Milestones**:
- Installation guide with dependency management (uv)
- Usage guide with command examples
- Configuration reference with all settings
- Architecture documentation with design decisions
- Troubleshooting guide for common issues

**Success Gates**:
- New users can install without asking questions (80%+ success)
- All configuration options documented with examples
- Common errors have documented solutions
- Architecture doc enables contributor onboarding

### Guardrails

- **Data Loss Prevention**: Before any schema changes, verify backup and restore procedures
- **Performance Regression**: Automated performance tests prevent CPU/memory increases
- **Privacy Validation**: Manual review of all data collection code for privacy compliance
- **Backward Compatibility**: Database migrations must preserve existing data
- **Permission Handling**: Never proceed with monitoring without explicit user permission grant

### Kill-Switch Criteria

- **Data Corruption**: More than 1% of database writes fail integrity checks
- **Privacy Breach**: Any evidence of data transmission outside local machine
- **Performance Collapse**: CPU usage exceeds 10% or memory exceeds 500MB
- **Permission Violations**: Attempt to bypass permission checks or accessibility requirements
- **Security Vulnerability**: Discovery of unencrypted keystroke storage or password exposure

---

## 8. Risks & Open Questions

### Technical Risks

**Risk**: macOS permission model changes break tracking in future OS versions
**Mitigation**: Modular platform abstraction allows quick updates; fallback tracker as safety net

**Risk**: SQLite database lock contention with multiple readers (widgets + CLI)
**Mitigation**: Async database access with connection pooling; widget queries use read-only connections

**Risk**: Shell history parsing breaks with custom shell configurations
**Mitigation**: Regex patterns handle multiple formats; graceful degradation for unparseable lines

**Risk**: Encryption key loss results in unrecoverable keystroke data
**Mitigation**: Document backup procedures; consider optional key recovery mechanism in future

**Risk**: Large database files (100GB+) cause performance degradation
**Mitigation**: Configurable auto-cleanup thresholds; database size monitoring; future archival feature

### Business/Market Risks

**Risk**: Privacy concerns prevent adoption despite local-first architecture
**Mitigation**: Transparent open-source code; comprehensive privacy documentation; optional features

**Risk**: Similar tools with better UX or features fragment user base
**Mitigation**: Focus on technical excellence and privacy; build community around extensibility

**Risk**: Lack of cross-platform support limits adoption to macOS users only
**Mitigation**: Fallback tracker provides basic functionality; Linux as next platform priority

**Risk**: Open-source sustainability challenges without funding model
**Mitigation**: Keep core simple and maintainable; accept community contributions; explore sponsorship

### Open Questions

**Q1**: Should we expose FastAPI endpoints for remote widget access?
**Context**: FastAPI/Uvicorn dependencies already included but not used
**Decision Needed**: Evaluate privacy implications vs. convenience for remote dashboards

**Q2**: What is the optimal database cleanup strategy for long-term users?
**Context**: Current auto-cleanup based on size threshold, but no time-based archival
**Decision Needed**: Define retention policies and archival format for historical data

**Q3**: Should terminal tracking support custom shell configurations?
**Context**: Currently handles standard bash/zsh/fish, but many users customize heavily
**Decision Needed**: Define plugin system or configuration for custom parsing

**Q4**: How should we handle screenshot features given privacy concerns?
**Context**: Infrastructure exists but disabled by default; unclear if valuable enough
**Decision Needed**: Either enhance with OCR/analysis or remove to reduce attack surface

**Q5**: What is the community contribution model for new trackers or widgets?
**Context**: No formal plugin system or contribution guidelines beyond standard OSS
**Decision Needed**: Define extension points, API stability guarantees, and review process

### Dependencies

**Internal Dependencies**:
- Pydantic Settings for configuration management
- SQLAlchemy async session management
- Rich library for terminal visualizations
- Structlog for logging infrastructure

**External Dependencies**:
- macOS Accessibility permissions (user-granted)
- System keychain for password storage
- Shell history file access (bash/zsh/fish)
- PyObjC frameworks for native macOS features

**Platform Dependencies**:
- Python 3.10+ runtime
- SQLite 3.8+ (included with Python)
- macOS 10.12+ for full features
- Terminal emulator with color support for visualizations

---

## Technical Architecture Overview

### System Components

```
ActivityMonitor (Orchestrator)
├── Platform Trackers
│   ├── MacOSInputTracker (PyObjC-based)
│   ├── MacOSWindowTracker (Quartz API)
│   └── FallbackTracker (pynput-based)
├── TerminalTracker (Shell history monitoring)
└── ActivityStore (Async SQLAlchemy)
    └── SQLite Database (encrypted keystrokes)

CLI Layer
├── selfspy (main daemon control)
├── selfviz (rich visualizations)
├── selfstats (basic statistics)
└── selfterminal (terminal analytics)

Desktop Widgets (macOS)
├── PyObjC Implementation
├── Data Integration Layer
└── Widget Configuration
```

### Data Flow

1. **Input Events**: Platform tracker captures keyboard/mouse/window events
2. **Buffering**: Events buffered in memory until flush threshold
3. **Encryption**: Keystroke text encrypted with Fernet before storage
4. **Async Write**: ActivityStore persists to SQLite asynchronously
5. **Querying**: CLI tools and widgets query database for statistics
6. **Visualization**: Rich library renders charts and dashboards

### Technology Stack

- **Language**: Python 3.10+ (async/await, type hints)
- **Database**: SQLite with aiosqlite and SQLAlchemy 2.0
- **Encryption**: cryptography library (Fernet with PBKDF2)
- **Configuration**: Pydantic Settings with environment variables
- **CLI**: Typer with Rich for terminal UI
- **Platform**: PyObjC for macOS, pynput for cross-platform
- **Logging**: structlog with structured logging
- **Dependency Management**: uv for fast, reliable installs

---

## Appendix: Key Design Decisions

### Why SQLite?
- Embedded database requires no server setup
- Excellent performance for local analytics workloads
- Battle-tested with billions of deployments
- Built-in Python support via aiosqlite
- File-based storage aligns with privacy goals

### Why Async SQLAlchemy?
- Non-blocking database operations prevent UI lag
- Future-proof for web dashboard or API endpoints
- Modern Python best practices (async/await)
- Better concurrency model for background tracking
- Enables efficient buffering and batching

### Why Pydantic Settings?
- Type-safe configuration with validation
- Environment variable support for deployment
- .env file integration for development
- Auto-generated documentation from types
- Computed properties for derived settings

### Why Rich Library?
- Beautiful terminal output without custom rendering
- Cross-platform terminal compatibility
- Built-in support for tables, charts, panels
- Live updating displays for real-time monitoring
- Extensive documentation and examples

### Why Local-First Architecture?
- Privacy by design (no data leaves machine)
- No cloud costs or server maintenance
- Works offline without limitations
- User owns all data in accessible format
- Compliance-friendly for regulated industries

---

## Success Criteria Summary

Selfspy is successful when:

1. **Users trust it**: Privacy-conscious users recommend it to peers based on transparent architecture
2. **It provides value**: Users gain actionable insights into productivity patterns within first week
3. **It's maintainable**: New contributors can understand codebase and add features successfully
4. **It's reliable**: Users run monitoring continuously without concerns about crashes or data loss
5. **It's private**: No privacy incidents, no data leaks, no surveillance concerns

---

**Document Prepared By**: Product analysis from implemented codebase
**Implementation Status**: Production-ready, actively maintained
**Next Review**: Quarterly or upon significant feature additions
