# Major TODOs to improve organization, testing, safety & security,reliability, error handling, QA, Monitoring, Evals

# Organization
- [x] Move tools to tools folder (meta_tools.py, agent_builder.py, sandbox_tools.py, checklist.py)
- [x] Move nodes to nodes folder (call_model.py, execute_tools.py, route_entry.py already have planner.py)
- [x] Make granular prompts to track changes
- [x] Sandbox as a class
- [x] Remove graph.py
- [ ] Better type safety - use dataclasses/TypedDict for complex return tuples
- [x] Centralized configuration file for all hardcoded values (timeouts, thresholds, limits)

# Safety & Security
- [ ] Sandbox with limited capability (consider Docker, WebAssembly, or restricted Python env instead of eval/exec) - later
- [ ] restrict builtins to only safe ones
- [ ] Prevent access to folders outside whitelisted ones
- [ ] Setup jailbreak guardrails
- [ ] Toxic NSFW/Harmful guardrails
- [ ] Add resource limits beyond timeout (eg, recursion limit)

# Error Handling
- [ ] Model Fallback
- [ ] Route entry to correct incorrect state ( remove broken tool calls, etc )
- [ ] Add default recursion limit - prevent extremely long runs, stuck in loop, throw proper error on hitting limit
- [ ] Log exceptions with stack traces, tool name, and arguments (currently just str(e))
- [ ] Add per-tool timeout with asyncio.wait_for() - tools can hang indefinitely
- [ ] Add timeout handling for search_functions, load_functions, and other meta tools
- [ ] Better fallback model logic - reuse fallback instance, check if fallback also fails

# Performance
- [ ] **CRITICAL: Fix sandbox recreation on every execute_python_code call** - context is lost
- [ ] Cache tool definitions - don't reload default tools on every code execution
- [ ] Create meta_tools once and reuse - currently recreated multiple times in execute_tools
- [ ] Cache model instances - load_chat_model called repeatedly in call_model and execute_tools
- [ ] Create sandbox once in route_entry, store in state or runtime context
- [ ] Optimize prompt structure for token usage - tool definitions repeated in every message
- [ ] Implement state compression and message summarization to prevent unbounded growth
- [ ] Remove all sync code
- [ ] Lazy initialization of registry, tools and sandbox


# State Management
- [X] **Simplify state queue _enqueue** - returns list(set(queue)) directly
- [ ] Implement state compression - messages array grows unbounded - possibly with langchain middlewares
- [ ] Clear separation of tool context, user context, and system context
- [ ] Reduce unused state vars, (eg agent name, description, etc.)

# QA
- [ ] Create smaller prompts to test after every release, eg, load tools called correctly
- [ ] List common edge cases
- [ ] Add plan validation - check plan matches conversation history and all required tools are available

# CI & CD
- [ ] Run tests on every pull requests to develop
- [ ] Run evals on every pull request to master/main

# Testing
- [ ] Write tests for individual tools (search_functions, load_functions, web_search, etc.)
- [ ] Write test for individual nodes (call_model, execute_tools, route_entry)
- [ ] Write sandbox tests (context persistence, timeout, error handling)
- [ ] Write basic sanity prompts for testing, eg, nth fibonacci number
- [ ] Write test for jailbreaks, toxicity, content moderation
- [ ] Test tool loading logic and validation
- [ ] Integration tests for full agent workflows
- [ ] Test sandbox resource limits and security constraints

# Evals
- [ ] Evals for agent sanity
- [ ] Evals for search functions - vector db search
- [ ] Evals for one off tasks
- [ ] Evals for agent builder tasks
- [ ] Evals for repeat agent runs

# Monitoring
- [x] Add logging with correct level at all required places for debugging
- [ ] Script to track metrics like time to first token, tool calls, recursion depth, time to completion from langsmith traces
- [ ] Add OpenTelemetry instrumentation (optional):
  - [ ] Tool execution times
  - [ ] Model invocation latency
  - [ ] Sandbox execution duration
- [ ] Track metrics for tool execution success/failure rates
- [ ] Alert on excessive model retries or fallback usage
- [ ] Monitor sandbox timeout frequency

# Agent Builder
- [ ] Use LangGraph's proper interrupt mechanism instead of boolean flags (ask_user)
- [ ] Add plan validation step before code generation

# Developer Experience
- [ ] Better type safety - replace tuple returns with dataclasses (e.g., load_functions returns 4-tuple)
- [x] Centralized configuration management (currently scattered: sandbox_timeout=20 vs 300, TOOL_THRESHOLD=0.75, etc.)
- [ ] Improved error messages with error codes and suggested fixes
- [ ] Better documentation for each node and tool
