# Executors Documentation

Executors orchestrate the **end-to-end research workflow** once a query or plan has been validated. They sit between:

- **Search engines** (Google, Bing, VertexAI, etc.)
- **Crawlers** (Basic, Tavily, Crawl4AI, …)
- **Content processing** (summaries, truncation, relevancy sorters)
- **LLM interfaces** (query refinement, structured outputs)
- **Progress reporters / logging**

All executors extend the shared base class defined in `base_executor.py`.

```python
class BaseWebSearchExecutor(ABC):
    async def run(self) -> tuple[list[ContentChunk], list[WebSearchLogEntry]]:
        """Returns ranked ContentChunk objects + user-visible log entries."""
```

---

## Architecture Overview

1. **Refinement / Planning** – Optional, performed by V1 (query refinement) or V2 (step plan) executors.
2. **Search Execution** – Calls the configured search engine(s).
3. **Crawling** – Fetches full page content when required.
4. **Content Processing** – Summaries/truncation into `WebPageChunk`.
5. **Chunk Selection** – Reduces to token budget, applies relevancy sorting.
6. **Output** – Returns `ContentChunk` list + log entries (for LLM conversation history or audit).

All steps push diagnostics into `WebSearchDebugInfo` for telemetry and debugging.

---

## Executor Decision Matrix

| Executor | Input Type | Best For | Query Strategy | Parallelism | Notes |
|----------|-----------|----------|----------------|-------------|-------|
| **WebSearchV1Executor** | `WebSearchToolParameters` (single query + date restrictions) | Traditional search flows | LLM-based query refinement (Basic/Advanced) | Sequential (queries iterated) | Ideal for iterative refinement and single-shot queries |
| **WebSearchV2Executor** | `WebSearchPlan` (multi-step plan) | Agentic / multi-step research | Executes explicit steps (search/read) | Parallel (steps scheduled concurrently) | Best for step-based plans generated by planning agents |

---

## Common Dependencies

Every executor receives:

- `search_service`: Any `SearchEngineTypes` instance from `services/search_engine`.
- `crawler_service`: Any `CrawlerTypes` instance from `services/crawlers`.
- `language_model_service` + `language_model` (LMI): For LLM calls (refinement/structuring).
- `content_processor`: Handles summarization/truncation into `WebPageChunk`.
- `chunk_relevancy_sorter` + config: Optional re-ranking using semantic similarity.
- `content_reducer`: Callback limiting chunks to token budget.
- `tool_progress_reporter`: Optional; surfaces progress back to LLM orchestration.
- `debug_info`: Aggregates timings, configs, and intermediate artifacts.

---

## WebSearchV1Executor

**File:** `web_search_v1_executor.py`  
**Input:** `WebSearchToolParameters`  
**Pipeline:** `Refine → Search → Crawl (optional) → Process → Rerank`

### Key Features

- **Query Refinement Agent** – Uses `RefineQueryMode` (DEACTIVATED / BASIC / ADVANCED) to turn raw user queries into more precise search statements.
- **Date Restriction Support** – Passes `date_restrict` down to search services capable of filtering by recency.
- **Sequential Multi-Query Search** – Executes refined queries one after another (max capped by config).
- **Automatic Crawling** – If the search engine returns snippet-only results.
- **Progress Notifications** – Updates `notify_name` / `notify_message` for UI or logging.
- **Structured Logs** – Produces `WebSearchLogEntry` for each search/crawl step.

### Configuration Options

| Parameter | Description |
|-----------|-------------|
| `mode` | `RefineQueryMode`: BASIC (default), ADVANCED (multi-query), DEACTIVATED |
| `max_queries` | Maximum number of refined queries to execute |
| `refine_query_system_prompt` | System prompt used by the query refinement agent |
| `chunk_relevancy_sort_config` | Enables semantic re-ranking of content chunks |
| `content_reducer` | Callable controlling token budget reduction |

### Usage Example

```python
executor = WebSearchV1Executor(
    company_id="acme",
    language_model_service=lm_service,
    language_model=lmi,
    search_service=search_engine,
    crawler_service=crawler,
    content_processor=content_processor,
    chunk_relevancy_sorter=chunk_sorter,
    chunk_relevancy_sort_config=sort_config,
    content_reducer=content_reducer,
    tool_call=tool_call,
    tool_parameters=WebSearchToolParameters(query="LLM agents", date_restrict="m1"),
    refine_query_system_prompt="Refine queries for web search.",
    debug_info=WebSearchDebugInfo(parameters={}),
    tool_progress_reporter=progress_reporter,
    mode=RefineQueryMode.ADVANCED,
    max_queries=5,
)

relevant_chunks, logs = await executor.run()
```

---

## WebSearchV2Executor

**File:** `web_search_v2_executor.py`  
**Input:** `WebSearchPlan` (structured plan)  
**Pipeline:** `Execute Steps (Search/Read) → Merge Results → Crawl (if needed) → Process → Rerank`

### Key Features

- **Step-Based Execution** – Supports `StepType.SEARCH` and `StepType.READ_URL`.
- **Parallel Scheduling** – Steps run via `asyncio.create_task`, allowing concurrent execution.
- **Max Step Enforcement** – `max_steps` prevents runaway plans; trims plan to safe size.
- **Per-Step Logging** – Each step tracked in `StepDebugInfo` with config and duration.
- **Flexible Crawler Usage** – `READ_URL` steps go straight to the crawler without a search engine.
- **Reuse of Base Components** – Shares the same content processing / reduction pipeline as V1.

### Configuration Options

| Parameter | Description |
|-----------|-------------|
| `max_steps` | Maximum plan steps to execute (default 3) |
| `tool_parameters` | Full `WebSearchPlan` with `steps`, `objective`, `expected_outcome` |
| `chunk_relevancy_sort_config` | Enables optional re-ranking |
| `tool_progress_reporter` | Emits progress updates for each step |

### Usage Example

```python
plan = WebSearchPlan(
    objective="Summarize latest breakthroughs in fusion energy",
    expected_outcome="3 bullet summary with citations",
    steps=[
        Step(step_type=StepType.SEARCH, query_or_url="fusion energy breakthroughs 2025"),
        Step(step_type=StepType.READ_URL, query_or_url="https://energy.gov/fusion-update"),
    ],
)

executor = WebSearchV2Executor(
    search_service=search_engine,
    language_model_service=lm_service,
    language_model=lmi,
    crawler_service=crawler,
    tool_call=tool_call,
    tool_parameters=plan,
    company_id="acme",
    content_processor=content_processor,
    chunk_relevancy_sorter=chunk_sorter,
    chunk_relevancy_sort_config=sort_config,
    content_reducer=content_reducer,
    debug_info=WebSearchDebugInfo(parameters={}),
    tool_progress_reporter=progress_reporter,
    max_steps=4,
)

relevant_chunks, logs = await executor.run()
```

---

## Choosing an Executor

### Use WebSearchV1Executor if:
- You have a single query (with optional date filters).
- You want the system to refine/expand the query automatically.
- You prioritize sequential control and consistent logging.
- You rely on search engines requiring HTTP scraping afterward.

### Use WebSearchV2Executor if:
- You already have a structured plan (from a planning LLM).
- You want to mix search steps with direct URL reads.
- You need concurrency to speed up research.
- You require explicit per-step telemetry and enforcement of step limits.

It’s common to use **V1 for “on-the-fly” queries** and **V2 for agent-generated research plans**.

---

## Adding a New Executor

1. **Create Config / Executor Class**:
   ```python
   class MyExecutor(BaseWebSearchExecutor):
       async def run(self):
           # orchestrate search + crawl + process
           return content_chunks, logs
   ```
2. **Wire it in**: Register in factory (if applicable) and add tests.
3. **Reuse Base Methods**: `_content_processing`, `_select_relevant_sources`, progress notifications, etc.
4. **Document**: Update this README and add integration tests similar to the existing executors.

---

## Best Practices

- **Respect timeouts**: Pass `config.timeout` through to crawlers, search engines, and API calls.
- **Use progress reporting**: Populate `notify_name` / `notify_message` to improve UX for long operations.
- **Log everything**: `debug_info.steps` is invaluable for troubleshooting.
- **Gracefully handle search/crawl failures**: Append error messages rather than raising to keep output stable.
- **Cap complexity**: Use `max_queries` (V1) and `max_steps` (V2) to prevent runaway plans.

---

## Related Modules

- `services/executors/configs/` – Houses V1/V2 mode configs & prompts.
- `services/content_processing/` – Summaries, truncation, chunk building.
- `services/crawlers/` – All crawler implementations (see corresponding README).
- `services/search_engine/` – All search engine integrations (see README).

---

Happy executing! ⚡


