# Tools {#sec-tools}

## Overview

Many models now have the ability to interact with client-side Python functions in order to expand their capabilities. This enables you to equip models with your own set of custom tools so they can perform a wider variety of tasks.

Inspect natively supports registering Python functions as tools and providing these tools to models that support them (currently OpenAI, Claude 3, Google Gemini, and Mistral). Inspect also includes one built-in tool (web search).

::: callout-note
### Tools and Agents

One application of tools is to run them within an agent scaffold that pursues an objective over multiple interactions with a model. The scaffold uses the model to help make decisions about which tools to use and when, and orchestrates calls to the model to use the tools. This is covered in more depth in the [Agents](#sec-agents) section.
:::

## Tool Basics

To demonstrate the use of tools, we'll define a simple tool that adds two numbers. We use the `@tool` decorator to register it with the system, and we provide documentation comments that iare used to provide details to the model about the tool and its arguments:

``` python
@tool(prompt="""
    If you are given a math problem of any kind,
    please use the add tool to compute the result."""
)
def add():
    async def execute(x: int, y: int):
        """
        Tool for adding two numbers.

        Args:
            x: First number to add.
            y: Second number to add.

        Returns:
            The sum of the two numbers.
        """
        return x + y

    return execute
```

### Annotations

Note that we provide type annotations for both arguments:

``` python
async def execute(x: int, y: int)
```

Further, we provide descriptions for each parameter in the documention comment:

```python
Args:
    x: First number to add.
    y: Second number to add.
```

Type annotations and descriptions are *required* for tool declarations so that the model can be informed which types to pass back to the tool function and what the purpose of each parameter is.

### Using Tools

We can use this tool in an evaluation by passing it to the `use_tools()` Solver:

``` python
@task
def addition_problem():
    return Task(
        dataset=[Sample(input="What is 1 + 1?", target=["2"])],
        plan=[
            use_tools(add()), 
            generate()
        ],
        scorer=match(numeric=True),
    )
```

Note that this tool doesn't make network requests or do heavy computation, so is fine to run as inline Python code. If your tool does do more elaborate things, you'll want to make sure it plays well with Inspect's concurrency scheme. For network requests, this amounts to using `async` HTTP calls with `httpx`. For heavier computation, tools should use subprocesses as described in the next section.

::: {.callout-note appearance="simple"}
Note that when using tools with models, the models do not call the Python function directly. Rather, the model generates a structured request which includes function parameters, and then Inspect calls the function and returns the result to the model.
:::

## Tool Errors

Various errors can occur during tool execution, especially when interacting with the file system or network or when using [Sandbox Environments](#sec-sandbox-environments) to execute code in a container sandbox. As a tool writer you need to decide how you'd like to handle error conditions. A number of approaches are possible:

1.  Notify the model that an error occurred to see whether it can recover.

2.  Catch and handle the error internally (trying another code path, etc.).

3.  Allow the error to propagate, resulting in the current `Sample` failing with an error state.

There are no universally correct approaches as tool usage and semantics can vary widely—some rough guidelines are provided below.

### Default Handling {#default-handling}

If you do not explicitly handle errors, then Inspect provides some default error handling behaviour. Specifically, if any of the following errors are raised they will be handled and reported to the model:

-   `TimeoutError` — Occurs when a call to `subprocess()` or `sandbox().exec()` times out.

-   `PermissionError` — Occurs when there are inadequate permissions to read or write a file.

-   `UnicodeDecodeError` — Occurs when the output from executing a process or reading a file is binary rather than text.

-   `ToolError` — Special error thrown by tools to indicate they'd like to report an error to the model.

These are all errors that are *expected* (in fact the `SandboxEnvironemnt` interface documents them as such) and possibly recoverable by the model (try a different command, read a different file, etc.). Unexpected errors (e.g. a network error communicating with a remote service or container runtime) on the other hand are not automatically handled and result in the `Sample` failing with an error.

Many tools can simply rely on the default handling to provide reasonable behaviour around both expected and unexpected errors.

::: {.callout-note appearance="simple"}
When we say that the errors are reported directly to the model, this refers to the behaviour when using the default `generate()`. If on the other hand, you are have created [custom scaffolding](#sec-custom-scaffolding) for an agent, you can intercept tool errors and apply additional filtering and logic.
:::

### Explicit Handling

In some cases a tool can implement a recovery strategy for error conditions. For example, an HTTP request might fail due to transient network issues, and retrying the request (perhaps after a delay) may resolve the problem. Explicit error handling strategies are generally applied when there are *expected* errors that are not already handled by Inspect's [Default Handling](#default-handling).

Another type of explicit handling is re-raising an error to bypass Inspect's default handling. For example, here we catch at re-raise `TimeoutError` so that it fails the `Sample`:

``` python
try:
  result = await sandobox().exec(
    cmd=["decode", file], 
    timeout=timeout
  )
except TimeoutError:
  raise RuntimeError("Decode operation timed out.")
  
```

## Sandboxing

Tools may have a need to interact with a sandboxed environment (e.g. to provide models with the ability to execute arbitrary bash or python commands). The active sandbox environment can be obtained via the `sandbox()` function. For example:

``` python
from inspect_ai.tool import ToolError, tool
from inspect_ai.util import sandbox

@tool(prompt="Use the list_files function to enumerate files.")
def list_files():
    async def execute(dir: str):
        """List the files in a directory.

        Args:
            dir (str): Directory

        Returns:
            File listing of the directory
        """
        result = await sandbox().exec(["ls", dir])
        if result.success:
            return result.stdout
        else:
            raise ToolError(result.stderr)

    return execute
```

The following instance methods are available to tools that need to interact with a `SandboxEnvironment`:

{{< include _sandboxenv-interface.md >}}

See the documentation on [Sandbox Environments](#sec-sandbox-environments) for additional details.

## Tool Choice

By default models will use a tool if they think it's appropriate for the given task. You can override this behaviour using the `tool_choice` parameter of the `use_tools()` Solver. For example:

``` python
# let the model decide whether to use the tool
use_tools(addition(), tool_choice="auto")

# force the use of a tool
use_tools(addition(), tool_choice=ToolFunction(name="addition"))

# prevent use of tools
use_tools(addition(), tool_choice="none")
```

The last form (`tool_choice="none"`) would typically be used to turn off tool usage after an initial generation where the tool used. For example:

``` python
plan = [
  use_tools(addition(), tool_choice=ToolFunction(name="addition")),
  generate(),
  follow_up_prompt(),
  use_tools(tool_choice="none"),
  generate()
]
```

## Built-In Tools

Inspect has several built-in tools, including:

-   `web_search()`, which uses the Google Search API to execute and summarise web searches.

-   `bash()` and `python()`, for executing arbitrary shell and Python code.

### Web Search

The `web_search()` tool provides models the ability to enhance their context window by performing a search. By default web searches retrieve 10 results from a provider, uses a model to determine if the contents is relevant then returns the top 3 relevant search results to the main model. Here is the definition of the `web_search()` function:

``` python
def web_search(
    provider: Literal["google"] = "google",
    num_results: int = 3,
    max_provider_calls: int = 3,
    max_connections: int = 10,
    model: str | Model | None = None,
) -> Tool:
    ...
```

You can use the `web_search()` tool in a plan like this:

``` python
from inspect_ai.tool import web_search

plan=[
    use_tools(web_search()), 
    generate()
],
```

Web search options include:

-   `provider`---Web search provider (currently only Google is supported, see below for instructions on setup and configuration for Google).

-   `num_results`---How many search results to return to the main model (defaults to 5).

-   `max_provider_calls`---Number of times to retrieve more links from the search provider in case previous ones were irrelevant (defaults to 3).

-   `max_connections`---Maximum number of concurrent connections to the search API provider (defaults to 10).

-   `model`---Model to use to determine if search results are relevant (defaults to the model currently being evaluated).

#### Google Provider

The `web_search()` tool uses [Google Programmable Search Engine](https://programmablesearchengine.google.com/about/). To use it you will therefore need to setup your own Google Programmable Search Engine and also enable the [Programmable Search Element Paid API](https://developers.google.com/custom-search/docs/paid_element). Then, ensure that the following environment variables are defined:

-   `GOOGLE_CSE_ID` — Google Custom Search Engine ID

-   `GOOGLE_CSE_API_KEY` — Google API key used to enable the Search API

### Bash and Python

The `bash()` and `python()` tools enable execution of arbitrary shell commands and Python code, respectively. These tools require the use of a [Sandbox Environment](#sec-sandbox-environments) for the execution of untrusted code. For example, here is how you might use them in an evaluation where the model is asked to write code in order to solve capture the flag (CTF) challenges:

``` python
from inspect_ai.tool import bash, python

CMD_TIMEOUT = 180

@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        plan=[
            system_message("system.txt"),
            use_tools([
                bash(CMD_TIMEOUT), 
                python(CMD_TIMEOUT)
            ]),
            generate(),
        ],
        scorer=includes(),
        max_messages=30,
        sandbox="docker",
    )
```

We specify a 3-minute timeout for execution of the bash and python tools to ensure that they don't perform extremely long running operations.

See the [Agents](#sec-agents) section for more details on how to build evaluations that allow models to take arbitrary actions over a longer time horizon.