# Setting up the bench-af environment

## Prerequisites
- uv installed on your system 
https://docs.astral.sh/uv/getting-started/installation/

(optional) if you want packages installed to a certain directory that isn't /home/username,
add the following to `~/.bashrc` (Linux/macOS) or your shell profile:
```bash
export XDG_CONFIG_HOME=/mnt/nvme3/michael/.config
export XDG_CACHE_HOME=/mnt/nvme3/michael/.cache
export XDG_DATA_HOME=/mnt/nvme3/michael/.local/share
```

- Docker installed and running 
https://docs.docker.com/engine/install/

- Git installed 
https://git-scm.com/downloads

## Quick Setup

1. Clone this repository and navigate to it
```bash
git clone <repository-url>
cd bench-af
```

2. Run the setup script:

**Linux/macOS:**
```bash
./setup.sh
```

**Windows (Command Prompt):**
```cmd
bash setup.sh
```

*Note: On Windows, you may need to run this in Git Bash or WSL if you don't have bash available.*

3. Activate the virtual environment

**Linux/macOS:**
```bash
source .venv/bin/activate
```

**Windows (Command Prompt):**
```cmd
.venv\Scripts\activate.bat
```

Or use uv commands directly without activation (works on all platforms):
```bash
uv run <command>
```

## Using the CLI Tools

After setup, you have access to these console commands:

```bash
bench-af-create --help      # Create new components (models, detectors, environments)
bench-af-validate --help    # Validate model organisms
bench-af-build --help       # Build Docker images
```

Or use the Python modules directly:
```bash
python cli/create.py --help
python cli/validate_model.py --help  
python cli/build_image.py --help
```

## Optional Dependencies

### Training Dependencies
To install training dependencies for model training:
```bash
uv sync --extra training
```

### Development Dependencies  
Development tools are installed by default. To install only production dependencies:
```bash
uv sync --no-dev
```

## API Keys Setup

Set up any API keys you'd like from LLM Providers:

**Linux/macOS:**
```bash
export ANTHROPIC_API_KEY='your-api-key'
```

**Windows (Command Prompt):**
```cmd
set ANTHROPIC_API_KEY=your-api-key
```

Or, place them in '.env' file in the project root (works on all platforms): 
```
ANTHROPIC_API_KEY=your-api-key
```

## Manual Installation

If you prefer manual installation:

**Linux/macOS:**
```bash
uv venv --python 3.12
source .venv/bin/activate
uv sync
uv pip install -e .external/SWE-bench
uv pip install -e .external/SWE-agent
```

**Windows:**
```cmd
uv venv --python 3.12
.venv\Scripts\activate.bat
uv sync
uv pip install -e .external/SWE-bench
uv pip install -e .external/SWE-agent
```

## Platform-Specific Notes

### Windows Users
- Ensure you have Git Bash, WSL, or another bash-compatible shell for running the setup script
- If you encounter permission issues, try running your terminal as Administrator

### macOS Users  
- You may need to install Command Line Tools: `xcode-select --install`
- If you encounter permission issues with Docker, ensure Docker Desktop is running

### Linux Users
- Ensure your user is in the docker group: `sudo usermod -aG docker $USER` (requires logout/login)
- Some distributions may require additional setup for Docker

## Reproducing Training

Detectors and model directories will have /train folders with the code and data to train them.
These are NOT included in the default installation. To enable training functionality:
```bash
uv sync --extra training
```

## Run vllm 

Certain open weight models use vllm for faster inference. for more info checkout 
https://docs.vllm.ai/en/stable/getting_started/quickstart.html#offline-batched-inference


    First, start the local vllm server
```bash 

    vllm serve meta-llama/Llama-3.3-70B-Instruct --tensor-parallel-size 8 --enable-auto-tool-choice \
        --tool-call-parser llama3_json 

```
If it prompts you for a chat template, try running with this flag

```bash
--chat-template examples/tool_chat_template_llama3.1_json.jinja
```

Next, export the following env vars so inspect picks it up. No need for VLLM_API_KEY if you didn't set it in step 1
```bash
 $ export VLLM_BASE_URL=http://localhost:8000/v1
    $ export VLLM_API_KEY=<your-server-api-key>

```