# Plan: HuggingFace Inference Provider Support

## Goal

Make the agent provider-agnostic so it can target HuggingFace's Inference API
(serverless and dedicated endpoints) in addition to the existing LM Studio
backend. The agent loop, tool handling, and context management stay untouched —
only the model discovery, connection setup, and LLM call plumbing change.

## Current State

- `call_llm()` hardcodes `openai/{model_id}`, `api_base`, and `api_key="lm-studio"`.
- `discover_model()` and `configure_context()` are LM Studio-specific (proprietary REST endpoints).
- `--base-url` defaults to `http://127.0.0.1:1234`.
- `--model` exists but still routes through the LM Studio `openai/` prefix.
- Existing tests call `call_llm()` with positional args matching the current
  8-parameter signature (see "Existing test migration" below).

## Design Decisions

### Provider abstraction

Introduce a `--provider` argument with values `lmstudio` (default) and
`huggingface`. This controls:

1. How the model identifier is resolved (auto-discover vs. user-supplied).
2. What LiteLLM model prefix is used (`openai/` vs. `huggingface/`).
3. What API key/base is passed.
4. Whether `discover_model`/`configure_context` run.

No abstract class or plugin system — just an `if`/`elif` branch in `main()` and
a provider-aware `call_llm()`. This keeps the diff small and the code readable.

### Authentication

HuggingFace requires an API token. Source it from (in order):

1. `--api-key` CLI argument (new).
2. `HF_TOKEN` environment variable (standard HF convention).
3. Error and exit if neither is set when `--provider huggingface`.

For `lmstudio`, the dummy key `"lm-studio"` is used as today.

### Model naming and normalization

HuggingFace models are specified as `org/model` (e.g.
`zai-org/GLM-5`). LiteLLM expects `huggingface/org/model`.
The user passes `--model zai-org/GLM-5` and the code
prepends `huggingface/`.

**Double-prefix guard:** Before prepending, strip a leading `huggingface/` if
present. This prevents the user accidentally producing
`huggingface/huggingface/org/model` when they copy-paste a full LiteLLM model
string. Implementation:

```python
if model_id.startswith("huggingface/"):
    model_id = model_id[len("huggingface/"):]
model_str = f"huggingface/{model_id}"
```

Additionally, validate at parse time that the resulting bare model ID contains
at least one `/` (the `org/model` separator). If not, exit with a clear error:

```
error: HuggingFace model must be in org/model format (e.g. zai-org/GLM-5)
```

`--model` becomes required when `--provider huggingface` (no auto-discovery).

### Context length

HuggingFace doesn't expose a context-length query. Two options:

- Accept `--max-context-tokens` as a user-supplied hint (already exists).
- If not provided, leave `context_length = None` — the clamping logic in
  `clamp_output_tokens()` already handles this by returning the requested max
  unchanged.

`configure_context()` is skipped entirely for HuggingFace.

### Base URL

For HuggingFace serverless inference, no `api_base` is needed (LiteLLM handles
it). For dedicated Inference Endpoints, the user passes `--base-url
https://xyz.endpoints.huggingface.cloud` and we forward it.

When `--provider huggingface` and `--base-url` is not explicitly set, we pass
`api_base=None` to let LiteLLM use its default HF routing.

## Changes

### 1. New CLI arguments

In `main()`, add:

```
--provider {lmstudio,huggingface}    (default: lmstudio)
--api-key TEXT                        (optional, overrides env var)
```

Change `--base-url` default to `None` and set it to `http://127.0.0.1:1234`
only when provider is `lmstudio` and no explicit value was given.

Validate: when `--provider huggingface`, `--model` is required.

### 2. Refactor `call_llm()` signature

Add `provider` and `api_key` parameters. **Keep `base_url` as the first
positional arg and append the new params as keyword-only after `verbose`** so
that existing positional callers (tests) break loudly rather than silently
misrouting args. The new params default to `provider="lmstudio"`,
`api_key=None` so that existing call sites that pass the original 8 positional
args still work without changes.

```python
def call_llm(base_url, model_id, messages, max_output_tokens, temperature,
             top_p, tools, verbose, *, provider="lmstudio", api_key=None):

    if provider == "lmstudio":
        model_str = f"openai/{model_id}"
        kwargs = {"api_base": f"{base_url}/v1", "api_key": "lm-studio"}
    elif provider == "huggingface":
        # Normalize: strip accidental huggingface/ prefix
        bare_id = model_id.removeprefix("huggingface/")
        model_str = f"huggingface/{bare_id}"
        kwargs = {"api_key": api_key}
        if base_url:
            kwargs["api_base"] = base_url

    # --- Provider-agnostic from here down: retain existing error mapping ---
    try:
        response = litellm.completion(
            model=model_str,
            messages=messages,
            max_tokens=max_output_tokens,
            temperature=temperature,
            top_p=top_p,
            tools=tools,
            tool_choice="auto",
            **kwargs,
        )
    except litellm.ContextWindowExceededError:
        raise ContextOverflowError(...)
    except litellm.BadRequestError as e:
        # regex match → ContextOverflowError, else sys.exit(1)
        ...
    except Exception as e:
        sys.exit(1)
```

The existing `ContextWindowExceededError` → `ContextOverflowError` mapping
and the regex-based `BadRequestError` reclassification (`agent.py:283-293`)
must be preserved exactly. Only the routing preamble (model string, kwargs)
changes; the try/except block and its branches are not modified.

By using keyword-only args with defaults for the new parameters, the 3 direct
`call_llm(...)` invocations in `test_compact.py:343,356,369` and the
`*args/**kwargs` passthrough in `test_logging.py:46` continue to work
unchanged.

### 3. Conditional model discovery

In `main()`, gate the LM Studio-specific logic:

```python
if args.provider == "lmstudio":
    api_base = args.base_url or "http://127.0.0.1:1234"
    if args.model:
        model_id = args.model
        current_context = None
    else:
        model_id, current_context = discover_model(api_base, args.verbose)
        ...
    if args.max_context_tokens is not None:
        configure_context(api_base, model_id, ...)

elif args.provider == "huggingface":
    api_base = args.base_url    # None unless user set it (dedicated endpoint)
    model_id = args.model       # already validated as required
    current_context = args.max_context_tokens   # None if not given
    # discover_model() and configure_context() are NOT called
```

### 4. Resolve API key for HuggingFace

```python
if args.provider == "huggingface":
    api_key = args.api_key or os.environ.get("HF_TOKEN")
    if not api_key:
        parser.error("--api-key or HF_TOKEN env var required for huggingface provider")
else:
    api_key = None
```

Add `import os` at the top (not currently imported).

### 5. Thread new params through agent loop

Every `call_llm()` call site in the agent loop (there are 3 — the initial call
and 2 retry-after-compaction calls) needs the new arguments. Cleanest approach:
build a `llm_kwargs` dict once before the loop and splat it into each call.

```python
llm_kwargs = {
    "provider": args.provider,
    "api_key": api_key,
}

# then in the loop:
msg, finish_reason = call_llm(
    api_base, model_id, messages, effective_max_output,
    args.temperature, args.top_p, tools, args.verbose,
    **llm_kwargs,
)
```

This keeps the original positional args intact (no breakage for tests that
monkeypatch `call_llm`) and passes the new provider/api_key as keyword args.

### 6. Migrate existing tests

Existing tests that interact with `call_llm` or build mock `args` namespaces:

| File | Lines | What to do |
|---|---|---|
| `tests/test_compact.py` | 343, 356, 369 | **No change needed** — calls use the original 8 positional args; new keyword-only params default to `lmstudio`/`None`. |
| `tests/test_logging.py` | 46, 104, 150, 210 | The `fake_call_llm(*args, **kwargs)` passthrough absorbs new kwargs automatically. **No change needed** to the fake. |
| `tests/test_logging.py` | 56, 111, 163, 219 | `SimpleNamespace` args objects are missing `provider` and `api_key`. Add `provider="lmstudio"` and `api_key=None` to each. While the monkeypatched `call_llm` doesn't use these, `main()` now reads `args.provider` before the loop, so the namespace must include them. |
| `tests/test_tools.py` | 293 | `lambda *a, **kw` absorbs new kwargs. **No change needed.** |

Run the full test suite (`uv run python -m pytest tests/ -v`) after the
signature change and before writing new tests to confirm nothing regresses.

### 7. Update argparse description and help text

Change the top-level description from "AI agent using LiteLLM with local LM
Studio" to something provider-neutral like "AI agent using LiteLLM".

### 8. New tests — `tests/test_provider.py`

#### call_llm routing

- Mock `litellm.completion`, call `call_llm` with `provider="lmstudio"`,
  assert it receives `model="openai/..."`, `api_key="lm-studio"`,
  `api_base="http://localhost/v1"`.
- Same for `provider="huggingface"`: assert `model="huggingface/org/name"`,
  `api_key="hf_test"`, no `api_base` when base_url is None.
- HF with explicit base_url: assert `api_base` is passed through.

#### Model ID normalization

- Input `zai-org/GLM-5` → model string
  `huggingface/zai-org/GLM-5`.
- Input `huggingface/zai-org/GLM-5` (already prefixed) →
  same result, no double prefix.

#### CLI validation

- `--provider huggingface` without `--model` → parser error.
- `--provider huggingface --model badname` (no `/`) → parser error.

#### API key resolution

- `--api-key hf_cli` set → uses it regardless of env.
- `--api-key` not set, `HF_TOKEN=hf_env` in env → uses env var.
- Neither set → parser error.

#### Provider path isolation

- When `provider="huggingface"`: assert `discover_model` is **never called**
  and `configure_context` is **never called**. (Monkeypatch both to raise if
  invoked.)
- When `provider="lmstudio"` with no `--model`: assert `discover_model` **is
  called**. With `--max-context-tokens`: assert `configure_context` **is
  called**.

### 9. Update documentation

#### CLAUDE.md

Add HuggingFace to the architecture description and usage examples:

```sh
# LM Studio (default, auto-discovers model)
uv run agent "your task"

# HuggingFace Inference API
export HF_TOKEN=hf_...
uv run agent "your task" --provider huggingface --model zai-org/GLM-5

# HuggingFace dedicated endpoint
uv run agent "your task" --provider huggingface --model zai-org/GLM-5 \
    --base-url https://xyz.endpoints.huggingface.cloud --api-key hf_...
```

#### README.md

- Update the tagline (line 6) to mention HuggingFace alongside LM Studio.
- Add `--provider` and `--api-key` to the CLI options table (line 66-79).
- Add a "HuggingFace" subsection under Examples (after "Advanced Configuration",
  line 120) with the same examples as CLAUDE.md.
- Update Prerequisites (line 27-31): LM Studio is only required for the default
  provider; HuggingFace requires an API token.
- Update "How It Works > Agent Loop" step 1 (line 191): note that discovery only
  applies to LM Studio; HF uses the user-supplied model directly.
- Add `HF_TOKEN` to the Environment Variables section (line 239-243).

## Files touched

| File | Change |
|---|---|
| `agent.py` | New CLI args, provider branching in `main()`, refactored `call_llm()` |
| `tests/test_logging.py` | Add `provider` and `api_key` to `SimpleNamespace` args (4 locations) |
| `tests/test_provider.py` | New test file for provider routing, normalization, validation, path isolation |
| `CLAUDE.md` | Updated architecture docs and usage examples |
| `README.md` | Updated tagline, CLI table, examples, prerequisites, env vars |

## Out of scope

- Other providers (Anthropic, OpenAI direct, Ollama) — same pattern applies
  later but not part of this change.
- Automatic model capability detection (checking if the HF model supports
  tool calling). The user is responsible for choosing a compatible model.
- Streaming support — the agent doesn't use streaming today and that's
  orthogonal to provider support.
