---
title: LLM Support
description: LLMs integrated with CocoIndex for various built-in functions
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

CocoIndex provides builtin functions integrating with various LLM APIs, for various inference tasks:
*   [Text Generation](#text-generation): use LLM to generate text.
*   [Text Embedding](#text-embedding): embed text into a vector space.

## LLM API Types

We support integrating with LLM with different types of APIs.
Each LLM API type is specified by a `cocoindex.LlmApiType` enum.

We support the following types of LLM APIs:

| API Name | `LlmApiType` enum | Text Generation | Text Embedding |
|----------|---------------------|--------------------|--------------------|
| [OpenAI](#openai) | `LlmApiType.OPENAI` | ✅ | ✅ |
| [Ollama](#ollama) | `LlmApiType.OLLAMA` | ✅ | ❌ |
| [Google Gemini](#google-gemini) | `LlmApiType.GEMINI` | ✅ | ✅ |
| [Vertex AI](#vertex-ai) | `LlmApiType.VERTEX_AI` | ✅ | ✅ |
| [Anthropic](#anthropic) | `LlmApiType.ANTHROPIC` | ✅ | ❌ |
| [Voyage](#voyage) | `LlmApiType.VOYAGE` | ❌ | ✅ |
| [LiteLLM](#litellm) | `LlmApiType.LITE_LLM` | ✅ | ❌ |
| [OpenRouter](#openrouter) | `LlmApiType.OPEN_ROUTER` | ✅ | ❌ |
| [vLLM](#vllm) | `LlmApiType.VLLM` | ✅ | ❌ |

## LLM Tasks

### Text Generation

Generation is used as a building block for certain CocoIndex functions that process data using LLM generation.

We have one builtin functions using LLM generation for now:

*  [`ExtractByLlm`](/docs/ops/functions#extractbyllm): it extracts information from input text.

#### LLM Spec

When calling a CocoIndex function that uses LLM generation, you need to provide a `cocoindex.LlmSpec` dataclass, to configure the LLM you want to use in these functions.
It has the following fields:

*   `api_type` (type: [`cocoindex.LlmApiType`](/docs/ai/llm#llm-api-types), required): The type of integrated LLM API to use, e.g. `cocoindex.LlmApiType.OPENAI` or `cocoindex.LlmApiType.OLLAMA`.
    See supported LLM APIs in the [LLM API integrations](#llm-api-integrations) section below.
*   `model` (type: `str`, required): The name of the LLM model to use.
*   `address` (type: `str`, optional): The address of the LLM API.
*   `api_config` (optional): Specific configuration for the LLM API. Only needed for specific LLM APIs (see below).


### Text Embedding

Embedding means converting text into a vector space, usually for similarity matching.

We provide a builtin function [`EmbedText`](/docs/ops/functions#embedtext) that converts a given text into a vector space.
The spec takes the following fields:

*   `api_type` (type: `cocoindex.LlmApiType`, required)
*   `model` (type: `str`, required)
*   `address` (type: `str`, optional)
*   `output_dimension` (type: `int`, optional)
*   `task_type` (type: `str`, optional)

See documentation for [`EmbedText`](/docs/ops/functions#embedtext) for more details about these fields.

## LLM API Integrations

CocoIndex integrates with various LLM APIs for these functions.

### OpenAI

To use the OpenAI LLM API, you need to set the environment variable `OPENAI_API_KEY`.
You can generate the API key from [OpenAI Dashboard](https://platform.openai.com/api-keys).

Currently we don't support custom address for OpenAI API.

You can find the full list of models supported by OpenAI [here](https://platform.openai.com/docs/models).

For text generation, a spec for OpenAI looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.OPENAI,
    model="gpt-4o",
)
```

</TabItem>
</Tabs>

For text embedding, a spec for OpenAI looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.OPENAI,
    model="text-embedding-3-small",
)
```

</TabItem>
</Tabs>

### Ollama

[Ollama](https://ollama.com/) allows you to run LLM models on your local machine easily. To get started:

*   [Download](https://ollama.com/download) and install Ollama.
*   Pull your favorite LLM models by the `ollama pull` command, e.g.
    ```bash
    ollama pull llama3.2
    ```
You can find the [list of models](https://ollama.com/library) supported by Ollama.

A spec for Ollama looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.OLLAMA,
    model="llama3.2:latest",
    # Optional, use Ollama's default port (11434) on localhost if not specified
    address="http://localhost:11434",
)
```

</TabItem>
</Tabs>

### Google Gemini

Google exposes Gemini through Google AI Studio APIs.
Based on [Gemini API recommendation](https://cloud.google.com/ai/gemini?hl=en), this is recommended for experimenting and prototyping purposes.
You may use [Vertex AI](#vertex-ai) for production usages.

To use the Gemini by Google AI Studio API, you need to set the environment variable `GEMINI_API_KEY`.
You can generate the API key from [Google AI Studio](https://aistudio.google.com/apikey).

You can find the full list of models supported by Gemini [here](https://ai.google.dev/gemini-api/docs/models).

For text generation, a spec looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.GEMINI,
    model="gemini-2.5-flash",
)
```

</TabItem>
</Tabs>

For text embedding, a spec looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.GEMINI,
    model="text-embedding-004",
    task_type="SEMANTICS_SIMILARITY",
)
```

</TabItem>
</Tabs>

All supported embedding models can be found [here](https://ai.google.dev/gemini-api/docs/embeddings#embeddings-models).
Gemini supports task type (optional), which can be found [here](https://ai.google.dev/gemini-api/docs/embeddings#supported-task-types).


### Vertex AI

Google Cloud Vertex AI offers production-level integration with Google Gemini models.

To use the Vertex AI API:

1.  Register / login in *Google Cloud*.
2.  In [Google Cloud Console](https://console.cloud.google.com/).
    -   Search for *Vertex AI API*. Enable this API.
    -   Search for *Billing*. Set the billing account for the current project.
3.  Setup [application default credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials).

    The easiest way during development is to [install the `gcloud` CLI](https://cloud.google.com/sdk/docs/install-sdk) and run

    ```sh
    gcloud auth application-default login
    ```

Spec for Vertex AI takes additional `api_config` field, in type `cocoindex.llm.VertexAiConfig` with the following fields:
-   `project` (type: `str`, required): The project ID of the Google Cloud project.
-   `region` (type: `str`, optional): The region of the Google Cloud project. Use `global` if not specified.

You can find the full list of models supported by Vertex AI [here](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions).

For text generation, a spec for Vertex AI looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.VERTEX_AI,
    model="gemini-2.0-flash",
    api_config=cocoindex.llm.VertexAiConfig(project="your-project-id"),
)
```

</TabItem>
</Tabs>


For text embedding, a spec for Vertex AI looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.VERTEX_AI,
    model="text-embedding-005",
    task_type="SEMANTICS_SIMILARITY",
    api_config=cocoindex.llm.VertexAiConfig(project="your-project-id"),
)
```

</TabItem>
</Tabs>

Vertex AI API supports task type (optional), which can be found [here](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api#parameter-list).

### Anthropic

To use the Anthropic LLM API, you need to set the environment variable `ANTHROPIC_API_KEY`.
You can generate the API key from [Anthropic API](https://console.anthropic.com/settings/keys).

A text generation spec for Anthropic looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.ANTHROPIC,
    model="claude-3-5-sonnet-latest",
)
```

</TabItem>
</Tabs>

You can find the full list of models supported by Anthropic [here](https://docs.anthropic.com/en/docs/about-claude/models/all-models).

### Voyage

To use the Voyage LLM API, you need to set the environment variable `VOYAGE_API_KEY`.
You can generate the API key from [Voyage dashboard](https://dashboard.voyageai.com/organization/api-keys).

A text embedding spec for Voyage looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.VOYAGE,
    model="voyage-code-3",
    task_type="document",
)
```

</TabItem>
</Tabs>

Voyage API supports `document` and `query` as task types (optional, a.k.a. `input_type` in Voyage API, see [Voyage API documentation](https://docs.voyageai.com/reference/embeddings-api) for details).

### LiteLLM

To use the LiteLLM API, you need to set the environment variable `LITELLM_API_KEY`.

#### 1. Install LiteLLM Proxy

```bash
pip install 'litellm[proxy]'
```

#### 2. Create a `config.yml` for LiteLLM

**Example for DeepSeek:**

Use this in your `config.yml`:

```yaml
model_list:
  - model_name: deepseek-chat
    litellm_params:
        model: deepseek/deepseek-chat
        api_key: os.environ/DEEPSEEK_API_KEY
```

You need to set the environment variable `DEEPSEEK_API_KEY` to your DeepSeek API key.

**Example for Groq:**

Use this in your `config.yml`:

```yaml
model_list:
  - model_name: groq-llama-3.3-70b-versatile
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: "os.environ/GROQ_API_KEY"
```

You need to set the environment variable `GROQ_API_KEY` to your Groq API key.


#### 3. Run LiteLLM Proxy

```bash
litellm --config config.yml
```

#### 4. A Spec for LiteLLM will look like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.LITE_LLM,
    model="deepseek-chat",
    address="http://127.0.0.1:4000", # default url of LiteLLM
)
```

</TabItem>
</Tabs>

You can find the full list of models supported by LiteLLM [here](https://docs.litellm.ai/docs/providers).

### OpenRouter

To use the OpenRouter API, you need to set the environment variable `OPENROUTER_API_KEY`.
You can generate the API key from [here](https://openrouter.ai/settings/keys).

A spec for OpenRouter looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.OPEN_ROUTER,
    model="deepseek/deepseek-r1:free",
)
```

</TabItem>
</Tabs>

You can find the full list of models supported by OpenRouter [here](https://openrouter.ai/models).

### vLLM

Install vLLM:

```bash
pip install vllm
```

Run vLLM Server

```bash
vllm serve deepseek-ai/deepseek-coder-1.3b-instruct
```


A spec for vLLM looks like this:

<Tabs>
<TabItem value="python" label="Python" default>

```python
cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.VLLM,
    model="deepseek-ai/deepseek-coder-1.3b-instruct",
    address="http://127.0.0.1:8000/v1",
)
```

</TabItem>
</Tabs>
