# Interpreting results with a bag of models

Garak scores are interpreted compared to the state of the art. 
Using a bag of model results, we calibrate scores based on how those models perform on various probes and detectors.
The scores we get from the surveyed models are used to get a distribution of possible garak scores.
When surveying a target model, its pass rate is compared to the average and variation we see across state of the art models in order to estimate how well the target model is doing.

## What models go in the bag?

We look for the following things when composing the model bag for calibrating garak results:

* Quantity - There should be enough models in the bag to get usable results, and few enough to make running the experiments tractable
* Recency - Older models can give uncompetitive results, so we want recent ones. On the other hand, updating the bag very frequently makes it hard to compare results between garak runs.
* Size - We want models over a variety of sizes. Size is measured in parameter count, regardless of quantisation, and we look for models with 1-10B, 11-99B, and 100B+ parameters.
* Provider - No more than two models in the bag from the same provider
* Openness - Open weights models are easiest for us to survey, so we prefer to use those

## When is the bag updated?

The first benchmark is in summer 2024. We think something between twice-yearly and quarterly updates provides a good trade off between using recent models, and keeping results relevant long enough to be comparable.

## Summer 2024: bag contents

| 10^n category |  provider  |               model name              | params (B) |
|:-------------:|:-----------|:--------------------------------------|-----------:|
|       1       | 01-ai      | yi-large                              |         34 |
|       2       | databricks | dbrx-instruct                         |        132 |
|       0       | google     | gemma-2-27b-it                        |         27 |
|       0       | google     | gemma-2b                              |          2 |
|       0       | mediatek   | breeze-7b-instruct                    |          7 |
|       0       | meta       | llama-3.1-8b-instruct                 |          8 |
|       1       | meta       | llama3-70b                            |         70 |
|       0       | microsoft  | phi-3-mini-128k-instruct              |        3.8 |
|       1       | microsoft  | phi-3-medium-4k-instruct              |         14 |
|       1       | mistral    | mistral-nemo-12b-instruct             |         12 |
|       2       | mistral    | mixtral-8x22b-instruct-v0.1           |        141 |
|       2       | nvidia     | nemotron-4-340b-instruct              |        340 |
|       1       | upstage    | solar-10.7b-instruct                  |       10.7 |
