::: {.content-visible when-format="html"}

## GSM8K {#sec-gsm8k}

[GSM8K](https://arxiv.org/abs/2110.14168) (Grade School Math 8K) is a dataset of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning. Here are some samples from the dataset:

| question                                                                                                           | answer                                                                                                                                                                                    |
|----------------------------|--------------------------------------------|
| James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?             | He writes each friend 3\*2=\<\<3\*2=6\>\>6 pages a week So he writes 6\*2=\<\<6\*2=12\>\>12 pages every week That means he writes 12\*52=\<\<12\*52=624\>\>624 pages a year \#### **624** |
| Weng earns \$12 an hour for babysitting. Yesterday, she just did 50 minutes of babysitting. How much did she earn? | Weng earns 12/60 = \$\<\<12/60=0.2\>\>0.2 per minute. Working 50 minutes, she earned 0.2 x 50 = \$\<\<0.2\*50=10\>\>10. \#### **10**                                                      |

: {tbl-colwidths="\[50,50\]"}

Note that the final numeric answers are contained at the end of the **answer** field after the `####` delimiter.

### Setup {.unlisted}

We'll start by importing what we need from Inspect and writing a couple of data handling functions:

1.  `record_to_sample()` to convert raw records to samples. Note that we need a function rather than just mapping field names with a `FieldSpec` because the **answer** field in the dataset needs to be divided into reasoning and the actual answer (which appears at the very end after `####`).
2.  `sample_to_fewshot()` to generate fewshot examples from samples.

::: {.content-hidden}
```{python}
"""
Training Verifiers to Solve Math Word Problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman
https://arxiv.org/abs/2110.14168

# run with default fewshots (10)
inspect eval gsm8k.py

# run with less  or no fewshots
inspect eval gsm8k.py -T fewshot=5
inspect eval gsm8k.py -T fewshot=false
"""
```
:::



```{python}
from inspect_ai import Task, task
from inspect_ai.dataset import Sample, hf_dataset
from inspect_ai.scorer import match
from inspect_ai.solver import (
    generate, prompt_template, system_message
)


def record_to_sample(record):
    DELIM = "####"
    input = record["question"]
    answer = record["answer"].split(DELIM)
    target = answer.pop().strip()
    reasoning = DELIM.join(answer)
    return Sample(
        input=input, 
        target=target, 
        metadata={"reasoning": reasoning.strip()}
    )


def sample_to_fewshot(sample):
    return (
        f"{sample.input}\n\nReasoning:\n"
        + f"{sample.metadata['reasoning']}\n\n"
        + f"ANSWER: {sample.target}"
    )
```

Note that we save the "reasoning" part of the answer in `metadata`—we do this so that we can use it to compose the fewshot prompt (as illustrated in `sample_to_fewshot()`).

Here's the prompt we'll used to elicit a chain of thought answer in the right format:

```python
# setup for problem + instructions for providing answer
MATH_PROMPT_TEMPLATE = """
Solve the following math problem step by step. The last line of your
response should be of the form "ANSWER: $ANSWER" (without quotes) 
where $ANSWER is the answer to the problem.

{prompt}

Remember to put your answer on its own line at the end in the form
"ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to 
the problem, and you do not need to use a \\boxed command.

Reasoning:
""".strip()
```


### Eval {.unlisted}

We'll load the dataset from [HuggingFace](https://huggingface.co/datasets/gsm8k) using the `hf_dataset()` function. By default we use 10 fewshot examples, but the `fewshot` task arg can be used to turn this up, down, or off. The `fewshot_seed` is provided for stability of fewshot examples across runs.

```{python}
@task
def gsm8k(fewshot=10, fewshot_seed=42):
    # build plan dynamically (may or may not be doing fewshot)
    plan = [prompt_template(MATH_PROMPT_TEMPLATE), generate()]
    if fewshot:
        fewshots = hf_dataset(
            path="gsm8k",
            data_dir="main",
            split="train",
            sample_fields=record_to_sample,
            shuffle=True,
            seed=fewshot_seed,
            limit=fewshot,
        )
        plan.insert(
            0,
            system_message(
                "\n\n".join([sample_to_fewshot(sample) for sample in fewshots])
            ),
        )

    # define task
    return Task(
        dataset=hf_dataset(
            path="gsm8k",
            data_dir="main",
            split="test",
            sample_fields=record_to_sample,
        ),
        plan=plan,
        scorer=match(numeric=True),
    )
```

We instruct the `match()` scorer to look for numeric matches at the end of the output. Passing `numeric=True` tells `match()` that it should disregard punctuation used in numbers (e.g. `$`, `,`, or `.` at the end) when making comparisons.

Now we run the evaluation, limiting the number of samples to 100 for development purposes:

```bash
inspect eval gsm8k.py --limit 100
```

:::