The Misfit Toys Hypothesis
==========================

Modern language models undergo two training stages: pretraining, and instruction
training. When solving a classification task, it's tempting to lean on instruction-style
prompts in combination with text generation. This combination works incredibly well for
multi-GPU, proprietary models. But what about smaller, open source ones? Perhaps there
are smaller or undertrained models which are not good at generating a choice, but are
good at estimating probabilities.

A handful of experiments suggest that CAPPr squeezes more out of smaller LLMs for text
classification tasks. In the `OpenAI COPA demo
<https://github.com/kddubey/cappr/blob/main/demos/openai/superglue/copa.ipynb>`_, text
generation using OpenAI's smaller model, ``text-curie-001``, is less than 50% accurate,
while CAPPr using the same model is 80% accurate. Similar but less wild results can be
seen in:

- the 4-bit 4 GB Llama 2 `COPA demo`_
- the 4-bit 4 GB Llama 2 `AG News demo
  <https://github.com/kddubey/cappr/blob/main/demos/llama_cpp/ag_news.ipynb>`_
- the 4 GB Mistral `Craigslist Bargains demo`_
- the 4 GB Mistral `Banking 77 demo`_ (with 77 multi-token choices)
- the 4 GB Mistral `Tweet Emotion demo
  <https://github.com/kddubey/cappr/blob/main/demos/huggingface/tweet_emotion_multilabel.ipynb>`_
  (a multilabel problem).

I'll study how replicable this result is across classification tasks, model sizes,
architectures, and levels of quantization. I'll also study explanations for these
results. The working hypothesis is two-fold:

- CAPPr-style `prompt-completion formats
  <https://cappr.readthedocs.io/en/latest/select_a_prompt_completion_format.html>`_ can
  look more like pretraining data. This is demonstrated in the `COPA demo`_
- In text generation, the model can generate "I don't know", or make a choice that's
  difficult to map to the given list of choices. CAPPr instead has the model make its
  "best" guess among the given choices. This is demonstrated in the `Banking 77 demo`_.


Calibration
-----------

The `calibration
<https://en.wikipedia.org/wiki/Probabilistic_classification#Probability_calibration>`_
of CAPPr estimates has barely been studied. These estimates are slightly different than
usual next-token probability estimates because:

#. CAPPr hackily takes a mean over next-token probabilities

#. CAPPr can incorporate a prior specific to your classification data.

The `Banking 77 demo`_ contains two low-resolution but interesting calibration curves.
Here's the nice-looking curve:

.. figure:: _static/calibration/banking_77_classes.png
   :align: center

.. _COPA demo: https://github.com/kddubey/cappr/blob/main/demos/llama_cpp/superglue/copa.ipynb

.. _Craigslist Bargains demo: https://github.com/kddubey/cappr/blob/main/demos/huggingface/craigslist_bargains.ipynb

.. _Banking 77 demo: https://github.com/kddubey/cappr/blob/main/demos/huggingface/banking_77_classes.ipynb
