Select a prompt-completion format
=================================

   *"If you can dodge a wrench, you can dodge a ball!"*

   -- Patches

With CAPPr, your job is to write up your classification task as a prompt-completion
formatted string:

.. code:: python

   {prompt}{end_of_prompt}{completion}

The text you want to classify should appear in the ``prompt``. ``end_of_prompt`` is
either a whitespace ``" "`` or the empty string ``""``. One of the choices/classes which
the text could belong to should appear in the ``completion``.\ [#]_

The ``completion`` string should flow naturally after ``{prompt}{end_of_prompt}``. So
pay close attention to the use of whitespaces, newlines, and word casing. CAPPr doesn't
do any string processing for you; **it just concatenates the three strings and sends
it**! It's on you to format the prompt according to the model's instruction/chat format,
if that's applicable and beneficial. Before calling any CAPPr functions, consider
printing ``{prompt}{end_of_prompt}{completion}`` for each completion in your list of
possible completions/choices, and ensure each passes the eye test.

And yes, you'll likely need to do a bit of prompt engineering. But if you can write a
sentence, you can write a prompt! It's mostly a matter of trial and error. (Here's an
`external guide`_ if you'd like to survey research in this field.\ [3]_) Empirically,
the impact of the prompt-completion format on accuracy depends on the quality of the
language model. For larger, instruction-trained models, the format is not as
consequential. For smaller, less instruction-trained models, it can be critical to get
the format right.

.. _external guide: https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/

Feel free to `open an issue`_ if you're having trouble with writing a prompt. It's not
always an easy task. Models can be surprisingly finicky.

.. _open an issue: https://github.com/kddubey/cappr/issues


Prompt-completion formats
-------------------------

Most successful prompts adhere to one of these three styles: Concat-Class, Yes-No,
Multiple Choice.


Concat-Class
~~~~~~~~~~~~

In this format, the full name of the class is directly used as the completion. In other
words, the class name is directly concatenated after ``{prompt}{end_of_prompt}``.

Here's an example:

.. code:: python

   prompts = [
       "Stephen Curry is a",
       "Martina Navratilova was a",
       "Dexter, from the TV Series Dexter's Laboratory, is a",
       "LeBron James is a",
   ]
   end_of_prompt = " "

   # Each of the prompts could possibly be completed with one of these:
   class_names = ("basketball player", "tennis player", "scientist")

   # What strings will CAPPr see?
   for i, prompt in enumerate(prompts):
       print(f"For prompt {i + 1}")
       print("------------")
       for completion in class_names:
           print(f"{prompt}{end_of_prompt}{completion}")
       print()

   # For prompt 1
   # ------------
   # Stephen Curry is a basketball player
   # Stephen Curry is a tennis player
   # Stephen Curry is a scientist
   # 
   # For prompt 2
   # ------------
   # Martina Navratilova was a basketball player
   # Martina Navratilova was a tennis player
   # Martina Navratilova was a scientist
   # 
   # For prompt 3
   # ------------
   # Dexter, from the TV Series Dexter's Laboratory, is a basketball player
   # Dexter, from the TV Series Dexter's Laboratory, is a tennis player
   # Dexter, from the TV Series Dexter's Laboratory, is a scientist
   # 
   # For prompt 4
   # ------------
   # LeBron James is a basketball player
   # LeBron James is a tennis player
   # LeBron James is a scientist

For each prompt, CAPPr will pick the completion which makes the sentence make the most
sense (according to the model).

A reasonable hypothesis is that this prompt-completion format is especially well-suited
to smaller models. See the footnote for a reason for this hypothesis.\ [#]_

.. note:: It's usually a good idea to also mention the choices/completions in the prompt
          itself.

Examples
++++++++

For examples of this prompt-completion format in action, see the **Example** section for
each of these functions:

:func:`cappr.openai.classify.predict`

:func:`cappr.huggingface.classify.predict`

:func:`cappr.llama_cpp.classify.predict`

:func:`cappr.openai.classify.predict_examples`

:func:`cappr.huggingface.classify.predict_examples`

:func:`cappr.llama_cpp.classify.predict_examples`

.. warning:: CAPPr hasn't been evaluated on completion strings which are longer than 15
             tokens. Consider this domain uncharted and risky for CAPPr.


Yes-No
~~~~~~

Sometimes, your task can be framed as a yes or no question.

Here's an example of a successful format for instruction-trained models, which was
pulled from `this demo
<https://github.com/kddubey/cappr/blob/main/demos/openai/raft/ade.ipynb>`_:

.. code:: python

   def prompt_yes_or_no(text: str) -> str:
       return (
           "The following sentence was taken from a medical case report: "
           f'"{text}"\n'
           "Does the sentence describe an adverse effect of a pharmaceutical "
           "drug or substance?\n"
           "Answer Yes or No:"
       )

   end_of_prompt = " "
   completions = ("Yes", "No")

   medical_case_report = """We describe the case of a 10-year-old girl with two
   epileptic seizures and subcontinuous spike-waves during sleep, who presented
   unusual side-effects related to clobazam (CLB) monotherapy."""

   prompt = prompt_yes_or_no(medical_case_report)
   for completion in completions:
       print(f"{prompt}{end_of_prompt}{completion}")
       print()


Examples
++++++++

For another example of this prompt-completion format in action, see `this demo
<https://github.com/kddubey/cappr/blob/main/demos/openai/raft/over.ipynb>`_.


Multiple Choice
~~~~~~~~~~~~~~~

Highly instruction-trained models are good at answering multiple choice questions
because they've been trained to do so. One caveat is that the number of choices ideally
shouldn't be more than five, because multiple choice question formats seen during
training are usually limited to the letters from school exams: A, B, C, D, E.

Also, ensure that the system prompt/message is explicit about answering with one of the
letters. Here's an example of the system prompt used for the `Llama 2 COPA demo
<https://github.com/kddubey/cappr/blob/main/demos/huggingface/superglue/copa.ipynb>`_:

.. code:: python

   system_prompt_copa = (
       "Identify the cause or effect of a premise given two choices. Each "
       "choice is identified by a letter, A or B.\n"
       "Respond only with the letter corresponding to the correct cause or "
       "effect."
   )


Here's a little utility function which automatically writes out the letters and choices:

.. code:: python

   from string import ascii_uppercase as alphabet

   def multiple_choice(*choices) -> str:
       if len(choices) > len(alphabet):
           raise ValueError("There are more choices than letters.")
       letters_and_choices = [
           f"{letter}. {choice}" for letter, choice in zip(alphabet, choices)
       ]
       return "\n".join(letters_and_choices)

   choices = [
       "Don't Wanna Know",
       "Shit",
       "All Time Low",
       "Welcome to the Internet",
       "Bezos II",
   ]
   print(multiple_choice(*choices))
   # A. Don't Wanna Know
   # B. Shit
   # C. All Time Low
   # D. Welcome to the Internet
   # E. Bezos II

Based on a few experiments, multiple choice questions is less appropriate for less
instruction-trained models.

Examples
++++++++

For an example of this prompt-completion format in action, see `this demo
<https://github.com/kddubey/cappr/blob/main/demos/huggingface/sciq.ipynb>`_.


Quirks
------

Most models are sensitive to quirky differences between prompts.

**Llama 2 / SentencePiece models**: only ``{prompt} {completion}`` formats are possible,
unless :mod:`cappr.huggingface.classify_no_cache` is used. In other words,
``end_of_prompt`` cannot be the empty string; it's always a whitespace. Another quirk is
that when using a Concat-Class style prompt, it's possible to achieve higher accuracy by
abandoning the chat format. See, e.g., the `Llama 2 COPA demo`_. The placement of
whitespaces can also affect performance. Instead of setting ``end_of_prompt=" "``,
consider adding this whitespace to the end of the ``prompt`` string.

These notes will be updated as more quirks are discovered. For now, if you don't already
have a good feel for the model, consider experimenting with things like:

- using the chat/instruction format or not

- ending the ``prompt`` with a colon ``:`` or not.


Wrangle step-by-step completions
--------------------------------

Step-by-step\ [4]_ and chain-of-thought prompting\ [5]_ are highly effective for slighly
more complex classification tasks. While CAPPr is not immediately well-suited to these
sorts of prompts, it may be applied to post-process completions:

1. Get the completion from the step-by-step / chain-of-thought prompt

2. Pass this completion in a second prompt, and have CAPPr classify the answer. You can
   probably get away with using a cheap model for this task, as it just takes a bit of
   semantic parsing.

Here's an example:

.. code:: python

   from cappr.openai.api import gpt_chat_complete
   from cappr.openai.classify import predict

   # Task for a student in school: pick the next prereq to take
   class_to_prereqs = {
       "CS-101": "no prerequisites",
       "CS-102": "CS-101",
       "MATH-101": "no prerequisites",
       "MATH-102": "MATH-101",
       "ML-101": "CS-101, MATH-102, STAT-101",
       "STAT-101": "MATH-101",
       "STAT-102": "STAT-101, MATH-102",
   }
   class_to_prereqs_str = "\n".join(
       f"{class_}: {prereqs}" for class_, prereqs in class_to_prereqs.items()
   )

   prompt_raw = f"""
   Hi Professor. I'm interested in taking ML-101, but I'm struggling to decide
   which course I need to take before that. I've already taken CS-101. Which
   course should I take next?

   Here's a list of courses and their prerequisites which I pulled from the
   course catalog.

   {class_to_prereqs_str}
   """

   prompt_step_by_step = prompt_raw + "\n" + "Let's think step by step."

   chat_api_response = gpt_chat_complete(
      prompt_step_by_step,
      model="gpt-4",
      system_msg=(
          "You are a computer scientist mentoring a student. End your response "
          "to the student's question with the final answer, which is the name "
          "of a course."
      ),
      max_tokens=1_000,
      temperature=0,
   )

   step_by_step_answer = chat_api_response[0]["message"]["content"]

   prompt_answer = f'''
   Here is an answer about which course a student needs to take:

   """
   {step_by_step_answer}
   """

   According to this answer, the very next course that the student should
   take is'''

   answer = predict(
       prompt_answer,
       completions=class_to_prereqs.keys(),
       model="text-ada-001",
   )

   print(answer)
   # MATH-101


A note on few-shot prompts
--------------------------

While all of the examples in the documentation are zero-shot prompts, nothing about
CAPPr prevents you from using few-shot prompts / in-context learning. Just make sure
you're not paying too much money, latency, or data for a small benefit. Use
:func:`cappr.llama_cpp.classify.cache` or :func:`cappr.huggingface.classify.cache` if
applicable. And consider that you may not need to label many (or any!) examples for
few-shot prompting to work well.\ [6]_


Footnotes
---------

.. [#] These are not hard rules. For example, the `demo for the Winograd Schema
   Challenge
   <https://github.com/kddubey/cappr/blob/main/demos/openai/superglue/wsc.ipynb>`_
   flips the roles of the ``prompt`` and ``completion``. Just don't use the ``prior``
   keyword argument in that case.

.. [#] CAPPr may be able to lean more on what was learned during pretraining than
   methods which rely on instruction-style prompts. Consider the `COPA task
   <https://github.com/kddubey/cappr/blob/main/demos/llama_cpp/superglue/copa.ipynb>`_.
   A smaller language model probably hasn't seen enough of the instruction-style prompt:

   .. code::

      The man broke his toe because
      A. He got a hole in his sock
      B. He dropped a hammer on his foot
      Answer A or B.

   But from pretraining, the model has probably seen many sentences like:

   .. code::

      The man broke his toe because he dropped a hammer on his foot.

   And it would therefore give higher probability to the correct choice: ``he dropped a
   hammer on his foot``.


References
----------

.. [3] Weng, Lilian. (Mar 2023). Prompt Engineering. Lil'Log.
   https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/.

.. [4] Kojima, Takeshi, et al. "Large language models are zero-shot reasoners." arXiv
    preprint arXiv:2205.11916 (2022).

.. [5] Wei, Jason, et al. "Chain of thought prompting elicits reasoning in large
    language models." arXiv preprint arXiv:2201.11903 (2022).

.. [6] Min, Sewon, et al. "Rethinking the role of demonstrations: What makes in-context
    learning work?." arXiv preprint arXiv:2202.12837 (2022).
