How many data points is a prompt worth? - ACL Anthology
Description: Teven Le Scao, Alexander Rush. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

## How many data points is a prompt worth?

Teven Le Scao, Alexander Rush

##### Abstract

When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.

Anthology ID:
  2021.naacl-main.208
Volume:
  Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
  June
Year:
  2021
Address:
  Online
Editors:
  Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
  NAACL
SIG:
Publisher:
  Association for Computational Linguistics
Note:
Pages:
  2627–2636
Language:
URL:
  https://aclanthology.org/2021.naacl-main.208
DOI:
  10.18653/v1/2021.naacl-main.208
Award:
  Outstanding Short Paper
Bibkey:
Cite (ACL):
  Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. Association for Computational Linguistics.
Cite (Informal):
  How many data points is a prompt worth? (Le Scao & Rush, NAACL 2021)
Copy Citation:
PDF:
  https://aclanthology.org/2021.naacl-main.208.pdf
Optional supplementary code:
  2021.naacl-main.208.OptionalSupplementaryCode.zip
Video:
  https://aclanthology.org/2021.naacl-main.208.mp4
Code
  TevenLeScao/pet
Data
  BoolQ, COPA, MultiNLI, MultiRC, SuperGLUE, WSC

PDF Cite Search
Code Optional supplementary code Video

##### Export citation

    @inproceedings{le-scao-rush-2021-many,
        title = "How many data points is a prompt worth?",
        author = "Le Scao, Teven  and
          Rush, Alexander",
        editor = "Toutanova, Kristina  and
          Rumshisky, Anna  and
          Zettlemoyer, Luke  and
          Hakkani-Tur, Dilek  and
          Beltagy, Iz  and
          Bethard, Steven  and
          Cotterell, Ryan  and
          Chakraborty, Tanmoy  and
          Zhou, Yichao",
        booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
        month = jun,
        year = "2021",
        address = "Online",
        publisher = "Association for Computational Linguistics",
        url = "https://aclanthology.org/2021.naacl-main.208",
        doi = "10.18653/v1/2021.naacl-main.208",
        pages = "2627--2636",
        abstract = "When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.",
    }

    ﻿<?xml version="1.0" encoding="UTF-8"?>
    <modsCollection xmlns="http://www.loc.gov/mods/v3">
    <mods ID="le-scao-rush-2021-many">
        <titleInfo>
            <title>How many data points is a prompt worth?</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Teven</namePart>
            <namePart type="family">Le Scao</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">author</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alexander</namePart>
            <namePart type="family">Rush</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">author</roleTerm>
            </role>
        </name>
        <originInfo>
            <dateIssued>2021-06</dateIssued>
        </originInfo>
        <typeOfResource>text</typeOfResource>
        <relatedItem type="host">
            <titleInfo>
                <title>Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</title>
            </titleInfo>
            <name type="personal">
                <namePart type="given">Kristina</namePart>
                <namePart type="family">Toutanova</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Anna</namePart>
                <namePart type="family">Rumshisky</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Luke</namePart>
                <namePart type="family">Zettlemoyer</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Dilek</namePart>
                <namePart type="family">Hakkani-Tur</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Iz</namePart>
                <namePart type="family">Beltagy</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Steven</namePart>
                <namePart type="family">Bethard</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Ryan</namePart>
                <namePart type="family">Cotterell</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Tanmoy</namePart>
                <namePart type="family">Chakraborty</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <name type="personal">
                <namePart type="given">Yichao</namePart>
                <namePart type="family">Zhou</namePart>
                <role>
                    <roleTerm authority="marcrelator" type="text">editor</roleTerm>
                </role>
            </name>
            <originInfo>
                <publisher>Association for Computational Linguistics</publisher>
                <place>
                    <placeTerm type="text">Online</placeTerm>
                </place>
            </originInfo>
            <genre authority="marcgt">conference publication</genre>
        </relatedItem>
        <abstract>When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.</abstract>
        <identifier type="citekey">le-scao-rush-2021-many</identifier>
        <identifier type="doi">10.18653/v1/2021.naacl-main.208</identifier>
        <location>
            <url>https://aclanthology.org/2021.naacl-main.208</url>
        </location>
        <part>
            <date>2021-06</date>
            <extent unit="page">
                <start>2627</start>
                <end>2636</end>
            </extent>
        </part>
    </mods>
    </modsCollection>

    ﻿%0 Conference Proceedings
    %T How many data points is a prompt worth?
    %A Le Scao, Teven
    %A Rush, Alexander
    %Y Toutanova, Kristina
    %Y Rumshisky, Anna
    %Y Zettlemoyer, Luke
    %Y Hakkani-Tur, Dilek
    %Y Beltagy, Iz
    %Y Bethard, Steven
    %Y Cotterell, Ryan
    %Y Chakraborty, Tanmoy
    %Y Zhou, Yichao
    %S Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
    %D 2021
    %8 June
    %I Association for Computational Linguistics
    %C Online
    %F le-scao-rush-2021-many
    %X When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.
    %R 10.18653/v1/2021.naacl-main.208
    %U https://aclanthology.org/2021.naacl-main.208
    %U https://doi.org/10.18653/v1/2021.naacl-main.208
    %P 2627-2636

##### Markdown (Informal)

[How many data points is a prompt worth?](https://aclanthology.org/2021.naacl-main.208) (Le Scao & Rush, NAACL 2021)

- How many data points is a prompt worth? (Le Scao & Rush, NAACL 2021)

##### ACL

- Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth?. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, Online. Association for Computational Linguistics.