---
############################################################
metrics:
  # Infrastructure metrics:
  - name: num_perplexity_tokens
    display_name: '# tokens'
    description: Average number of tokens in the predicted output (for language modeling, the input too).
  - name: num_bytes
    display_name: '# bytes'
    description: Average number of bytes in the predicted output (for language modeling, the input too).

  - name: num_references
    display_name: '# ref'
    description: Number of references.
  - name: num_train_trials
    display_name: '# trials'
    description: Number of trials, where in each trial we choose an independent, random set of training instances.
  - name: estimated_num_tokens_cost
    display_name: 'cost'
    description: An estimate of the number of tokens (including prompt and output completions) needed to perform the request.
  - name: num_prompt_tokens
    display_name: '# prompt tokens'
    description: Number of tokens in the prompt.
  - name: num_prompt_characters
    display_name: '# prompt chars'
    description: Number of characters in the prompt.
  - name: num_completion_tokens
    display_name: '# completion tokens'
    description: Actual number of completion tokens (over all completions).
  - name: num_output_tokens
    display_name: '# output tokens'
    description: Actual number of output tokens.
  - name: max_num_output_tokens
    display_name: 'Max output tokens'
    description: Maximum number of output tokens (overestimate since we might stop earlier due to stop sequences).
  - name: num_requests
    display_name: '# requests'
    description: Number of distinct API requests.
  - name: num_instances
    display_name: '# eval'
    description: Number of evaluation instances.
  - name: num_train_instances
    display_name: '# train'
    description: Number of training instances (e.g., in-context examples).
  - name: prompt_truncated
    display_name: truncated
    description: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).
  - name: finish_reason_length
    display_name: finish b/c length
    description: Fraction of instances where the the output was terminated because of the max tokens limit.
  - name: finish_reason_stop
    display_name: finish b/c stop
    description: Fraction of instances where the the output was terminated because of the stop sequences.
  - name: finish_reason_endoftext
    display_name: finish b/c endoftext
    description: Fraction of instances where the the output was terminated because the end of text token was generated.
  - name: finish_reason_unknown
    display_name: finish b/c unknown
    description: Fraction of instances where the the output was terminated for unknown reasons.
  - name: num_completions
    display_name: '# completions'
    description: Number of completions.
  - name: predicted_index
    display_name: Predicted index
    description: Integer index of the reference (0, 1, ...) that was predicted by the model (for multiple-choice).

  # Accuracy metrics:
  - name: exact_match
    display_name: Exact match
    short_display_name: EM
    description: Fraction of instances that the predicted output matches a correct reference exactly.
    lower_is_better: false

############################################################
perturbations: []

############################################################
metric_groups:
  - name: accuracy
    display_name: Accuracy
    hide_win_rates: true
    metrics:
      - name: ${main_name}
        split: ${main_split}

  - name: efficiency
    display_name: Efficiency
    metrics:
    - name: inference_runtime
      split: ${main_split}

  - name: general_information
    display_name: General information
    hide_win_rates: true
    metrics:
    - name: num_instances
      split: ${main_split}
    - name: num_train_instances
      split: ${main_split}
    - name: prompt_truncated
      split: ${main_split}
    - name: num_prompt_tokens
      split: ${main_split}
    - name: num_output_tokens
      split: ${main_split}

############################################################

run_groups:
  - name: thai_scenarios
    display_name: Thai Scenarios
    description: Thai-language scenarios
    category: All scenarios
    subgroups:
      - thai_exam
      - thai_exam_onet
      - thai_exam_ic
      - thai_exam_tgat
      - thai_exam_tpat1
      - thai_exam_a_level


  - name: thai_exam
    display_name: ThaiExam
    description: >
      Macro-averaged accuracy on all ThaiExam examinations.
    metric_groups:
      - accuracy
      - efficiency
      - general_information
    environment:
      main_name: exact_match
      main_split: test
    taxonomy:
      task: question answering
      what: n/a
      who: n/a
      when: "?"
      language: Thai and English

  - name: thai_exam_onet
    display_name: ONET
    description: >
      The Ordinary National Educational Test (ONET) is an examination for students in Thailand.
      We select the grade-12 ONET exam, which comprises 5 subjects and each question has 5 choices.
      These subjects are Thai, English, Mathematics, Social Studies, and Science.
      Amounting to a total of 170 questions and options.
    metric_groups:
      - accuracy
      - efficiency
      - general_information
    environment:
      main_name: exact_match
      main_split: test
    taxonomy:
      task: question answering
      what: high school / medical school academic knowledge
      who: n/a
      when: "?"
      language: Thai and English

  - name: thai_exam_ic
    display_name: IC
    description: >
      The Investment Consultant (IC) examination, a licensing test for investment professionals in Thailand.
      Developed by the Stock Exchange of Thailand (SET), features 4 choices per question.
      We extracted questions for levels 1, 2, and 3 resulting in a total of 95 questions and options.
    metric_groups:
      - accuracy
      - efficiency
      - general_information
    environment:
      main_name: exact_match
      main_split: test
    taxonomy:
      task: question answering
      what: licensing for investment professionals
      who: n/a
      when: "?"
      language: Thai

  - name: thai_exam_tgat
    display_name: TGAT
    description: >
      The Thai General Aptitude Test (TGAT), a national high school examination in Thailand.
      Focuses on critical and logical thinking skills.
      We collected a total of 90 questions and answers. The TGAT consists of four choices per question.
    metric_groups:
      - accuracy
      - efficiency
      - general_information
    environment:
      main_name: exact_match
      main_split: test
    taxonomy:
      task: question answering
      what: high school level questions on reasoning
      who: n/a
      when: "?"
      language: English

  - name: thai_exam_tpat1
    display_name: TPAT-1
    description: >
      The Thai Professional Aptitude Test 1 (TPAT-1) is a national high school examination in Thailand.
      The Exam assesses students’ professional skills requirement in medical schools.
      This subset contains reasoning and medical ethics. We collected a total of 116 questions and answers.
      The TPAT-1 consists of 5 choices per question.
    metric_groups:
      - accuracy
      - efficiency
      - general_information
    environment:
      main_name: exact_match
      main_split: test
    taxonomy:
      task: question answering
      what: high school / medical school academic knowledge
      who: n/a
      when: "?"
      language: Thai

  - name: thai_exam_a_level
    display_name: A-Level
    description: >
      An academic knowledge assessment examination (Applied Knowledge Level)
      that covers general foundational subjects taught in schools.
      The content assessed in this examination aligns with the curriculum guidelines
      and emphasizes the practical application of knowledge in daily life.
      We collected a total of 175 questions and answers.
    metric_groups:
      - accuracy
      - efficiency
      - general_information
    environment:
      main_name: exact_match
      main_split: test
    taxonomy:
      task: question answering
      what: high school academic knowledge
      who: n/a
      when: "?"
      language: Thai and English
