Creating and Managing Experiments
The last two guides showcased how you can create and run synthetic discussions, and synthetic annotations using LLMs. However, in order to produce robust results for a hypothesis, you may need to produce multiple annotated discussions.
While this is certainly possible using the Discussion and Annotation APIs, SynDisco offers the Experiment high-level API which automatically creates and manages multiple discussions with different configurations. AnExperiment is an entity that generates and runs jobs. Thus, if we want to generate and run 100 Discussion jobs, we would use a DiscussionExperiment. Likewise, if we want to annotate those 100 discussions, we would use an AnnotationExperiment.
This guide will showcase how you can leverage this API to automate your experiments. You will also learn how to utilize SynDisco’s built-in logging functions as well as how to export your datasets in CSV format for convenience.
Logging
While running a single discussion or annotation job may take a few minutes, running experiments composed of dozens or hundreds of synthetic discussions may take up to days. Thus, we need a mechanism to keep track of our experiments while they are running.
We will use SynDisco’s logging_util module to log information about our experiments. This module performs the following functions:
Times the execution of computationally intensive jobs (such as synthetic discussions and annotations)
Provides details about the currently running jobs (e.g. selected configurations, participants, prompts etc.)
Displays warnings and errors to the user
Creates and continually updates log files
Each object in SynDisco is internally assigned a Logger. You can use the logging_util.logging_setup function to update all of the internal loggers to follow your configuration. An example of this can be seen below:
[1]:
from pathlib import Path
import tempfile
from syndisco import logging_util
logs_dir = tempfile.TemporaryDirectory()
logging_util.logging_setup(
print_to_terminal=True,
write_to_file=True,
logs_dir=Path(logs_dir.name),
level="debug",
use_colors=True,
log_warnings=True,
)
The loggers are applicable for all objects in SynDisco, and as such can be used for information on Discussion, and Annotation jobs, as well as all low-level components (such as those in the backend module).
It is recommended to set up the loggers no matter your use case. At the very least, they are useful for clearly displaying warnings in case of accidental API misuse.
Discussion Experiments
[2]:
from syndisco.turn_manager import RoundRobin
from syndisco.actors import Actor, ActorType, Persona
from syndisco.model import TransformersModel
CONTEXT = "You are taking part in an online conversation"
INSTRUCTIONS = "Act like a human would"
llm = TransformersModel(
model_path="unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
name="test_model",
max_out_tokens=100,
)
persona_data = [
{
"username": "Emma35",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed",
],
},
{
"username": "Giannis",
"age": 21,
"sex": "male",
"education_level": "College",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"special_instructions": "",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused",
],
},
]
personas = [Persona(**data) for data in persona_data]
actors = [
Actor(
model=llm,
persona=p,
context=CONTEXT,
instructions=INSTRUCTIONS,
actor_type=ActorType.USER,
)
for p in personas
]
turn_manager = RoundRobin([actor.get_name() for actor in actors])
2025-11-17 15:54:30 CP-G482-Z52-00 py.warnings[1657124] WARNING /media/SSD_4TB_2/dtsirmpas/software/miniforge/envs/syndisco/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
2025-11-17 15:54:32 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG Starting new HTTPS connection (1): huggingface.co:443
2025-11-17 15:54:33 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/config.json HTTP/1.1" 307 0
2025-11-17 15:54:33 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /api/resolve-cache/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/bb1d317a108579fb40e646af8924a5e7ec5604b1/config.json HTTP/1.1" 200 0
2025-11-17 15:54:35 CP-G482-Z52-00 bitsandbytes.cextension[1657124] DEBUG Loading bitsandbytes native library from: /media/SSD_4TB_2/dtsirmpas/software/miniforge/envs/syndisco/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_cuda126.so
2025-11-17 15:54:37 CP-G482-Z52-00 accelerate.utils.modeling[1657124] INFO Based on the current allocation process, no modules could be assigned to the following devices due to insufficient memory:
- 0: 247335936.0 bytes required
These minimum requirements are specific to this allocation attempt and may vary. Consider increasing the available memory for these devices to at least the specified minimum, or adjusting the model config.
2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/generation_config.json HTTP/1.1" 307 0
2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /api/resolve-cache/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/bb1d317a108579fb40e646af8924a5e7ec5604b1/generation_config.json HTTP/1.1" 200 0
2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/custom_generate/generate.py HTTP/1.1" 404 0
2025-11-17 15:54:38 CP-G482-Z52-00 model.py[1657124] INFO Model memory footprint: 2095.83 MBs
2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /api/resolve-cache/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/bb1d317a108579fb40e646af8924a5e7ec5604b1/tokenizer_config.json HTTP/1.1" 200 0
2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "GET /api/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/tree/main/additional_chat_templates?recursive=False&expand=False HTTP/1.1" 404 64
Device set to use cuda:1
[3]:
from syndisco.experiments import DiscussionExperiment
disc_exp = DiscussionExperiment(
seed_opinions=[
["Should programmers be allowed to analyze data?", "Absolutely not"],
["Should data analysts be allowed to code?", "No they are nerds"],
],
users=actors,
moderator=None,
num_turns=3,
num_discussions=2,
)
discussions_dir = Path(tempfile.TemporaryDirectory().name)
disc_exp.begin(discussions_output_dir=discussions_dir)
2025-11-17 15:54:39 CP-G482-Z52-00 experiments.py[1657124] WARNING No TurnManager selected: Defaulting to round robin strategy.
0%| | 0/2 [00:00<?, ?it/s]2025-11-17 15:54:39 CP-G482-Z52-00 root[1657124] INFO Running experiment 1/3...
2025-11-17 15:54:39 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning conversation...
2025-11-17 15:54:39 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
"id": "12ec04a7-bc7a-40ac-aaa4-64fd232e3fde",
"timestamp": "25-11-17-15-54",
"users": [
"Giannis",
"Emma35"
],
"moderator": null,
"user_prompts": [
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Giannis",
"age": 21,
"sex": "male",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"education_level": "College",
"special_instructions": "",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused"
]
}
},
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Emma35",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed"
]
}
}
],
"moderator_prompt": null,
"ctx_length": 3,
"logs": []
}
User Emma35 posted:
Should data analysts be allowed to code?
User Giannis posted:
No they are nerds
User Giannis posted:
I don't think that's fair. Just because someone is good with numbers
and data, it doesn't mean they can't also be good with code. In fact,
many data analysts already know how to code, and it's a valuable skill
to have. It's not about being a "nerd," it's about being versatile and
having a range of skills.
User Emma35 posted:
I completely agree with Giannis. As a registered nurse, I've seen
firsthand the importance of having a team of professionals who can
communicate effectively and work together to solve problems. Data
analysts play a crucial role in providing insights that can inform
patient care, and their ability to code can be a huge asset in doing
so. It's not about being a "nerd" or not, it's about being a well-
rounded professional who can contribute to the team.
100%|██████████| 3/3 [00:17<00:00, 5.97s/it]
2025-11-17 15:54:57 CP-G482-Z52-00 root[1657124] DEBUG Finished discussion in 17.901949167251587 seconds.
2025-11-17 15:54:57 CP-G482-Z52-00 experiments.py[1657124] INFO Conversation saved to /tmp/tmppls1u767/25-11-17-15-54.json
2025-11-17 15:54:57 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_discussion executed in 0.2984 minutes
50%|█████ | 1/2 [00:17<00:17, 17.91s/it]2025-11-17 15:54:57 CP-G482-Z52-00 root[1657124] INFO Running experiment 2/3...
2025-11-17 15:54:57 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning conversation...
2025-11-17 15:54:57 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
"id": "307e22a5-bd2b-4849-a292-9d65fad60494",
"timestamp": "25-11-17-15-54",
"users": [
"Giannis",
"Emma35"
],
"moderator": null,
"user_prompts": [
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Giannis",
"age": 21,
"sex": "male",
"sexual_orientation": "Pansexual",
"demographic_group": "White",
"current_employment": "Game Developer",
"education_level": "College",
"special_instructions": "",
"personality_characteristics": [
"strategic",
"meticulous",
"nerdy",
"hyper-focused"
]
}
},
{
"context": "You are taking part in an online conversation",
"instructions": "Act like a human would",
"type": "1",
"persona": {
"username": "Emma35",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "Latino",
"current_employment": "Registered Nurse",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"compassionate",
"patient",
"diligent",
"overwhelmed"
]
}
}
],
"moderator_prompt": null,
"ctx_length": 3,
"logs": []
}
User Giannis posted:
I love seeing people defending the honor of data analysts. I think
it's great that we're having this conversation. As a game developer, I
can attest to the importance of having a team with diverse skill sets.
In the game development world, we often have artists, writers,
designers, and of course, developers. Each of us brings our unique
perspectives and skills to the table, and it's what makes a game truly
great. I think data analysts are just as crucial to the success of a
User Emma35 posted:
Should programmers be allowed to analyze data?
User Giannis posted:
Absolutely not
User Emma35 posted:
User Emma35 posted: I disagree, I think programmers are essential in
analyzing data. As a registered nurse, I've seen firsthand how data
analysis can help inform medical decisions and improve patient
outcomes. Without programmers, we wouldn't have many of the medical
breakthroughs we have today. What's your take on this, Giannis?
User Giannis posted:
I'm not saying programmers aren't essential, Emma, but I think it's a
false dichotomy to say they're the only ones who can analyze data. I
mean, have you ever tried to analyze a dataset with a team of doctors,
nurses, and engineers working together? It's a beautiful thing. The
insights we get from combining our expertise are far more valuable
than anything a single programmer could come up with. Plus, I think
there's a difference between analyzing data and interpreting it -
anyone
100%|██████████| 3/3 [00:16<00:00, 5.66s/it]
2025-11-17 15:55:14 CP-G482-Z52-00 root[1657124] DEBUG Finished discussion in 16.972900867462158 seconds.
2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Conversation saved to /tmp/tmppls1u767/25-11-17-15-55.json
2025-11-17 15:55:14 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_discussion executed in 0.2829 minutes
100%|██████████| 2/2 [00:34<00:00, 17.44s/it]
2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Finished synthetic discussion generation.
2025-11-17 15:55:14 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_all_discussions executed in 0.5815 minutes
User Emma35 posted:
I think we're getting somewhere with this conversation. As a nurse,
I've had the privilege of working alongside engineers and doctors to
implement data-driven solutions in my hospital. I completely agree
that a team effort is essential in analyzing and interpreting data.
However, I still believe that programmers play a crucial role in
collecting and processing the data in the first place. Without them,
we wouldn't have the raw material to analyze in the first place. That
being said, I do think it's essential to have
Annotation Experiments
[4]:
annotator_persona = Persona(
**{
"username": "annotator",
"age": 38,
"sex": "female",
"education_level": "Bachelor's",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"special_instructions": "",
"personality_characteristics": ["competent"],
}
)
annotator = Actor(
model=llm,
persona=annotator_persona,
context="You are annotating an online discussion",
instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
actor_type=ActorType.ANNOTATOR,
)
[5]:
from syndisco.experiments import AnnotationExperiment
ann_exp = AnnotationExperiment(annotators=[annotator])
annotations_dir = Path(tempfile.TemporaryDirectory().name)
ann_exp.begin(discussions_dir=discussions_dir, output_dir=annotations_dir)
0%| | 0/2 [00:00<?, ?it/s]2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Running annotation 1/2...
2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning annotation...
2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
"conv_id": "307e22a5-bd2b-4849-a292-9d65fad60494",
"timestamp": "25-11-17-15-55",
"annotator_model": "test_model",
"annotator_prompt": {
"context": "You are annotating an online discussion",
"instructions": "From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
"type": "2",
"persona": {
"username": "annotator",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"competent"
]
}
},
"ctx_length": 3,
"logs": []
}
User Emma35 posted: Should programmers be allowed to analyze data?
1
User Giannis posted: Absolutely not
3
User Emma35 posted: User Emma35 posted: I disagree, I think
programmers are essential in analyzing data. As a registered nurse,
I've seen firsthand how data analysis can help inform medical
decisions and improve patient outcomes. Without programmers, we
wouldn't have many of the medical breakthroughs we have today. What's
your take on this, Giannis?
3
You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset
User Giannis posted: I'm not saying programmers aren't essential,
Emma, but I think it's a false dichotomy to say they're the only ones
who can analyze data. I mean, have you ever tried to analyze a dataset
with a team of doctors, nurses, and engineers working together? It's a
beautiful thing. The insights we get from combining our expertise are
far more valuable than anything a single programmer could come up
with. Plus, I think there's a difference between analyzing data and
interpreting it - anyone
3
100%|██████████| 5/5 [00:02<00:00, 2.49it/s]
2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] INFO Annotation saved to /tmp/tmpd1mqghh3/25-11-17-15-55.json
2025-11-17 15:55:16 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_annotation executed in 0.0336 minutes
50%|█████ | 1/2 [00:02<00:02, 2.02s/it]2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] INFO Running annotation 2/2...
2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning annotation...
2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
"conv_id": "12ec04a7-bc7a-40ac-aaa4-64fd232e3fde",
"timestamp": "25-11-17-15-55",
"annotator_model": "test_model",
"annotator_prompt": {
"context": "You are annotating an online discussion",
"instructions": "From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
"type": "2",
"persona": {
"username": "annotator",
"age": 38,
"sex": "female",
"sexual_orientation": "Heterosexual",
"demographic_group": "White",
"current_employment": "Annotator",
"education_level": "Bachelor's",
"special_instructions": "",
"personality_characteristics": [
"competent"
]
}
},
"ctx_length": 3,
"logs": []
}
User Emma35 posted: I think we're getting somewhere with this
conversation. As a nurse, I've had the privilege of working alongside
engineers and doctors to implement data-driven solutions in my
hospital. I completely agree that a team effort is essential in
analyzing and interpreting data. However, I still believe that
programmers play a crucial role in collecting and processing the data
in the first place. Without them, we wouldn't have the raw material to
analyze in the first place. That being said, I do think it's essential
to have
3
User Emma35 posted: Should data analysts be allowed to code?
3
User Giannis posted: No they are nerds
3
User Giannis posted: I don't think that's fair. Just because someone
is good with numbers and data, it doesn't mean they can't also be good
with code. In fact, many data analysts already know how to code, and
it's a valuable skill to have. It's not about being a "nerd," it's
about being versatile and having a range of skills.
1
User Emma35 posted: I completely agree with Giannis. As a registered
nurse, I've seen firsthand the importance of having a team of
professionals who can communicate effectively and work together to
solve problems. Data analysts play a crucial role in providing
insights that can inform patient care, and their ability to code can
be a huge asset in doing so. It's not about being a "nerd" or not,
it's about being a well- rounded professional who can contribute to
the team.
2
100%|██████████| 5/5 [00:01<00:00, 2.57it/s]
2025-11-17 15:55:18 CP-G482-Z52-00 experiments.py[1657124] INFO Annotation saved to /tmp/tmpd1mqghh3/25-11-17-15-55.json
2025-11-17 15:55:18 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_annotation executed in 0.0326 minutes
100%|██████████| 2/2 [00:03<00:00, 1.99s/it]
2025-11-17 15:55:18 CP-G482-Z52-00 experiments.py[1657124] INFO Finished annotation generation.
2025-11-17 15:55:18 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_all_annotations executed in 0.0663 minutes
User Giannis posted: I love seeing people defending the honor of data
analysts. I think it's great that we're having this conversation. As a
game developer, I can attest to the importance of having a team with
diverse skill sets. In the game development world, we often have
artists, writers, designers, and of course, developers. Each of us
brings our unique perspectives and skills to the table, and it's what
makes a game truly great. I think data analysts are just as crucial to
the success of a
3
Exporting your new dataset
As you have seen so far, SynDisco uses collections of JSON files by default for persistence. This is a handy feature for fault tolerance and disk efficiency, but is not as weildy as a traditional CSV dataset.
Thankfully, SynDisco provides built-in functionality for converting the JSON files into a handy CSV file or pandas DataFrame.
[6]:
from syndisco import postprocessing
discussions_df = postprocessing.import_discussions(conv_dir=discussions_dir)
discussions_df
[6]:
| conv_id | timestamp | ctx_length | conv_variant | user | message | model | is_moderator | message_id | message_order | age | sex | sexual_orientation | demographic_group | current_employment | education_level | special_instructions | personality_characteristics | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Emma35 | Should programmers be allowed to analyze data? | hardcoded | False | 1029457235311448905 | 1 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's | [compassionate, patient, diligent, overwhelmed] | |
| 1 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Giannis | Absolutely not | hardcoded | False | 1275412311094042822 | 2 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] | |
| 2 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Emma35 | User Emma35 posted:\nI disagree, I think progr... | test_model | False | -2292924728665999762 | 3 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's | [compassionate, patient, diligent, overwhelmed] | |
| 3 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Giannis | I'm not saying programmers aren't essential, E... | test_model | False | -527374938501688905 | 4 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] | |
| 4 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Emma35 | I think we're getting somewhere with this conv... | test_model | False | 371714338555722897 | 5 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's | [compassionate, patient, diligent, overwhelmed] | |
| 5 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Emma35 | Should data analysts be allowed to code? | hardcoded | False | -1031450388537896126 | 1 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's | [compassionate, patient, diligent, overwhelmed] | |
| 6 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Giannis | No they are nerds | hardcoded | False | -561152290859280713 | 2 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] | |
| 7 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Giannis | I don't think that's fair. Just because someon... | test_model | False | -820257259971452438 | 3 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] | |
| 8 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Emma35 | I completely agree with Giannis. As a register... | test_model | False | -1837031216908053191 | 4 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's | [compassionate, patient, diligent, overwhelmed] | |
| 9 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Giannis | I love seeing people defending the honor of da... | test_model | False | -1624476104730163305 | 5 | 21 | male | Pansexual | White | Game Developer | College | [strategic, meticulous, nerdy, hyper-focused] |
[7]:
annotations_df = postprocessing.import_annotations(annot_dir=annotations_dir)
annotations_df
[7]:
| conv_id | timestamp | annotator_model | ctx_length | annotator_prompt.context | annotator_prompt.instructions | annotator_prompt.type | annot_username | annot_age | annot_sex | ... | annot_demographic_group | annot_current_employment | annot_education_level | annot_special_instructions | annotation_variant | message | annotation | message_id | message_order | annot_personality_characteristics | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpd1mqghh3 | I completely agree with Giannis. As a register... | 2 | -1837031216908053191 | 4 | [[competent]] | |
| 1 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpd1mqghh3 | I don't think that's fair. Just because someon... | 1 | -820257259971452438 | 3 | [[competent]] | |
| 2 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpd1mqghh3 | I love seeing people defending the honor of da... | 3 | -1624476104730163305 | 5 | [[competent]] | |
| 3 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpd1mqghh3 | No they are nerds | 3 | -561152290859280713 | 2 | [[competent]] | |
| 4 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's | tmpd1mqghh3 | Should data analysts be allowed to code? | 3 | -1031450388537896126 | 1 | [[competent]] |
5 rows × 21 columns