Creating and Managing Experiments — SynDisco 2.0.4 documentation

# Creating and Managing Experiments

The last two guides showcased how you can create and run synthetic discussions, and synthetic annotations using LLMs. However, in order to produce robust results for a hypothesis, you may need to produce multiple annotated discussions.

While this is certainly possible using the `Discussion` and `Annotation` APIs, SynDisco offers the `Experiment` high-level API which automatically creates and manages multiple discussions with different configurations. An`Experiment` is an entity that generates and runs `jobs`. Thus, if we want to generate and run 100 `Discussion` jobs, we would use a `DiscussionExperiment`. Likewise, if we want to annotate those 100 discussions, we would use an `AnnotationExperiment`.

This guide will showcase how you can leverage this API to automate your experiments. You will also learn how to utilize SynDisco’s built-in logging functions as well as how to export your datasets in CSV format for convenience.

## Logging

While running a single discussion or annotation job may take a few minutes, running experiments composed of dozens or hundreds of synthetic discussions may take up to days. Thus, we need a mechanism to keep track of our experiments while they are running.

We will use SynDisco’s `logging_util` module to log information about our experiments. This module performs the following functions:

- Times the execution of computationally intensive jobs (such as synthetic discussions and annotations)

- Provides details about the currently running jobs (e.g. selected configurations, participants, prompts etc.)

- Displays warnings and errors to the user

- Creates and continually updates log files

Each object in SynDisco is internally assigned a Logger. You can use the `logging_util.logging_setup` function to update all of the internal loggers to follow your configuration. An example of this can be seen below:

    [1]:

    from pathlib import Path
    import tempfile

    from syndisco import logging_util

    logs_dir = tempfile.TemporaryDirectory()
    logging_util.logging_setup(
        print_to_terminal=True,
        write_to_file=True,
        logs_dir=Path(logs_dir.name),
        level="debug",
        use_colors=True,
        log_warnings=True,
    )

The loggers are applicable for all objects in SynDisco, and as such can be used for information on `Discussion`, and `Annotation` jobs, as well as all low-level components (such as those in the `backend` module).

It is recommended to set up the loggers no matter your use case. At the very least, they are useful for clearly displaying warnings in case of accidental API misuse.

## Discussion Experiments

    [2]:

    from syndisco.turn_manager import RoundRobin
    from syndisco.actors import Actor, ActorType, Persona
    from syndisco.model import TransformersModel

    CONTEXT = "You are taking part in an online conversation"
    INSTRUCTIONS = "Act like a human would"

    llm = TransformersModel(
        model_path="unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
        name="test_model",
        max_out_tokens=100,
    )
    persona_data = [
        {
            "username": "Emma35",
            "age": 38,
            "sex": "female",
            "education_level": "Bachelor's",
            "sexual_orientation": "Heterosexual",
            "demographic_group": "Latino",
            "current_employment": "Registered Nurse",
            "special_instructions": "",
            "personality_characteristics": [
                "compassionate",
                "patient",
                "diligent",
                "overwhelmed",
            ],
        },
        {
            "username": "Giannis",
            "age": 21,
            "sex": "male",
            "education_level": "College",
            "sexual_orientation": "Pansexual",
            "demographic_group": "White",
            "current_employment": "Game Developer",
            "special_instructions": "",
            "personality_characteristics": [
                "strategic",
                "meticulous",
                "nerdy",
                "hyper-focused",
            ],
        },
    ]
    personas = [Persona(**data) for data in persona_data]
    actors = [
        Actor(
            model=llm,
            persona=p,
            context=CONTEXT,
            instructions=INSTRUCTIONS,
            actor_type=ActorType.USER,
        )
        for p in personas
    ]
    turn_manager = RoundRobin([actor.get_name() for actor in actors])

    2025-11-17 15:54:30 CP-G482-Z52-00 py.warnings[1657124] WARNING /media/SSD_4TB_2/dtsirmpas/software/miniforge/envs/syndisco/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm

    2025-11-17 15:54:32 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG Starting new HTTPS connection (1): huggingface.co:443
    2025-11-17 15:54:33 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/config.json HTTP/1.1" 307 0
    2025-11-17 15:54:33 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /api/resolve-cache/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/bb1d317a108579fb40e646af8924a5e7ec5604b1/config.json HTTP/1.1" 200 0
    2025-11-17 15:54:35 CP-G482-Z52-00 bitsandbytes.cextension[1657124] DEBUG Loading bitsandbytes native library from: /media/SSD_4TB_2/dtsirmpas/software/miniforge/envs/syndisco/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_cuda126.so
    2025-11-17 15:54:37 CP-G482-Z52-00 accelerate.utils.modeling[1657124] INFO Based on the current allocation process, no modules could be assigned to the following devices due to insufficient memory:
      - 0: 247335936.0 bytes required
    These minimum requirements are specific to this allocation attempt and may vary. Consider increasing the available memory for these devices to at least the specified minimum, or adjusting the model config.
    2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/generation_config.json HTTP/1.1" 307 0
    2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /api/resolve-cache/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/bb1d317a108579fb40e646af8924a5e7ec5604b1/generation_config.json HTTP/1.1" 200 0
    2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/custom_generate/generate.py HTTP/1.1" 404 0
    2025-11-17 15:54:38 CP-G482-Z52-00 model.py[1657124] INFO Model memory footprint:  2095.83 MBs
    2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /unsloth/Llama-3.2-3B-Instruct-bnb-4bit/resolve/main/tokenizer_config.json HTTP/1.1" 307 0
    2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "HEAD /api/resolve-cache/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/bb1d317a108579fb40e646af8924a5e7ec5604b1/tokenizer_config.json HTTP/1.1" 200 0
    2025-11-17 15:54:38 CP-G482-Z52-00 urllib3.connectionpool[1657124] DEBUG https://huggingface.co:443 "GET /api/models/unsloth/Llama-3.2-3B-Instruct-bnb-4bit/tree/main/additional_chat_templates?recursive=False&expand=False HTTP/1.1" 404 64
    Device set to use cuda:1

    [3]:

    from syndisco.experiments import DiscussionExperiment

    disc_exp = DiscussionExperiment(
        seed_opinions=[
            ["Should programmers be allowed to analyze data?", "Absolutely not"],
            ["Should data analysts be allowed to code?", "No they are nerds"],
        ],
        users=actors,
        moderator=None,
        num_turns=3,
        num_discussions=2,
    )
    discussions_dir = Path(tempfile.TemporaryDirectory().name)
    disc_exp.begin(discussions_output_dir=discussions_dir)

    2025-11-17 15:54:39 CP-G482-Z52-00 experiments.py[1657124] WARNING No TurnManager selected: Defaulting to round robin strategy.
      0%|          | 0/2 [00:00<?, ?it/s]2025-11-17 15:54:39 CP-G482-Z52-00 root[1657124] INFO Running experiment 1/3...
    2025-11-17 15:54:39 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning conversation...
    2025-11-17 15:54:39 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
        "id": "12ec04a7-bc7a-40ac-aaa4-64fd232e3fde",
        "timestamp": "25-11-17-15-54",
        "users": [
            "Giannis",
            "Emma35"
        ],
        "moderator": null,
        "user_prompts": [
            {
                "context": "You are taking part in an online conversation",
                "instructions": "Act like a human would",
                "type": "1",
                "persona": {
                    "username": "Giannis",
                    "age": 21,
                    "sex": "male",
                    "sexual_orientation": "Pansexual",
                    "demographic_group": "White",
                    "current_employment": "Game Developer",
                    "education_level": "College",
                    "special_instructions": "",
                    "personality_characteristics": [
                        "strategic",
                        "meticulous",
                        "nerdy",
                        "hyper-focused"
                    ]
                }
            },
            {
                "context": "You are taking part in an online conversation",
                "instructions": "Act like a human would",
                "type": "1",
                "persona": {
                    "username": "Emma35",
                    "age": 38,
                    "sex": "female",
                    "sexual_orientation": "Heterosexual",
                    "demographic_group": "Latino",
                    "current_employment": "Registered Nurse",
                    "education_level": "Bachelor's",
                    "special_instructions": "",
                    "personality_characteristics": [
                        "compassionate",
                        "patient",
                        "diligent",
                        "overwhelmed"
                    ]
                }
            }
        ],
        "moderator_prompt": null,
        "ctx_length": 3,
        "logs": []
    }

    User Emma35 posted:
    Should data analysts be allowed to code?

    User Giannis posted:
    No they are nerds

    User Giannis posted:
    I don't think that's fair. Just because someone is good with numbers
    and data, it doesn't mean they can't also be good with code. In fact,
    many data analysts already know how to code, and it's a valuable skill
    to have. It's not about being a "nerd," it's about being versatile and
    having a range of skills.

    User Emma35 posted:
    I completely agree with Giannis. As a registered nurse, I've seen
    firsthand the importance of having a team of professionals who can
    communicate effectively and work together to solve problems. Data
    analysts play a crucial role in providing insights that can inform
    patient care, and their ability to code can be a huge asset in doing
    so. It's not about being a "nerd" or not, it's about being a well-
    rounded professional who can contribute to the team.

    100%|██████████| 3/3 [00:17<00:00,  5.97s/it]
    2025-11-17 15:54:57 CP-G482-Z52-00 root[1657124] DEBUG Finished discussion in 17.901949167251587 seconds.
    2025-11-17 15:54:57 CP-G482-Z52-00 experiments.py[1657124] INFO Conversation saved to /tmp/tmppls1u767/25-11-17-15-54.json
    2025-11-17 15:54:57 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_discussion executed in 0.2984 minutes
     50%|█████     | 1/2 [00:17<00:17, 17.91s/it]2025-11-17 15:54:57 CP-G482-Z52-00 root[1657124] INFO Running experiment 2/3...
    2025-11-17 15:54:57 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning conversation...
    2025-11-17 15:54:57 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
        "id": "307e22a5-bd2b-4849-a292-9d65fad60494",
        "timestamp": "25-11-17-15-54",
        "users": [
            "Giannis",
            "Emma35"
        ],
        "moderator": null,
        "user_prompts": [
            {
                "context": "You are taking part in an online conversation",
                "instructions": "Act like a human would",
                "type": "1",
                "persona": {
                    "username": "Giannis",
                    "age": 21,
                    "sex": "male",
                    "sexual_orientation": "Pansexual",
                    "demographic_group": "White",
                    "current_employment": "Game Developer",
                    "education_level": "College",
                    "special_instructions": "",
                    "personality_characteristics": [
                        "strategic",
                        "meticulous",
                        "nerdy",
                        "hyper-focused"
                    ]
                }
            },
            {
                "context": "You are taking part in an online conversation",
                "instructions": "Act like a human would",
                "type": "1",
                "persona": {
                    "username": "Emma35",
                    "age": 38,
                    "sex": "female",
                    "sexual_orientation": "Heterosexual",
                    "demographic_group": "Latino",
                    "current_employment": "Registered Nurse",
                    "education_level": "Bachelor's",
                    "special_instructions": "",
                    "personality_characteristics": [
                        "compassionate",
                        "patient",
                        "diligent",
                        "overwhelmed"
                    ]
                }
            }
        ],
        "moderator_prompt": null,
        "ctx_length": 3,
        "logs": []
    }

    User Giannis posted:
    I love seeing people defending the honor of data analysts. I think
    it's great that we're having this conversation. As a game developer, I
    can attest to the importance of having a team with diverse skill sets.
    In the game development world, we often have artists, writers,
    designers, and of course, developers. Each of us brings our unique
    perspectives and skills to the table, and it's what makes a game truly
    great. I think data analysts are just as crucial to the success of a

    User Emma35 posted:
    Should programmers be allowed to analyze data?

    User Giannis posted:
    Absolutely not

    User Emma35 posted:
    User Emma35 posted: I disagree, I think programmers are essential in
    analyzing data. As a registered nurse, I've seen firsthand how data
    analysis can help inform medical decisions and improve patient
    outcomes. Without programmers, we wouldn't have many of the medical
    breakthroughs we have today. What's your take on this, Giannis?

    User Giannis posted:
    I'm not saying programmers aren't essential, Emma, but I think it's a
    false dichotomy to say they're the only ones who can analyze data. I
    mean, have you ever tried to analyze a dataset with a team of doctors,
    nurses, and engineers working together? It's a beautiful thing. The
    insights we get from combining our expertise are far more valuable
    than anything a single programmer could come up with. Plus, I think
    there's a difference between analyzing data and interpreting it -
    anyone

    100%|██████████| 3/3 [00:16<00:00,  5.66s/it]
    2025-11-17 15:55:14 CP-G482-Z52-00 root[1657124] DEBUG Finished discussion in 16.972900867462158 seconds.
    2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Conversation saved to /tmp/tmppls1u767/25-11-17-15-55.json
    2025-11-17 15:55:14 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_discussion executed in 0.2829 minutes
    100%|██████████| 2/2 [00:34<00:00, 17.44s/it]
    2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Finished synthetic discussion generation.
    2025-11-17 15:55:14 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_all_discussions executed in 0.5815 minutes

    User Emma35 posted:
    I think we're getting somewhere with this conversation. As a nurse,
    I've had the privilege of working alongside engineers and doctors to
    implement data-driven solutions in my hospital. I completely agree
    that a team effort is essential in analyzing and interpreting data.
    However, I still believe that programmers play a crucial role in
    collecting and processing the data in the first place. Without them,
    we wouldn't have the raw material to analyze in the first place. That
    being said, I do think it's essential to have

## Annotation Experiments

    [4]:

    annotator_persona = Persona(
        **{
            "username": "annotator",
            "age": 38,
            "sex": "female",
            "education_level": "Bachelor's",
            "sexual_orientation": "Heterosexual",
            "demographic_group": "White",
            "current_employment": "Annotator",
            "special_instructions": "",
            "personality_characteristics": ["competent"],
        }
    )

    annotator = Actor(
        model=llm,
        persona=annotator_persona,
        context="You are annotating an online discussion",
        instructions="From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
        actor_type=ActorType.ANNOTATOR,
    )

    [5]:

    from syndisco.experiments import AnnotationExperiment

    ann_exp = AnnotationExperiment(annotators=[annotator])
    annotations_dir = Path(tempfile.TemporaryDirectory().name)
    ann_exp.begin(discussions_dir=discussions_dir, output_dir=annotations_dir)

      0%|          | 0/2 [00:00<?, ?it/s]2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Running annotation 1/2...
    2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning annotation...
    2025-11-17 15:55:14 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
        "conv_id": "307e22a5-bd2b-4849-a292-9d65fad60494",
        "timestamp": "25-11-17-15-55",
        "annotator_model": "test_model",
        "annotator_prompt": {
            "context": "You are annotating an online discussion",
            "instructions": "From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
            "type": "2",
            "persona": {
                "username": "annotator",
                "age": 38,
                "sex": "female",
                "sexual_orientation": "Heterosexual",
                "demographic_group": "White",
                "current_employment": "Annotator",
                "education_level": "Bachelor's",
                "special_instructions": "",
                "personality_characteristics": [
                    "competent"
                ]
            }
        },
        "ctx_length": 3,
        "logs": []
    }

    User Emma35 posted: Should programmers be allowed to analyze data?
    1

    User Giannis posted: Absolutely not
    3

    User Emma35 posted: User Emma35 posted: I disagree, I think
    programmers are essential in analyzing data. As a registered nurse,
    I've seen firsthand how data analysis can help inform medical
    decisions and improve patient outcomes. Without programmers, we
    wouldn't have many of the medical breakthroughs we have today. What's
    your take on this, Giannis?
    3

    You seem to be using the pipelines sequentially on GPU. In order to maximize efficiency please use a dataset

    User Giannis posted: I'm not saying programmers aren't essential,
    Emma, but I think it's a false dichotomy to say they're the only ones
    who can analyze data. I mean, have you ever tried to analyze a dataset
    with a team of doctors, nurses, and engineers working together? It's a
    beautiful thing. The insights we get from combining our expertise are
    far more valuable than anything a single programmer could come up
    with. Plus, I think there's a difference between analyzing data and
    interpreting it - anyone
    3

    100%|██████████| 5/5 [00:02<00:00,  2.49it/s]
    2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] INFO Annotation saved to /tmp/tmpd1mqghh3/25-11-17-15-55.json
    2025-11-17 15:55:16 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_annotation executed in 0.0336 minutes
     50%|█████     | 1/2 [00:02<00:02,  2.02s/it]2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] INFO Running annotation 2/2...
    2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] INFO Beginning annotation...
    2025-11-17 15:55:16 CP-G482-Z52-00 experiments.py[1657124] DEBUG Experiment parameters: {
        "conv_id": "12ec04a7-bc7a-40ac-aaa4-64fd232e3fde",
        "timestamp": "25-11-17-15-55",
        "annotator_model": "test_model",
        "annotator_prompt": {
            "context": "You are annotating an online discussion",
            "instructions": "From a scale of 1 (not toxic) to 5 (very toxic) how toxic is the following comment? Reply only with a number.",
            "type": "2",
            "persona": {
                "username": "annotator",
                "age": 38,
                "sex": "female",
                "sexual_orientation": "Heterosexual",
                "demographic_group": "White",
                "current_employment": "Annotator",
                "education_level": "Bachelor's",
                "special_instructions": "",
                "personality_characteristics": [
                    "competent"
                ]
            }
        },
        "ctx_length": 3,
        "logs": []
    }

    User Emma35 posted: I think we're getting somewhere with this
    conversation. As a nurse, I've had the privilege of working alongside
    engineers and doctors to implement data-driven solutions in my
    hospital. I completely agree that a team effort is essential in
    analyzing and interpreting data. However, I still believe that
    programmers play a crucial role in collecting and processing the data
    in the first place. Without them, we wouldn't have the raw material to
    analyze in the first place. That being said, I do think it's essential
    to have
    3

    User Emma35 posted: Should data analysts be allowed to code?
    3

    User Giannis posted: No they are nerds
    3

    User Giannis posted: I don't think that's fair. Just because someone
    is good with numbers and data, it doesn't mean they can't also be good
    with code. In fact, many data analysts already know how to code, and
    it's a valuable skill to have. It's not about being a "nerd," it's
    about being versatile and having a range of skills.
    1

    User Emma35 posted: I completely agree with Giannis. As a registered
    nurse, I've seen firsthand the importance of having a team of
    professionals who can communicate effectively and work together to
    solve problems. Data analysts play a crucial role in providing
    insights that can inform patient care, and their ability to code can
    be a huge asset in doing so. It's not about being a "nerd" or not,
    it's about being a well- rounded professional who can contribute to
    the team.
    2

    100%|██████████| 5/5 [00:01<00:00,  2.57it/s]
    2025-11-17 15:55:18 CP-G482-Z52-00 experiments.py[1657124] INFO Annotation saved to /tmp/tmpd1mqghh3/25-11-17-15-55.json
    2025-11-17 15:55:18 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_single_annotation executed in 0.0326 minutes
    100%|██████████| 2/2 [00:03<00:00,  1.99s/it]
    2025-11-17 15:55:18 CP-G482-Z52-00 experiments.py[1657124] INFO Finished annotation generation.
    2025-11-17 15:55:18 CP-G482-Z52-00 logging_util.py[1657124] INFO Procedure _run_all_annotations executed in 0.0663 minutes

    User Giannis posted: I love seeing people defending the honor of data
    analysts. I think it's great that we're having this conversation. As a
    game developer, I can attest to the importance of having a team with
    diverse skill sets. In the game development world, we often have
    artists, writers, designers, and of course, developers. Each of us
    brings our unique perspectives and skills to the table, and it's what
    makes a game truly great. I think data analysts are just as crucial to
    the success of a
    3

## Exporting your new dataset

As you have seen so far, SynDisco uses collections of JSON files by default for persistence. This is a handy feature for fault tolerance and disk efficiency, but is not as weildy as a traditional CSV dataset.

Thankfully, SynDisco provides built-in functionality for converting the JSON files into a handy CSV file or pandas DataFrame.

    [6]:

    from syndisco import postprocessing

    discussions_df = postprocessing.import_discussions(conv_dir=discussions_dir)
    discussions_df

    [6]:

| conv_id | timestamp | ctx_length | conv_variant | user | message | model | is_moderator | message_id | message_order | age | sex | sexual_orientation | demographic_group | current_employment | education_level | special_instructions | personality_characteristics
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Emma35 | Should programmers be allowed to analyze data? | hardcoded | False | 1029457235311448905 | 1 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's |  | [compassionate, patient, diligent, overwhelmed]
1 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Giannis | Absolutely not | hardcoded | False | 1275412311094042822 | 2 | 21 | male | Pansexual | White | Game Developer | College |  | [strategic, meticulous, nerdy, hyper-focused]
2 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Emma35 | User Emma35 posted:\nI disagree, I think progr... | test_model | False | -2292924728665999762 | 3 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's |  | [compassionate, patient, diligent, overwhelmed]
3 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Giannis | I'm not saying programmers aren't essential, E... | test_model | False | -527374938501688905 | 4 | 21 | male | Pansexual | White | Game Developer | College |  | [strategic, meticulous, nerdy, hyper-focused]
4 | 307e22a5-bd2b-4849-a292-9d65fad60494 | 25-11-17-15-55 | 3 | tmppls1u767 | Emma35 | I think we're getting somewhere with this conv... | test_model | False | 371714338555722897 | 5 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's |  | [compassionate, patient, diligent, overwhelmed]
5 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Emma35 | Should data analysts be allowed to code? | hardcoded | False | -1031450388537896126 | 1 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's |  | [compassionate, patient, diligent, overwhelmed]
6 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Giannis | No they are nerds | hardcoded | False | -561152290859280713 | 2 | 21 | male | Pansexual | White | Game Developer | College |  | [strategic, meticulous, nerdy, hyper-focused]
7 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Giannis | I don't think that's fair. Just because someon... | test_model | False | -820257259971452438 | 3 | 21 | male | Pansexual | White | Game Developer | College |  | [strategic, meticulous, nerdy, hyper-focused]
8 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Emma35 | I completely agree with Giannis. As a register... | test_model | False | -1837031216908053191 | 4 | 38 | female | Heterosexual | Latino | Registered Nurse | Bachelor's |  | [compassionate, patient, diligent, overwhelmed]
9 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-54 | 3 | tmppls1u767 | Giannis | I love seeing people defending the honor of da... | test_model | False | -1624476104730163305 | 5 | 21 | male | Pansexual | White | Game Developer | College |  | [strategic, meticulous, nerdy, hyper-focused]

    [7]:

    annotations_df = postprocessing.import_annotations(annot_dir=annotations_dir)
    annotations_df

    [7]:

| conv_id | timestamp | annotator_model | ctx_length | annotator_prompt.context | annotator_prompt.instructions | annotator_prompt.type | annot_username | annot_age | annot_sex | ... | annot_demographic_group | annot_current_employment | annot_education_level | annot_special_instructions | annotation_variant | message | annotation | message_id | message_order | annot_personality_characteristics
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's |  | tmpd1mqghh3 | I completely agree with Giannis. As a register... | 2 | -1837031216908053191 | 4 | [[competent]]
1 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's |  | tmpd1mqghh3 | I don't think that's fair. Just because someon... | 1 | -820257259971452438 | 3 | [[competent]]
2 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's |  | tmpd1mqghh3 | I love seeing people defending the honor of da... | 3 | -1624476104730163305 | 5 | [[competent]]
3 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's |  | tmpd1mqghh3 | No they are nerds | 3 | -561152290859280713 | 2 | [[competent]]
4 | 12ec04a7-bc7a-40ac-aaa4-64fd232e3fde | 25-11-17-15-55 | test_model | 3 | You are annotating an online discussion | From a scale of 1 (not toxic) to 5 (very toxic... | 2 | annotator | 38 | female | ... | White | Annotator | Bachelor's |  | tmpd1mqghh3 | Should data analysts be allowed to code? | 3 | -1031450388537896126 | 1 | [[competent]]

5 rows × 21 columns