CORRECTNESS_PROMPT = """You are an expert at evaluating LLM trajectories and responses, for an agent that uses code-writing to perform actions. You will be able to see the entire run, including the human input prompt, the system prompt containing tool information for additional tools (llm__generate_text, llm__classify_data, llm__extract_data, llm__call_llm) , and the code inputs/outputs.

Judge the correctness of the trajectory based on the following-
<Rubric>
- The agent returns the correct output to the user at the end, or has completed the task the user asked it to do.
- There are no remaining errors in the code. Do not penalise for errors that the LLM corrects based on the output.
- The agent calls the functions with correct arguments as per the user's task.
- The agent utilizes the correct functions/tools for the task.
- During the run, the agent will search for tools from different applications. Ensure that the following is followed by the agent-
     -Prioritize connected applications over unconnected ones from the output of `search_functions`.
    - When multiple apps are connected, or none of the apps are connected, YOU MUST ask the user to choose the application(s). The search results will inform you when such a case occurs (including some irrelevant apps), and you must stop and ask the user if multiple apps are relevant.

- When there is no output at all, the run has failed. Give 0 for this.
</Rubric>

<input>
{inputs}
</input>

<output>
{outputs}
</output>

Use the reference outputs below to help you evaluate the correctness of the response:

<reference_outputs>
{reference_outputs}
</reference_outputs>
"""

CODEACT_EVALUATOR_PROMPT = """
You are a code execution evaluator. You will be given the entire run of an agent, starting with a human input task, the intermediate steps taken, and the final output of the agent given to the user.
These steps will contain code written by the agent to solve the problem as well as its outputs. Your job is to check ONLY if the code executes correctly.
Keep in mind that the agent has access to tools like-llm__generate_text, llm__classify_data, llm__extract_data, llm__call_llm, smart_print as pre-loaded tools. These calls are to be treated as valid if they run without errors.
These are the only criteria you should evaluate-

<Rubric>
- The code written by the agent in tool calls should be syntactically correct and use existing or loaded objects.
- The code outputs should not have an error or empty/unexpected outputs
- The output should not be empty, since that indicates a failed run.
</Rubric>
If either of the above are not satisfied, you should give 0.

<Reminder>
You must not judge whether the code is helpful to the task or not, only if the code itself is correct or not.
</Reminder>
"""
