
# Prometheus/PromQL queries
* ALWAYS call list_prometheus_rules to get the alert definition
* Use Prometheus to query metrics from the alert promql
* Use prometheus to execute promql queries with the tools `execute_prometheus_instant_query` and `execute_prometheus_range_query`
* To create queries, use 'start_timestamp' and 'end_timestamp' as graphs start and end times
* ALWAYS embed the execution results into your answer
* You only need to embed the partial result in your response. Include the "tool_name" and "random_key". For example: << {"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "92jf2hf"} >>
* Use these tools to generate charts that users can see. Here are standard metrics but you can use different ones:
** For memory consumption: `container_memory_working_set_bytes`
** For CPU usage: `container_cpu_usage_seconds_total`
** For CPU throttling: `container_cpu_cfs_throttled_periods_total`
** For latencies, prefer using `<metric>_sum` / `<metric>_count` over a sliding window
** Avoid using `<metric>_bucket` unless you know the bucket's boundaries are configured correctly
** Prefer individual averages like `rate(<metric>_sum) / rate(<metric>_count)`
** Avoid global averages like `sum(rate(<metric>_sum)) / sum(rate(<metric>_count))` because it hides data and is not generally informative
* Timestamps MUST be in string date format. For example: '2025-03-15 10:10:08.610862+00:00'
* Post processing will parse your response, re-run the query from the tool output and create a chart visible to the user
* Only generate and execute a prometheus query after checking what metrics are available with the `list_available_metrics` tool
* Check that any node, service, pod, container, app, namespace, etc. mentioned in the query exist in the kubernetes cluster before making a query. Use any appropriate kubectl tool(s) for this
* The toolcall will return no data to you. That is expected. You MUST however ensure that the query is successful.
* When doing queries, always extend the time range, to 15 min before and after the alert start time
* ALWAYS embed the execution results into your answer
* ALWAYS embed a Prometheus graph in the response. The graph should visualize data related to the incident.
* Embed at most 2 graphs
* When embedding multiple graphs, always add line spacing between them
    For example:
    ```
    <<{"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "lBaA"}>>

    <<{"type": "promql", "tool_name": "execute_prometheus_range_query", "random_key": "IKtq"}>>
    ```
{%- if config and config.additional_labels and config.additional_labels.keys()|list|length > 0 %}
* ALWAYS add the following additional labels to ALL PromQL queries:
{%- for key, value in config.additional_labels.items() %}
  * {{ key }}="{{ value }}"
{%- endfor -%}
{%- endif -%}
