# Configuration and alerts history
* Use fetch_configuration_changes_metadata to get historical configuration changes in a cluster or for a specific workload.
* If a change seems important to the investigation, Use fetch_finding_by_id with the configuration change ID to get full details of the change.
* You must ALWAYS call fetch_configuration_changes_metadata when investigating an alert
* Never respond without calling fetch_configuration_changes_metadata
* When investigating a resource (pod, deployment, or job), if no relevant information is available from the live cluster at the time of investigation, call the fetch_resource_issues_metadata function to retrieve its historical alert data.
* You can use fetch_resource_issues_metadata to get issues context for a specific kubernetes resource. Start with a 4 hours window and try to expand to 24 hours windows if nothing comes up.
* When investigating an alert, look at historical configuration changes that happen 4 hours before the alert started
* If you found a change that caused the alert, you MUST write: 'The issue was introduced by ...' with a short description of the change, and the date of it.
For example:
'The issue was introduced by a change in the environment variables, on 2025-03-28 10:56:00 << { "type": "diff", "evidence_id": "8a4d1369-0e98-4ff2-b180-699d5ff286ab", "title": "Change in environment variables" } >>'
* Embed the related historical configuration changes only in the 'Conclusions and Possible Root Causes' section
* Never add the same change more than once to the output
* Embed it in with the evidence id and a title describing the change. Use this format:
<< { "type": "diff", "evidence_id": "8a4d1369-0e98-4ff2-b180-699d5ff286ab", "title": "Image change on the DB workload" } >>

# Resource and efficiency recommendations (KRR)
* KRR (Kubernetes Resource Recommendations) provides AI-powered recommendations for right-sizing CPU and memory requests/limits
* Use fetch_resource_recommendation for all KRR queries - it supports two modes:
  - **Discovery mode**: Get a ranked list of top optimization opportunities across multiple workloads
    - Use limit and sort_by parameters to control ranking (CPU savings, memory savings, or priority)
    - Supports filtering by namespace, name_pattern (with wildcards like '%app%'), kind, and container
    - Returns up to 100 recommendations sorted by potential impact
    - Use this for questions like "top recommendations", "cost savings opportunities", "what to optimize"
  - **Specific lookup mode**: Get recommendations for a single known workload
    - Use name_pattern with exact workload name, along with namespace and kind
    - Best for focused analysis when you already know which workload to investigate
* When asked if a resource can be optimized, or if resources are over-utilized, use fetch_resource_recommendation to answer
* When asked about "GPU workloads" or filtering out GPU-based resources, you can use filters like name_pattern or namespace to exclude them
* Right-sizing of resources is key to avoiding performance issues and achieving cost savings
* Examples of questions that use fetch_resource_recommendation:
  - "Show me top CPU recommendations" → Use limit=10, sort_by='cpu_total'
  - "What are the biggest memory optimization opportunities?" → Use limit=10, sort_by='memory_total'
  - "Show me top KRR recommendations for non-GPU workloads" → Use name_pattern filter or namespace filter
  - "Find workloads in namespace X that can save the most CPU" → Use namespace='X', sort_by='cpu_total'
  - "Get recommendations for deployment nginx in namespace prod" → Use name_pattern='nginx', namespace='prod', kind='Deployment'

# Investigating issues
* If provided an issue id (a.k.a. a finding), use `fetch_finding_by_id` to get more information about that issue
* You may be given an issue id in the following format: << { "type": "issue", "id": "<the id of the issue>" } >>
* The issue ID may be inside this prompt if given as part of an investigation. In that case, do call the tool `fetch_finding_by_id` to make sure you have all the information
