---
phase: 04-add-jq-and-yq-for-querying-large-json-and-yaml-files
plan: 2
type: execute
wave: 2
depends_on: ["04-01"]
files_modified:
  - src/fs_mcp/server.py
autonomous: true

must_haves:
  truths:
    - "Agent can query JSON files using jq expressions"
    - "Agent can query YAML files using yq expressions"
    - "Query results are bounded to 100 items max"
    - "Queries timeout after 30 seconds to prevent hangs"
    - "read_files suggests query tools for large JSON/YAML files"
  artifacts:
    - path: "src/fs_mcp/server.py"
      provides: "query_json and query_yaml tools with bounded output"
      exports: ["query_json", "query_yaml"]
      min_lines: 120
    - path: "src/fs_mcp/server.py"
      provides: "Enhanced read_files with large file detection"
      contains: "large_file_passthrough"
      min_lines: 15
  key_links:
    - from: "query_json"
      to: "jq subprocess"
      via: "subprocess.run with timeout"
      pattern: "subprocess\\.run.*\\['jq'"
    - from: "query_yaml"
      to: "yq subprocess"
      via: "subprocess.run with timeout"
      pattern: "subprocess\\.run.*\\['yq'"
    - from: "read_files"
      to: "token size check"
      via: "approximate token calculation"
      pattern: "len\\(content\\) / 4"
---

<objective>
Implement query_json and query_yaml tools for efficient structured data exploration, and enhance read_files to guide agents toward query tools for large JSON/YAML files.

Purpose: Enable agents to explore large structured files (OpenAPI specs, dbt manifests, package-lock.json) without context overflow, completing the grep → read → query pattern.
Output: Two new MCP tools (query_json, query_yaml) with bounded results and timeout protection; enhanced read_files that blocks oversized JSON/YAML files by default.
</objective>

<execution_context>
@~/.config/opencode/get-shit-done/workflows/execute-plan.md
@~/.config/opencode/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-add-jq-and-yq-for-querying-large-json-and-yaml-files/04-CONTEXT.md
@.planning/phases/04-add-jq-and-yq-for-querying-large-json-and-yaml-files/04-RESEARCH.md
@.planning/phases/04-add-jq-and-yq-for-querying-large-json-and-yaml-files/04-01-SUMMARY.md
@src/fs_mcp/server.py
</context>

<tasks>

<task type="auto">
  <name>Task 1: Implement query_json tool</name>
  <files>src/fs_mcp/server.py</files>
  <action>
Create query_json() tool following the complete pattern from RESEARCH.md (lines 293-330).

**Location:** Add after the grep_content tool (around line 697 in server.py).

**Tool signature:**
```python
@mcp.tool()
def query_json(file_path: str, jq_expression: str, timeout: int = 30) -> str:
```

**Docstring (optimized for weak models):**
Use the exact docstring pattern from RESEARCH.md (lines 336-365), which includes:
- Clear description of purpose (query large JSON without reading entire content)
- Common query patterns with examples (.field_name, .items[], select, slicing)
- Workflow example (4-step process from structure overview to filtering)
- Result limit notice (first 100 results)
- Args documentation with jqlang.github.io/jq reference

**Implementation (following RESEARCH.md lines 293-330):**
1. Check IS_JQ_AVAILABLE first; if False, call check_jq() and return error with install instructions
2. Validate file_path using validate_path() (security barrier)
3. Build command: ['jq', '-c', jq_expression, str(validated_path)]
4. Execute with subprocess.run(capture_output=True, text=True, timeout=timeout, check=False)
5. Handle three error cases:
   - FileNotFoundError: Return "Error: 'jq' command not found. Please ensure jq is installed and in your PATH."
   - subprocess.TimeoutExpired: Return f"Error: Query timed out after {timeout} seconds. Please simplify your query."
   - result.returncode != 0: Return f"Query error: {result.stderr.strip()}"
6. Handle empty results: Check if output is empty or whitespace-only, return "No results found."
7. Limit results to 100 lines:
   - Split output by newlines
   - If >100 lines: return first 100 + truncation message: "\n\n--- Truncated. Showing 100 of {len(lines)} results. ---\nRefine your query or use jq slicing: .items[100:200]"
   - Otherwise return full output

**Pattern to follow:** Identical error handling structure to grep_content (lines 622-697), adapted for jq specifics.

**Do NOT:**
- Use shell=True (security risk)
- Parse JSON in Python first (defeats the purpose)
- Forget timeout parameter (can hang on large files)
- Ignore stderr (contains valuable syntax errors)
  </action>
  <verify>
Test with sample JSON file:
```bash
echo '{"items":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]}' > /tmp/test.json
# Start server, call query_json("/tmp/test.json", ".items[]")
# Expected: Two lines of compact JSON
# Call query_json with invalid syntax, expect stderr error returned
```
  </verify>
  <done>
query_json tool exists, accepts file_path + jq_expression + optional timeout, returns bounded compact JSON results, handles errors gracefully, returns install instructions if jq missing.
  </done>
</task>

<task type="auto">
  <name>Task 2: Implement query_yaml tool</name>
  <files>src/fs_mcp/server.py</files>
  <action>
Create query_yaml() tool as parallel implementation to query_json.

**Location:** Add immediately after query_json.

**Tool signature:**
```python
@mcp.tool()
def query_yaml(file_path: str, yq_expression: str, timeout: int = 30) -> str:
```

**Docstring (adapted from query_json, yq-specific):**
```
Query a YAML file using yq expressions (mikefarah/yq with jq-like syntax). Use this to efficiently explore large YAML files without reading the entire content into memory.

**Common Query Patterns:**
- Get specific field: '.field_name'
- Array iteration: '.items[]'
- Filter array: '.items[] | select(.active == true)'
- Select fields: '.items[] | {name, id}'
- Array slice: '.items[0:100]' (first 100 items)
- Count items: '.items | length'

**Workflow Example:**
1. Get structure overview: query_yaml("config.yaml", "keys")
2. Count array items: query_yaml("config.yaml", ".services | length")
3. Explore first few: query_yaml("config.yaml", ".services[0:5]")
4. Filter specific: query_yaml("config.yaml", ".services[] | select(.enabled == true)")

**Result Limit:** Returns first 100 results. For more, use slicing: .items[100:200]

Args:
    file_path: Path to YAML file (relative or absolute)
    yq_expression: yq query expression (jq-like syntax, see mikefarah.gitbook.io/yq)
    timeout: Query timeout in seconds (default: 30)

Returns:
    Compact JSON results (one per line), or error message
```

**Implementation:**
Identical to query_json, with these differences:
1. Check IS_YQ_AVAILABLE instead of IS_JQ_AVAILABLE
2. Call check_yq() for error messages
3. Build command: ['yq', '-o=json', '-I=0', yq_expression, str(validated_path)]
   - Note: -o=json -I=0 ensures compact JSON output matching jq's -c flag
4. Error messages reference yq instead of jq
5. All other logic identical (validation, subprocess execution, error handling, result limiting)

**Pattern to follow:** Exact copy of query_json logic with yq-specific adjustments.
  </action>
  <verify>
Test with sample YAML file:
```bash
echo 'items:\n  - id: 1\n    name: Alice\n  - id: 2\n    name: Bob' > /tmp/test.yaml
# Start server, call query_yaml("/tmp/test.yaml", ".items[]")
# Expected: Two lines of compact JSON (yq outputs JSON by default with -o=json)
```
  </verify>
  <done>
query_yaml tool exists, accepts file_path + yq_expression + optional timeout, returns bounded compact JSON results, handles errors gracefully, returns install instructions if yq missing.
  </done>
</task>

<task type="auto">
  <name>Task 3: Enhance read_files with large JSON/YAML detection</name>
  <files>src/fs_mcp/server.py</files>
  <action>
Enhance read_files() tool to detect large JSON/YAML files and guide agents toward query tools.

**Location:** Modify existing read_files function (starts around line 160).

**Add parameter to function signature:**
```python
def read_files(files: List[FileReadRequest], large_file_passthrough: bool = False) -> str:
```

**Add to docstring (after existing parameter descriptions):**
```
    large_file_passthrough: If False (default), blocks reading JSON/YAML files >100k tokens and suggests using query_json/query_yaml instead. Set to True to read anyway.
```

**Implementation (add after path validation, before file reading loop):**
For each file in the files list:
1. Check file extension: ends with .json or .yaml or .yml
2. If structured file, check size: get file size in bytes via os.path.getsize(validated_path)
3. Approximate token count: tokens ≈ bytes / 4 (standard rough estimate)
4. If tokens > 100_000 and not large_file_passthrough:
   - Return error message:
```
Error: {file_path} is a large {JSON|YAML} file (~{tokens:,} tokens).

Reading the entire file may overflow your context window. Consider using:
- query_json("{file_path}", "keys") to explore structure
- query_json("{file_path}", ".items[0:10]") to preview data
- query_json("{file_path}", ".items[] | select(.field == 'value')") to filter

Or set large_file_passthrough=True to read anyway.
```
5. Continue normal file reading for files that pass the check

**Token threshold:** 100k tokens = 400k bytes = ~400 KB (matches CONTEXT.md decision)

**Pattern to follow:** Add checks before existing read logic, don't break existing functionality for normal files.
  </action>
  <verify>
Test with large JSON file:
```bash
# Create 500KB JSON file (>100k tokens)
python -c "import json; data = {'items': [{'id': i, 'data': 'x'*100} for i in range(5000)]}; open('/tmp/large.json', 'w').write(json.dumps(data))"
# Call read_files with /tmp/large.json
# Expected: Error message suggesting query_json
# Call with large_file_passthrough=True
# Expected: File content returned
```
  </verify>
  <done>
read_files tool has large_file_passthrough parameter (default False), detects JSON/YAML files >100k tokens, blocks reading and suggests query tools, allows override with passthrough=True.
  </done>
</task>

</tasks>

<verification>
1. Check query_json tool exists with correct signature and docstring
2. Check query_yaml tool exists with correct signature and docstring
3. Test query_json with valid and invalid jq expressions
4. Test query_yaml with valid and invalid yq expressions
5. Test result limiting: query returning >100 results shows truncation message
6. Test timeout: complex query on large file times out gracefully
7. Test read_files large file detection with JSON and YAML files
8. Verify security: validate_path() called before subprocess execution
9. Grep for pattern consistency: both tools follow same error handling structure
</verification>

<success_criteria>
1. query_json tool accepts file_path, jq_expression, optional timeout; returns bounded compact JSON or error
2. query_yaml tool accepts file_path, yq_expression, optional timeout; returns bounded compact JSON or error
3. Both tools limit results to 100 items with truncation message suggesting slicing
4. Both tools timeout after 30 seconds (configurable) with helpful error message
5. Both tools validate paths with validate_path() before subprocess execution
6. Tool docstrings include syntax examples and workflow guidance for weak models
7. read_files detects large JSON/YAML (>100k tokens) and suggests query tools
8. read_files allows override with large_file_passthrough=True flag
9. No shell=True usage (security requirement)
</success_criteria>

<output>
After completion, create `.planning/phases/04-add-jq-and-yq-for-querying-large-json-and-yaml-files/04-02-SUMMARY.md`
</output>
