# v1.0.7 Implementation Plan - Smart Query Enhancements

**Date:** December 15, 2025
**Based on:** Kiro's production testing feedback
**Scope:** Critical bug fixes and intelligent enhancements

---

## Overview

Based on real-world testing by Kiro, we've identified critical issues and opportunities to make smart_query production-ready. This release focuses on correct results, intelligent routing, and better user experience.

---

## Critical Fixes

### 1. Aggregation Fallback Bug (CRITICAL)

**Issue:** When analytical endpoint fails for aggregation queries, fallback to relational returns raw data instead of performing aggregation.

**Example Query:**
```sql
SELECT PRODUCTCATEGORYID, COUNT(*) as ProductCount, AVG(PRICE) as AvgPrice
FROM SAP_SC_FI_V_ProductsDim
GROUP BY PRODUCTCATEGORYID
```

**Current Behavior:**
- Analytical fails → Falls back to relational
- Returns all 35 raw rows (incorrect)

**Expected Behavior:**
- Analytical fails → Try execute_query (SQL parser)
- If that fails → Perform client-side aggregation on relational data
- If not feasible → Return clear error explaining limitation

**Implementation Strategy:**
1. Detect aggregation functions: `COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `GROUP BY`
2. When fallback to relational is triggered for aggregation query:
   - Option A: Try execute_query method first (has SQL parsing)
   - Option B: Implement client-side aggregation using pandas/python
   - Option C: Return error: "Aggregations require analytical endpoint. Table X does not support analytical queries."

**Chosen Approach:** Multi-tier fallback
- Primary: Analytical endpoint
- Fallback 1: execute_query (SQL parsing might work)
- Fallback 2: Client-side aggregation (if simple GROUP BY)
- Fallback 3: Clear error message with suggestions

---

## High-Priority Enhancements

### 2. Asset Capability Detection

**Current:** Routes queries without checking if asset supports analytical queries
**Problem:** Wastes time trying analytical endpoints on relational-only assets

**Solution:**
```python
# Before routing, check asset capabilities
asset_info = await search_tables(table_name)
supports_analytical = asset_info.get('supportsAnalyticalQueries', False)

if has_aggregations and not supports_analytical:
    execution_log.append(f"⚠️ Asset {table_name} does not support analytical queries")
    execution_log.append("→ Aggregation queries not possible on this asset")
    # Skip analytical routing, go straight to error or client-side aggregation
```

**Benefits:**
- Faster query execution (skip doomed attempts)
- Better error messages
- Smarter routing decisions

---

### 3. Fuzzy Table Name Matching

**Current:** "Table not found" errors with no suggestions
**Problem:** Users must know exact table names (case-sensitive)

**Solution:**
```python
if table_not_found:
    # Search for similar names
    search_results = await search_tables(partial_table_name)
    suggestions = [asset['name'] for asset in search_results[:5]]

    return {
        "error": f"Table '{table_name}' not found",
        "suggestions": suggestions,
        "tip": "Table names are case-sensitive. Try one of the suggestions above."
    }
```

**Examples:**
- User enters: `products` → Suggests: `SAP_SC_FI_V_ProductsDim`, `SAP_SC_FI_SQL_Products`
- User enters: `sales_orders` → Suggests: `SAP_SC_SALES_V_SalesOrders`

---

### 4. Enhanced Error Messages

**Current:** Generic API errors
**Problem:** Users don't know what to do next

**Solution:** Context-aware errors with next steps

**Before:**
```
Error: HTTP 404 - Endpoint not found
```

**After:**
```
Query Failed - Table Not Found

Table: SAP_SC_FI_V_ProductsDim
Space: SAP_CONTENT
Method Attempted: query_relational_entity

Possible Causes:
1. Table name is case-sensitive (try exact capitalization)
2. Table doesn't exist in this space
3. You may not have permissions to access this table

Next Steps:
✓ Use search_tables("products") to find exact table names
✓ Use list_catalog_assets(space_id="SAP_CONTENT") to see all available tables
✓ Check table permissions with get_asset_details()

Similar Tables Found:
- SAP_SC_FI_SQL_ProductHierarchy
- SAP_SC_FI_SQL_ProductTexts

Try This Query:
SELECT * FROM SAP_SC_FI_SQL_ProductHierarchy LIMIT 5
```

---

### 5. Query Optimization - LIMIT Pushdown

**Current:** Fetches all data then applies Python limit
**Problem:** Slow for large datasets

**Solution:**
```python
# Extract LIMIT from SQL
limit_match = re.search(r'LIMIT\s+(\d+)', query, re.IGNORECASE)
if limit_match:
    sql_limit = int(limit_match.group(1))
    api_limit = min(sql_limit, limit)  # Use smaller of SQL LIMIT or parameter
    params["$top"] = api_limit
```

**Benefits:**
- Faster queries (less data transfer)
- Lower API load
- Better performance

---

## Implementation Details

### Client-Side Aggregation Logic

For simple aggregations when analytical fails:

```python
def perform_client_side_aggregation(data, query):
    """
    Perform aggregation on raw data for simple GROUP BY queries
    Supports: COUNT, SUM, AVG, MIN, MAX
    """
    import re
    from collections import defaultdict

    # Extract aggregation details from SQL
    group_by_match = re.search(r'GROUP BY\s+(\w+)', query, re.IGNORECASE)
    if not group_by_match:
        return None  # Can't do client-side without GROUP BY

    group_column = group_by_match.group(1)

    # Parse SELECT clause for aggregations
    select_match = re.search(r'SELECT\s+(.+?)\s+FROM', query, re.IGNORECASE)
    if not select_match:
        return None

    select_clause = select_match.group(1)

    # Find aggregation functions
    agg_functions = re.findall(
        r'(COUNT|SUM|AVG|MIN|MAX)\s*\(\s*([*\w]+)\s*\)\s*(?:as\s+(\w+))?',
        select_clause,
        re.IGNORECASE
    )

    # Group data
    groups = defaultdict(list)
    for row in data:
        key = row.get(group_column)
        groups[key].append(row)

    # Perform aggregations
    results = []
    for key, rows in groups.items():
        result = {group_column: key}

        for func, column, alias in agg_functions:
            func = func.upper()
            alias = alias or f"{func}_{column}"

            if func == "COUNT":
                result[alias] = len(rows)
            elif func == "SUM":
                result[alias] = sum(row.get(column, 0) for row in rows)
            elif func == "AVG":
                values = [row.get(column, 0) for row in rows]
                result[alias] = sum(values) / len(values) if values else 0
            elif func == "MIN":
                result[alias] = min(row.get(column, float('inf')) for row in rows)
            elif func == "MAX":
                result[alias] = max(row.get(column, float('-inf')) for row in rows)

        results.append(result)

    return results
```

---

## Testing Strategy

### Test Cases

1. **Aggregation on Analytical Asset** (should work)
   ```sql
   SELECT COMPANYNAME, SUM(GROSSAMOUNT)
   FROM SAP_SC_SALES_AM_SalesOrders
   GROUP BY COMPANYNAME
   ```

2. **Aggregation on Relational-Only Asset** (should trigger client-side or error)
   ```sql
   SELECT PRODUCTCATEGORYID, COUNT(*)
   FROM SAP_SC_FI_V_ProductsDim
   GROUP BY PRODUCTCATEGORYID
   ```

3. **Non-existent Table** (should suggest alternatives)
   ```sql
   SELECT * FROM products LIMIT 5
   ```

4. **Complex Aggregation** (should explain limitations)
   ```sql
   SELECT CATEGORY, AVG(PRICE), COUNT(*), MIN(PRICE), MAX(PRICE)
   FROM SAP_SC_FI_V_ProductsDim
   GROUP BY CATEGORY
   HAVING AVG(PRICE) > 1000
   ```

5. **LIMIT Optimization**
   ```sql
   SELECT * FROM SAP_SC_FI_V_ProductsDim LIMIT 5
   ```
   Should use `$top=5` not fetch all then limit

---

## Success Criteria

- ✅ Aggregation queries return correct aggregated results (not raw data)
- ✅ Asset capabilities checked before routing
- ✅ Table name suggestions provided for typos
- ✅ Error messages include next steps and examples
- ✅ LIMIT clauses pushed to API level
- ✅ All existing functionality preserved (backward compatible)
- ✅ Execution logs clearly show routing decisions

---

## Files to Modify

1. **sap_datasphere_mcp_server.py** - smart_query handler (lines 2246-2503)
2. **tool_descriptions.py** - Update smart_query description with new features
3. **pyproject.toml** - Version 1.0.7
4. **CHANGELOG_v1.0.7.md** - New changelog

---

## Rollout Plan

1. Implement changes locally
2. Test with Kiro's example queries
3. Commit to git
4. Build package
5. Publish to PyPI
6. Create GitHub release
7. Notify Kiro for testing

---

## Risk Assessment

**Low Risk:**
- Fuzzy matching (additive feature)
- Error message improvements (cosmetic)
- LIMIT optimization (API-compatible)

**Medium Risk:**
- Asset capability detection (new API calls)
- Client-side aggregation (new code path)

**Mitigation:**
- Extensive testing with known queries
- Fallback logic for all new features
- Detailed execution logs for debugging

---

## Timeline

- **Implementation:** 2-3 hours
- **Testing:** 1 hour
- **Documentation:** 30 minutes
- **Release:** 30 minutes
- **Total:** ~4-5 hours

---

**Status:** Ready to implement
**Assigned:** Claude
**Priority:** High (Critical bug + Production readiness)
