# v1.0.3 Implementation Plan
## Adding 2 Competitive Advantage Tools

**Date:** 2025-12-13
**Objective:** Add `find_assets_by_column` and `analyze_column_distribution` tools
**Target:** 44 total tools (from 42)
**Estimated Effort:** 6-8 hours

---

## Tools to Implement

### 1. find_assets_by_column

**Purpose:** Find all assets (tables/views) containing a specific column name across spaces

**Value Proposition:**
- Data lineage discovery (find all tables with `CUSTOMER_ID`)
- Impact analysis before schema changes
- Locate datasets for specific use cases
- Competitive advantage over rahulsethi/SAPDatasphereMCP

**Parameters:**
```python
{
    "column_name": str,  # Required - column to search for
    "space_id": str | None,  # Optional - limit to one space
    "max_assets": int = 50,  # Optional - limit results
    "asset_types": list[str] = [],  # Optional - filter by View, Table, etc.
    "case_sensitive": bool = False  # Optional - case-sensitive search
}
```

**Implementation Strategy:**
1. Use existing `discover_catalog` or catalog API to get all assets
2. For each asset, use existing `get_table_schema` to get column list
3. Filter assets where column_name matches (case-insensitive by default)
4. Return structured results with metadata

**APIs to Use:**
- `/api/v1/datasphere/consumption/catalog/spaces` - List spaces
- `/api/v1/datasphere/consumption/catalog/spaces/{spaceId}/assets` - List assets
- Existing `get_table_schema` logic for column discovery

**Response Structure:**
```json
{
  "column_name": "CUSTOMER_ID",
  "case_sensitive": false,
  "search_scope": {
    "spaces_searched": 2,
    "assets_checked": 45,
    "assets_with_schema": 38
  },
  "matches": [
    {
      "space_id": "SAP_CONTENT",
      "asset_name": "CUSTOMER_DATA",
      "asset_type": "View",
      "column_name": "CUSTOMER_ID",
      "column_type": "NVARCHAR(10)",
      "column_position": 1,
      "total_columns": 15,
      "consumption_url": "/api/v1/datasphere/consumption/..."
    }
  ],
  "execution_time_seconds": 2.3
}
```

**Authorization Level:** READ (catalog discovery)

---

### 2. analyze_column_distribution

**Purpose:** Statistical analysis of a specific column's data distribution

**Value Proposition:**
- Data quality assessment (null rates, outliers)
- Statistical profiling for ML/analytics
- Understand data patterns before modeling
- Unique competitive advantage

**Parameters:**
```python
{
    "space_id": str,  # Required
    "asset_name": str,  # Required
    "column_name": str,  # Required
    "sample_size": int = 1000,  # Optional - records to analyze
    "include_outliers": bool = True,  # Optional - detect outliers
    "include_distribution": bool = True  # Optional - frequency analysis
}
```

**Implementation Strategy:**
1. Use existing `execute_query` tool for data sampling
2. Execute SQL queries to get statistics:
   - `SELECT COUNT(*), COUNT(DISTINCT column), MIN(column), MAX(column), AVG(column) FROM asset`
   - `SELECT column, COUNT(*) FROM asset GROUP BY column ORDER BY COUNT(*) DESC LIMIT 20`
3. Calculate client-side:
   - Null percentage
   - Distinct value count
   - Basic percentiles (if numeric)
   - Top values distribution
4. For numeric columns:
   - Calculate IQR for outlier detection
   - Provide distribution bins

**SQL Queries Pattern:**
```sql
-- Basic stats
SELECT
    COUNT(*) as total_count,
    COUNT(CUSTOMER_ID) as non_null_count,
    COUNT(DISTINCT CUSTOMER_ID) as distinct_count,
    MIN(CUSTOMER_ID) as min_value,
    MAX(CUSTOMER_ID) as max_value
FROM SALES_DATA
LIMIT {sample_size};

-- Top values (if distinct count < 100)
SELECT
    CUSTOMER_ID as value,
    COUNT(*) as frequency
FROM SALES_DATA
GROUP BY CUSTOMER_ID
ORDER BY frequency DESC
LIMIT 20;
```

**Response Structure:**
```json
{
  "column_name": "AMOUNT",
  "column_type": "DECIMAL(18,2)",
  "sample_analysis": {
    "rows_sampled": 1000,
    "sampling_method": "top_n"
  },
  "basic_stats": {
    "count": 1000,
    "null_count": 5,
    "null_percentage": 0.5,
    "completeness_rate": 99.5,
    "distinct_count": 847,
    "cardinality": "high"
  },
  "numeric_stats": {
    "min": 10.50,
    "max": 99999.99,
    "mean": 5234.67,
    "percentiles": {
      "p25": 1000.00,
      "p50": 3500.00,
      "p75": 7500.00
    }
  },
  "distribution": {
    "top_values": [
      {"value": "100.00", "frequency": 45, "percentage": 4.5},
      {"value": "250.00", "frequency": 38, "percentage": 3.8}
    ],
    "unique_values_sample": 20
  },
  "outliers": {
    "method": "IQR",
    "outlier_count": 12,
    "outlier_percentage": 1.2,
    "examples": [99999.99, 95000.00]
  },
  "data_quality": {
    "completeness": "excellent",
    "cardinality_level": "high",
    "potential_issues": []
  }
}
```

**Authorization Level:** READ (data sampling)

---

## Implementation Checklist

### Phase 1: Tool Descriptions (1-2 hours)

- [ ] Add `find_assets_by_column` to tool_descriptions.py
- [ ] Add `analyze_column_distribution` to tool_descriptions.py
- [ ] Define input schemas with proper validation
- [ ] Define expected output schemas

### Phase 2: Authorization & Validation (30 min)

- [ ] Add authorization rules to auth/authorization.py
  - `find_assets_by_column`: AuthorizationLevel.READ
  - `analyze_column_distribution`: AuthorizationLevel.READ
- [ ] Add validation rules to auth/tool_validators.py
  - column_name: STRING, required, max 100 chars
  - space_id: STRING, optional/required, max 50 chars
  - asset_name: STRING, required, max 100 chars
  - sample_size: INTEGER, min 10, max 10000

### Phase 3: Implementation (3-4 hours)

#### find_assets_by_column
- [ ] Implement handler in sap_datasphere_mcp_server.py
- [ ] Add mock data support (MOCK_DATA extension)
- [ ] Use existing catalog APIs for space/asset discovery
- [ ] Use existing get_table_schema logic for column matching
- [ ] Add caching for performance
- [ ] Add progress tracking for large searches
- [ ] Comprehensive error handling

#### analyze_column_distribution
- [ ] Implement handler in sap_datasphere_mcp_server.py
- [ ] Add mock data support
- [ ] Use existing execute_query for SQL execution
- [ ] Implement statistical calculations:
  - Basic stats (count, null, distinct)
  - Numeric stats (min, max, mean, percentiles)
  - Distribution analysis (top values)
  - Outlier detection (IQR method)
- [ ] Handle different column types (numeric, string, date)
- [ ] Add execution timeout handling
- [ ] Comprehensive error handling

### Phase 4: Testing (1-2 hours)

- [ ] Test `find_assets_by_column`:
  - Single space search
  - Multi-space search
  - Case-sensitive vs insensitive
  - Asset type filtering
  - Empty results handling
- [ ] Test `analyze_column_distribution`:
  - Numeric columns
  - String columns
  - Columns with nulls
  - Small vs large sample sizes
  - Outlier detection accuracy

### Phase 5: Documentation (1 hour)

- [ ] Update README.md with new tool count (44 tools)
- [ ] Update TOOLS_CATALOG.md with detailed specs
- [ ] Create CHANGELOG_v1.0.3.md
- [ ] Update competitive advantage messaging (44 vs 11 = 300%)

### Phase 6: Release (30 min)

- [ ] Bump version to 1.0.3
- [ ] Build package
- [ ] Publish to PyPI
- [ ] Create GitHub release (optional)

---

## File Changes Required

### 1. tool_descriptions.py
- Add 2 new tool definitions (~150 lines)
- Insert after `browse_marketplace` tool

### 2. sap_datasphere_mcp_server.py
- Add 2 new handlers (~200 lines total)
- Insert after `browse_marketplace` handler

### 3. auth/authorization.py
- Add 2 tool authorizations (~4 lines)

### 4. auth/tool_validators.py
- Add 2 validation rule sets (~30 lines)

### 5. mock_data.py (if needed)
- Add sample data for testing (~20 lines)

### 6. README.md
- Update tool count: 42 → 44
- Update competitive advantage: 280% → 300%
- Add "What's New in v1.0.3" section

### 7. TOOLS_CATALOG.md
- Add 2 new tool entries with examples

### 8. CHANGELOG_v1.0.3.md (new file)
- Document new tools and competitive positioning

### 9. pyproject.toml & setup.py
- Bump version: 1.0.2 → 1.0.3

---

## Risk Assessment

**Risk Level:** MEDIUM

**Key Risks:**
1. **Performance** - Column search across many assets might be slow
   - Mitigation: Add caching, limit max_assets, use batching

2. **API Limitations** - Some catalog APIs might not exist
   - Mitigation: Fallback to existing discover_catalog logic

3. **SQL Query Complexity** - Statistical queries might timeout on large datasets
   - Mitigation: Use LIMIT clauses, configurable sample_size

**Confidence Level:** HIGH - Using proven existing APIs and patterns

---

## Success Criteria

✅ Both tools work with real SAP Datasphere data
✅ Both tools have comprehensive error handling
✅ Tests pass for common use cases
✅ Documentation updated
✅ Version 1.0.3 published to PyPI
✅ Competitive advantage maintained (300%+ more tools)

---

## Next Steps

1. Review this plan
2. Confirm approach
3. Start implementation in next session
4. Complete in 1-2 sessions (~6-8 hours total)

---

**Ready to proceed?** This will give us 44 production-quality tools vs. competitor's 11 proof-of-concept tools = **300% competitive advantage**.
