# Portfolio Scoring Redesign Notes

> **Source conversations:**
> - [Portfolio Scoring Review](fa0cfdf7-2229-4a76-a910-5105a70af802) -- initial review
> - [APAC Validation & MCP Bypass Fix](fa0cfdf7-2229-4a76-a910-5105a70af802) -- test validation
> **Date:** 2026-03-15

---

## Context

Review of the AI instructions and scoring rules used by `get_portfolio_data`
for assessing business opportunities, risks, upsell, cross-sell, and renewal
signals across the CloudSense customer portfolio.

The current scoring system covers four dimensions (Renewal Risk, Version
Currency, Seat Utilization, Product Breadth) with an Opportunity Detection
table for five signal types. This document captures the gaps found, validated
against a real APAC portfolio analysis test, and proposed improvements.

### Validated Against Real Test

The APAC test (`c8b5a764` in TCS workspace) asked: *"do i have any business
opportunity in apac for cloudsense customers"*. Findings:
- **Raw data accuracy: EXCELLENT** -- zero numerical hallucinations across
  20 APAC customers, seat utilization %, expiry days, package counts all correct
- **Interpretive gaps found** -- cross-sell errors, missed renewals, no
  scoring guidance (because AI bypassed MCP tools)

---

## What's Working Well (Keep As-Is)

- **Renewal Risk** -- clear thresholds, well-reasoned lead times, highest weight (0.30)
- **Seat Utilization** -- non-linear scoring catches both over- and under-usage
- **Historical Trends** -- snapshot diffing detects new/lost customers, version changes, package shifts
- **What-If Scenarios** -- well documented with concrete examples
- **Report Personas** -- Executive, AM, Tech Lead, CS give audience-tailored framing
- **License Status Nuance** -- correctly handles mixed Active/Trial status

---

## Gap 1: Cross-Sell Detection Needs a Companion Package Matrix

**Current state:** The Opportunity Detection table says:
```
CROSS-SELL | Missing companion packages (e.g. has cscfga but not CSPOFA)
```
One example, no actual matrix. The AI has to guess which packages are companions.

**Proposed fix:** Define natural package clusters in the scoring rules:

| Cluster | Packages | Rationale |
|---------|----------|-----------|
| CPQ Core | cscfga + CSPOFA + csord | Configure-price-quote always needs orchestration and order mgmt |
| Telco Stack | csbb + cscfga + csord + csslm | Broadband customers typically need the full telco suite |
| Digital Commerce | CSPOFA + cscfga + [digital pkgs] | Digital journey orchestration |

Cross-sell trigger: customer has ≥2 packages from a cluster but is missing
one or more companions from the same cluster.

**Open question:** Do we know all the real-world companion relationships?
Should this be seeded from actual portfolio data (which clusters appear
together most often)?

---

## Gap 2: Upsell Signals Are One-Dimensional

**Current state:** Only `seat utilization > 90%` triggers UPSELL.

**Proposed additional signals:**

| Signal | Trigger | Rationale |
|--------|---------|-----------|
| Seat growth trend | Utilization increased >15 points between two snapshots | Approaching capacity before hitting 90% |
| Site license expansion | Has Site License on core pkg + no named-license add-ons | May benefit from additional named-license packages |
| Package tier upgrade | On an older/basic package variant when a premium exists | Version-based upsell (needs package tier data) |

**Dependency:** Trend-based signals require ≥2 historical snapshots.

---

## Gap 3: Revenue Data -- Intentionally Excluded

**Decision: Do NOT add revenue/ARR data.**

Reasons:
- Revenue figures are highly confidential commercial data
- Data transits through external AI provider APIs (Cursor → Claude)
- Violates the spirit of the safety rules (no customer business data)
- LMA likely doesn't store ARR on license records anyway
- Role-based access cannot be enforced by the AI

**Use proxies instead.** Add explicit guidance to scoring rules:

> For prioritisation purposes, use these as revenue proxies:
> - **Licensed seat count** -- more seats ≈ larger contract
> - **Product breadth** (package count) -- more packages ≈ deeper deal
> - **Core package presence** -- CSPOFA, cscfga, csord, csbb = strategic
> - **Corporate group size** -- parent with multiple subsidiaries = enterprise

This should be stated explicitly so the AI doesn't attempt to query or
infer revenue data.

---

## Gap 4: No Actionable Recommendations Per Signal

**Current state:** Signals are flagged but the AI gets no guidance on
what to recommend for each.

**Proposed recommendation playbook:**

| Signal | Recommended Action |
|--------|--------------------|
| UPSELL (>90% seats) | Recommend seat expansion conversation with AM. Quantify: current used/licensed ratio, projected shortfall. |
| CROSS-SELL (missing companion) | Identify which companion package(s) are missing. Suggest demo/trial for the missing package. Reference what peer customers in same industry use. |
| UPGRADE (≤R35) | Flag upgrade urgency. Mention end-of-support timelines if known. Recommend sandbox upgrade test first. Note cloud service implications. |
| AT-RISK (low util + near expiry) | Recommend executive engagement before renewal. Suggest health check to understand low utilization root cause. Propose usage workshop. |
| STRATEGIC (large + low health) | Escalate to leadership. Recommend dedicated account review. Create improvement plan with quarterly milestones. |
| RENEWAL (by tier) | CRITICAL (<90d): Immediate AM escalation. URGENT (90-180d): Schedule renewal kickoff. WATCH (180-365d): Add to quarterly planning. |

---

## Gap 5: Churn Prediction Needs Compound Signals

**Current state:** AT-RISK = `utilization < 30% + expiry < 180 days` only.

**Proposed additional churn indicators:**

| Signal | Trigger | Risk Level |
|--------|---------|------------|
| Stale customer | Same oldest_release across ≥2 snapshots (no upgrade activity) | Medium |
| Contracting | Package count decreased between snapshots (SHRINK detected but not scored) | High |
| Dormant + long tenure | <10% utilization + been a customer for 2+ years | High |
| All-Trial | Every license is Trial status with zero Active | Medium (conversion opp or churn) |

**Dependency:** Most of these require historical snapshots to compare.

---

## Gap 6: Corporate Group Analysis Is Thin

**Current state:** `_group_by_parent` aggregates oldest release and nearest
expiry, but not seats or composite scores.

**Proposed improvements:**
- Roll up composite health score at group level (weighted by package count per subsidiary)
- Compare package coverage across siblings (Subsidiary A has CSPOFA but B doesn't = sibling cross-sell)
- Flag groups where health variance is high (one subsidiary healthy, another critical = inconsistent)

**Limitation:** Can't identify subsidiaries NOT in the portfolio (that data
doesn't exist in LMA). Sibling cross-sell is limited to comparing existing
customers under the same parent.

---

## Gap 7: Cloud Service Risk Not Scored

**Current state:** `instructions.py` mentions cloud service mappings:
- CSPOFA → Orchestrator Cloud Service
- cscfga → Configurator Cloud Service
- csslm → Solution Management Cloud Service

But this isn't reflected in scoring at all.

**Proposed:** Add a note in the scoring rules that when these packages
appear in scope, the AI should flag cloud service upgrade considerations
as a secondary dimension. Not a scored dimension (we don't have cloud
service version data), but a qualitative callout.

---

## Gap 8: Industry-Specific Opportunity Patterns Undefined

**Current state:** Instructions say to use industry for "recommendations"
but never define what those recommendations are.

**Proposed minimum guidance:**

| Industry | Expected Packages | Cross-Sell Signal |
|----------|-------------------|-------------------|
| Telco/Comms | csbb, cscfga, csord, CSPOFA, csslm | Missing any of these |
| Media | cscfga, CSPOFA, csord | Missing CSPOFA or csord |
| Utilities | cscfga, csord | Missing csord |

**Open question:** Do we have enough domain knowledge to define these
accurately? May need input from product/sales teams.

---

## Gap 9: Namespace-to-Display-Name Mapping Missing (from APAC test)

**Found during validation:** The AI claimed NBN Co was "missing csord"
but NBN Co actually has "CS Order Implementation Module Telco A" and
"CloudSense Order Codebase". Same issue with PropertyGuru -- it has
"Cloud Sense Media Sales Order" but the AI said csord was missing.

**Root cause:** Scoring rules use namespace prefixes (cscfga, csord,
CSPOFA, csbb) but LMA data uses display names. No mapping exists.

**Proposed fix:** Add a mapping table to `DEFAULT_SCORING_RULES`:

```
| Namespace | Display Names (may vary)                                                 |
|-----------|--------------------------------------------------------------------------|
| cscfga    | CloudSense Configurator                                                  |
| CSPOFA    | Process Orchestration Framework                                          |
| csord     | CS Order Implementation*, CloudSense Order Codebase, Cloud Sense Media Sales Order |
| csbb      | CloudSense Basket Builder                                                |
| csslm     | CloudSense Solution Management                                           |
```

Names with * may have suffixes (e.g. "CS Order Implementation Module
Telco A"). Match by prefix, not exact name.

**Files:** `DEFAULT_SCORING_RULES` in `get_portfolio_data.py`

---

## Gap 10: RENEWAL Missing from Opportunity Detection Table (from APAC test)

**Found during validation:** The AI completely skipped renewal analysis
until the user explicitly asked. Renewal Risk is scored (highest weight
at 0.30) but there's no RENEWAL row in the Opportunity Detection table.
The AI only flagged the five listed signals.

**Proposed fix:** Add to Opportunity Detection:

```
| RENEWAL   | nearest_expiry_days < 180 (including negative = overdue) |
```

Also update `instructions.py` PORTFOLIO ANALYSIS section to mandate
renewal analysis in any business opportunity assessment.

**Files:** `DEFAULT_SCORING_RULES` in `get_portfolio_data.py`,
`instructions.py`

---

## Gap 11: Batch Expiry Pattern Not Detected (from APAC test)

**Found during validation:** 20+ customers across the portfolio share
`nearest_expiry_days: -156`. In APAC alone: Foxtel, StarHub, Telstra
Corp Limited, GES I, GES-A, Wholesale, and Sensis all show exactly
-156. The AI never flagged this suspicious pattern.

**Proposed fix:** Add a "Data Quality Signals" section:

> When multiple customers share the exact same expiry date (e.g. 5+
> at -156 days), flag this pattern. It may indicate a master agreement
> or a bulk data update in the LMA.

**Files:** `DEFAULT_SCORING_RULES` in `get_portfolio_data.py`

---

## Gap 12: Revenue Proxy Disclaimer (from APAC test)

**Found during validation:** The AI stated "Revenue impact: StarHub,
Telstra GES-A, and Momentum Energy are the highest-value upsell targets"
without clarifying this was based on seat count proxies, not actual
revenue data.

**Proposed fix:** Add explicit guidance:

> When discussing business impact, always state that seat counts and
> package breadth are used as proxies. Do NOT use "revenue impact" or
> "revenue at risk" without qualifying.

**Files:** `DEFAULT_SCORING_RULES` in `get_portfolio_data.py`

---

## To-Do List

Changes are listed in priority order. All are edits to
`DEFAULT_SCORING_RULES` in `get_portfolio_data.py` and/or
`instructions.py`. No tool logic or query changes needed.

| # | Change | Effort | Impact | Status |
|---|--------|--------|--------|--------|
| 1 | MCP bypass prevention (DO NOT rule + JSON warning) | 5 min | Critical | **DONE** (v0.7.1) |
| 2 | Seat utilization in `get_customer_licenses` (SOQL fields, parse/aggregate seats, summary table) | 30 min | High -- enables upsell/capacity analysis per customer | **DONE** (v0.8.0) |
| 3 | Trialforce org filtering (word-boundary `_is_genuine_company` check on Company__c queries) | 15 min | High -- eliminates misleading training org data | **DONE** (v0.8.1, camelCase fix v0.8.5) |
| 4 | Per-package expiry + `upcoming_expiry_days` + License Expiry Interpretation rules | 30 min | Critical -- prevents hallucinated renewals on perpetual/stale licenses | **DONE** (v0.8.3) |
| 5 | README update (version, tool descriptions, workflow) | 10 min | Medium -- keeps docs current | **DONE** (v0.8.4) |
| 6 | Namespace-to-display-name mapping (Gap 9) | 15 min | High -- prevents cross-sell hallucinations | To do |
| 7 | Add RENEWAL to Opportunity Detection (Gap 10) | 10 min | High -- prevents missed renewals | To do |
| 8 | Companion package matrix (Gap 1) | 15 min | High -- enables real cross-sell detection | To do |
| 9 | Per-signal recommendation playbook (Gap 4) | 15 min | High -- makes insights actionable | To do |
| 10 | Revenue proxy disclaimer (Gap 12) | 5 min | Medium -- prevents misleading language | To do |
| 11 | Batch expiry warning (Gap 11) | 5 min | Medium -- catches data quality issues | To do |
| 12 | Compound churn signals (Gap 5) | 10 min | Medium -- better risk detection | To do |
| 13 | Trend-based upsell signals (Gap 2) | 10 min | Medium -- catches early upsell | To do |
| 14 | Corporate group rollup (Gap 6) | 20 min | Medium -- better enterprise view | To do |
| 15 | Industry-specific patterns (Gap 8) | 15 min | Low-Medium -- needs domain validation | To do |
| 16 | Cloud service qualitative callout (Gap 7) | 5 min | Low -- informational only | To do |

---

## Decision Log

| Date | Decision | Rationale |
|------|----------|-----------|
| 2026-03-15 | Do NOT include revenue/ARR data | Security risk -- data transits external AI APIs, violates business data safety rules |
| 2026-03-15 | Use package count + seat count as revenue proxies | Available in LMA, no additional queries, good enough for prioritisation |
| 2026-03-15 | All changes go in scoring rules + instructions only | No tool logic or query changes needed -- keeps implementation simple |
| 2026-03-15 | Prevent MCP bypass with DO NOT rule + JSON warning | AI was reading portfolio-data.json from disk, missing scoring rules entirely (v0.7.1) |
| 2026-03-15 | Add seat utilization to `get_customer_licenses` | Customer-level tool was missing Licensed_Seats and Used_Licenses fields, blocking upsell/capacity analysis (v0.8.0) |
| 2026-03-15 | Filter trialforce orgs via word-boundary check | Company__c LIKE '%keyword%' was pulling in training orgs (e.g. StudentR36Sep25Starhub1), inflating org counts. Word-boundary regex is more robust than starts-with (v0.8.1) |
| 2026-03-15 | Replace `nearest_expiry_days` with `upcoming_expiry_days` for renewal scoring | `nearest_expiry_days` includes long-expired packages (e.g. -3452 days) masking real upcoming renewals. `upcoming_expiry_days` tracks nearest future expiry only (v0.8.3) |
| 2026-03-15 | Add License Expiry Interpretation rules | Prevents AI from hallucinating renewal opportunities on perpetual licenses or flagging stale trials as churn (v0.8.3) |
| 2026-03-15 | Add camelCase fallback to `_is_genuine_company` | Word-boundary regex `\bNBN\b` failed on `Company__c = "NBNCo"` (no space), silently dropping all 514 active sandbox licenses for NBN Co. CamelCase normalization + `^` anchor fixes this while still rejecting trialforce identifiers (v0.8.5) |
