# Ossuary Scoring Methodology

This document describes the methodology used by Ossuary to assess governance-based supply chain risk in open source packages.

## Executive Summary

Ossuary calculates a risk score (0-100) based on observable governance signals in public package metadata. The methodology focuses on detecting **governance failures** - conditions that historically precede supply chain attacks like maintainer abandonment, frustration-driven sabotage, or social engineering takeovers.

**Key Finding**: In validation testing against 164 packages across 8 ecosystems using a scoped evaluation framework, the methodology achieved **96.0% precision** (1 false positive) and **77.4% in-scope recall** (F1 0.857). Incidents are classified by detectability tier — only those where governance weakness was observable before the attack count toward recall. Out-of-scope incidents (credential theft on healthy projects, CI/CD exploits) are included in the dataset to validate detection boundaries but are not penalized as false negatives.

**Version**: 6.0 (March 2026)
**Validation Dataset**: 164 packages across npm, PyPI, Cargo, RubyGems, Packagist, NuGet, Go, and GitHub

---

## 1. Problem Statement

### 1.1 The Governance Gap

Modern software relies heavily on open source dependencies. A typical application may have hundreds of transitive dependencies, many maintained by individuals or small teams. When these maintainers become unavailable, frustrated, or compromised, the entire dependency chain is at risk.

### 1.2 Historical Incidents

| Incident | Year | Attack Vector | Governance Signal |
|----------|------|---------------|-------------------|
| event-stream | 2018 | Maintainer handoff to stranger | Abandoned, 75% concentration, "free work" frustration |
| colors/faker | 2022 | Intentional sabotage | 100% concentration, "protest", "exploitation" keywords |
| xz-utils | 2024 | 2.6-year social engineering | Sole maintainer, proportion shift detectable 12 months early |
| left-pad | 2016 | Registry removal | Single maintainer, mass breakage |
| ctx | 2022 | Expired domain takeover | Abandoned, bus_factor=1, years of inactivity |
| polyfill.io | 2024 | Project sold to malicious CDN | Ownership transfer without safeguards |
| ua-parser-js | 2021 | Account compromise | *Not governance-detectable* — active project, healthy metrics |
| chalk (2025) | 2025 | Maintainer phished | *Not governance-detectable* — strong governance, credential theft |
| tj-actions | 2025 | CI/CD cascade exploit | *Not governance-detectable* — CI trust chain, not governance |

### 1.3 Research Question

> Can publicly observable metadata predict which packages are vulnerable to governance-based attacks **before** incidents occur?

---

## 2. Related Work

This section reviews prior academic research relevant to software supply chain security and open source sustainability.

### 2.1 Software Supply Chain Security

The field has seen significant academic attention following high-profile incidents:

**Taxonomies and Systematization**

Ladisa et al. (2023) presented "SoK: Taxonomy of Attacks on Open-Source Software Supply Chains" at IEEE S&P, identifying **12 distinct attack categories** independent of specific languages or ecosystems. This taxonomy provides the foundation for understanding governance-based attacks as a distinct category.

Ohm et al. (2020) published "Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks" at DIMVA, cataloging real-world attacks and establishing patterns that inform detection approaches.

**Scale and Impact**

Research quantifies the growing threat: Sonatype recorded a **700% increase** in supply chain attacks (ICSE '23), while 97% of applications now use open source components with 78% of code originating from OSS (Synopsys 2022).

Torres-Arias et al. (2019) introduced in-toto, a framework for software supply chain integrity, and Lamb & Zacchiroli (2021) addressed "Reproducible Builds" in IEEE Software, focusing on build integrity rather than governance.

**Risk Assessment Frameworks**

The ACM TOSEM paper "Research Directions in Software Supply Chain Security" (2024) identifies three major attack vectors: vulnerabilities in dependencies, infiltration of build infrastructure, and **social engineering** - the last being most relevant to governance-based risks.

### 2.2 Open Source Sustainability and Maintainer Health

**Maintainer Burnout**

Raman et al. (2020) published "Stress and Burnout in Open Source" at ICSE-NIER, finding that toxic conversations demotivate and burn out developers. They developed an SVM classifier to detect toxic discussions - an approach that influenced Ossuary's sentiment analysis component.

Guo et al. (2024) in "Sustaining Maintenance Labor for Healthy Open Source Software Projects" (arXiv) argue that **depleted maintainer capacity** leads to unmaintained projects with security consequences. This directly supports Ossuary's activity modifier component.

**The Bus Factor Problem**

Eghbal's foundational work "Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure" (Ford Foundation, 2016) found that the **majority of open source projects are maintained by one or two people** - validating maintainer concentration as a key risk signal.

The CHAOSS project formalized this as the "Contributor Absence Factor" metric, which Ossuary operationalizes in its concentration scoring.

### 2.3 The xz-utils Case Study

The xz-utils backdoor (CVE-2024-3094) represents the most sophisticated governance attack documented. Academic analysis (arXiv 2504.17473) details:

- **2.6-year attack timeline** (2021-2024)
- **Phased social engineering**: trust-building → maintainer duties → infrastructure control → code injection
- **Exploitation of maintainer burnout**: The sole maintainer cited "long-term mental health issues" before ceding control

This incident validates Ossuary's approach: the attacker specifically targeted a project with high concentration and a burned-out maintainer - both signals Ossuary detects.

**Proportion Shift Detection (v4.0)**: Ossuary's takeover detection compares each contributor's historical commit share against their recent (12-month) share. Applied to xz-utils, this detects Jia Tan by **March 2023** — a full 12 months before the backdoor was discovered in March 2024. Jia Tan's proportion shift: 0.8% historical → 50% recent = **+49.5 percentage point shift**, far exceeding the 30pp detection threshold. See Section 4.4 for technical details.

### 2.4 Existing Tools and Gap Analysis

| Tool | Focus | Gap Addressed by Ossuary |
|------|-------|-------------------------|
| **OpenSSF Scorecard** | Security best practices | Doesn't assess governance/abandonment risk |
| **CHAOSS/Augur** | Community health metrics | No actionable risk score or sentiment analysis |
| **Snyk/Dependabot** | Known vulnerabilities | Reactive to CVEs, not predictive |
| **Socket.dev** | Behavioral analysis | Detects malicious code, not governance risk |
| **deps.dev** | Dependency metadata | Informational, no risk scoring |
| **LFX Insights** | Project criticality ranking | Identifies important projects but not governance risk; complementary as prioritization input |
| **OSSInsight** | GitHub analytics at scale | Raw activity dashboards (stars, commits, PRs) but no risk scoring or governance assessment |

**The Gap**: No existing tool combines maintainer concentration, activity patterns, and frustration signals into a predictive governance risk score validated against historical incidents.

### 2.5 Academic Contribution

Ossuary contributes to this body of research by:

1. **Operationalizing** CHAOSS metrics into an actionable risk score
2. **Adding sentiment analysis** for frustration/burnout detection (extending Raman et al.)
3. **Validating predictively** against real incidents (T-1 analysis)
4. **Achieving 96.0% precision** with 1 false positive across 164 packages (v6.2)
5. **Detecting social engineering takeovers** via proportion shift analysis, validated against the xz-utils timeline (12-month early detection)
6. **Explicitly validating detection boundaries** — including out-of-scope attack types in the validation set to empirically demonstrate what governance scoring can and cannot detect

---

## 3. Detection Scope

### 3.1 What Ossuary Detects

| Signal | Description | Example |
|--------|-------------|---------|
| **Maintainer Abandonment** | Single maintainer with declining activity | event-stream pre-2018 |
| **High Concentration Risk** | >90% commits from one person | minimist, rimraf |
| **Economic Frustration** | Burnout/resentment signals in communications | colors pre-2022 |
| **Governance Centralization** | No succession plan, single point of failure | husky |
| **Newcomer Takeover** | Unknown contributor suddenly dominates a mature project | xz-utils/Jia Tan |

### 3.2 What Ossuary Cannot Detect

| Attack Type | Why Undetectable | Examples | Validation Cases |
|-------------|------------------|----------|-----------------|
| **Account Compromise** | Active project, healthy governance metrics | ua-parser-js, chalk (2025), solana-web3.js, num2words | 8 cases (T4), all expected FN |
| **CI/CD Pipeline Exploits** | Workflow misconfigurations, not governance | tj-actions, reviewdog, rspack, ultralytics, Nx | 6 cases (T5), all expected FN |
| **Protestware by reputable maintainer** | Reputation correctly reduces risk score | es5-ext, is-promise | Detected by scope but missed by threshold (T2 FN) |
| **Typosquatting** | New package, no governance to analyze | crossenv, boltdb-go/bolt | Not tested (no repo to score) |
| **Dependency Confusion** | Build system attack, not governance | PyTorch-nightly | Not tested |

These are classified as **expected false negatives** — the methodology explicitly does not attempt to detect them. The validation set includes 14 out-of-scope cases (T4+T5) to empirically confirm the detection boundary (see §8.6).

### 3.3 The Detection Boundary

The key insight from validation is that governance scoring and credential/CI/CD-based detection are **complementary, not competing** approaches:

- **Governance scoring detects**: Conditions that make attacks possible (abandonment, concentration, frustration)
- **Credential/CI/CD detection requires**: Runtime analysis, provenance attestation, workflow auditing

A well-governed project can still be compromised via phishing (chalk 2025, solana-web3.js) or CI/CD exploits (tj-actions, Nx). Conversely, a poorly-governed project might never be attacked. The two dimensions are orthogonal — organizations should assess both.

### 3.4 Case Study: The 2025 npm Phishing Wave

The 2025 npm supply chain attacks represent the most significant series of package ecosystem compromises to date, affecting over 2.6 billion weekly downloads across multiple interconnected attack waves. Three packages from our validation set — `chalk`, `eslint-config-prettier`, and `is` — were compromised in this campaign, and their contrasting Ossuary scores illustrate the detection boundary precisely.

#### Timeline

| Wave | Date | Vector | Scale |
|------|------|--------|-------|
| eslint-config-prettier | July 17–19 | Phishing via typosquatted `npnjs.com` | 5 packages, 78M weekly downloads |
| `is` package | July 19 | Phished inactive maintainer + social engineering | 1 package, 6 hours undetected |
| s1ngularity / Nx | August 21–31 | GitHub Actions injection → token theft | 500+ packages, 2,349 credentials harvested |
| chalk / "Great Heist" | September 8 | Phishing via `npmjs.help` | 18+ packages, 2.6B weekly downloads |
| Shai-Hulud 1.0 | September | Self-replicating worm using stolen tokens | 500+ packages |
| Shai-Hulud 2.0 | November 21–24 | Unrotated tokens from prior waves | 796 packages, 14,000 secrets exposed |

The waves were causally linked: credentials stolen in the s1ngularity attack (August) were never rotated, directly enabling the Shai-Hulud worm (September–November). CISA issued alert AA25-266A on September 23, and CERT/CC published VU#534320 identifying "longstanding design weaknesses in npm's architecture."

#### Three Packages, Three Outcomes

| Package | Ossuary | Scorecard | Classification | Why |
|---------|---------|-----------|----------------|-----|
| **`is`** | **100 CRITICAL** | 3.4 | **True Positive** | Single inactive maintainer (100% concentration, 0 commits/year), no protective factors |
| **eslint-config-prettier** | 35 LOW | 4.5 | Expected FN | Prettier org, active development, 32 commits/year, multiple contributors |
| **chalk** | 20 LOW | 3.8 | Expected FN | Sindre Sorhus (Tier 1 reputation), established project, strong protective factors (−60) |

All three were compromised by the same attack class (credential phishing), yet Ossuary correctly scored them on opposite sides of the risk threshold. This is not a bug — it reflects the fundamental distinction between **governance vulnerability** and **attack occurrence**:

**`is`** — The `is` package had a single maintainer (enricomarino) who had not committed in years, with 100% contributor concentration and no organizational backing. Ossuary scored it 100 CRITICAL. When the attacker phished this inactive maintainer's npm credentials and then social-engineered the current team into re-granting publish access, the governance weakness was the enabling condition: a dormant account with live publish rights on a package with no review process for ownership changes.

**chalk** — chalk is maintained by Sindre Sorhus, one of npm's most prolific contributors (Tier 1 reputation in Ossuary's system), with an active contributor base and organizational backing. Ossuary scored it 20 LOW. The attacker phished a co-maintainer (Qix-) via a fake `npmjs.help` domain and published malicious versions that intercepted cryptocurrency wallet transactions. The attack succeeded not because of governance weakness but because npm's authentication infrastructure allowed phishable TOTP-based MFA and long-lived publish tokens.

**eslint-config-prettier** — Part of the Prettier organization with active development (32 commits/year). Ossuary scored it 35 LOW. Maintainer JounQin was phished via the typosquatted `npnjs.com` domain, and malicious versions delivered a Windows RAT via disguised postinstall scripts. Again, the attack vector was credential theft against a well-governed project.

#### The Complementarity Argument

The 2025 npm wave demonstrates that governance scoring and credential/infrastructure hardening are **complementary layers**, not alternative approaches:

| Defense Layer | Would Prevent | 2025 Example |
|---------------|---------------|--------------|
| **Governance: Mandatory phishing-resistant MFA** | Credential phishing entirely | `npmjs.help` and `npnjs.com` attacks used phishable TOTP codes; FIDO/WebAuthn keys cannot be phished |
| **Governance: Short-lived scoped tokens** | Cross-wave credential reuse | s1ngularity tokens (August) enabled Shai-Hulud (September–November) because they were never rotated |
| **Governance: Publish review for dormant accounts** | `is`-style social engineering | Inactive maintainer regained publish access via email claim |
| **Governance scoring (Ossuary)** | Identifying high-risk targets | `is` scored 100 CRITICAL — the governance weakness that made it a soft target was detectable months before the attack |
| **Detection: Anomalous publish monitoring** | Rapid exploitation | Aikido detected chalk compromise in 5 minutes; 2.5M downloads still occurred in the 2-hour remediation window |
| **Detection: Credential scanning** | Token reuse chains | The 2,349 tokens leaked by s1ngularity were publicly visible on GitHub; automated scanning would have flagged them |

The critical insight for the thesis: **Ossuary cannot prevent the chalk attack** (score 20, correctly assessed as low governance risk), **but it can identify packages like `is` where governance weakness creates the preconditions for exactly this type of attack** (score 100). A security team using both Ossuary and credential monitoring would have:

1. **Pre-attack**: Flagged `is` as CRITICAL risk (governance scoring), prioritized it for publish-access audit
2. **During attack**: Detected anomalous publish from dormant account (credential monitoring)
3. **Post-attack**: Known that chalk's governance was healthy (score 20) and focused remediation on the credential vector, not governance restructuring

Neither tool alone provides this complete picture. The npm phishing wave is empirical evidence that the two dimensions — governance risk and credential/infrastructure security — require distinct measurement approaches used in concert.

#### References

- CISA Alert AA25-266A: "Widespread Supply Chain Compromise Impacting npm Ecosystem" (September 23, 2025)
- CERT/CC VU#534320: npm architecture design weaknesses
- Sonatype: "npm chalk and debug packages hit in software supply chain attack" (September 2025)
- Aikido Security: Detection timeline for the September 8 chalk compromise
- StepSecurity: "Another npm Supply Chain Attack — The 'is' Package Compromise" (July 2025)
- SafeDep: "eslint-config-prettier: Major npm Supply Chain Hack" (July 2025)

---

## 4. Scoring Formula

```
Final Score = Base Risk + Activity Modifier + Protective Factors
              (20-100)     (-30 to +20)        (-100 to +40)

Score Range: 0-100 (clamped)
```

### 4.0 Two-Track Scoring (Mature vs. Non-Mature Projects)

A critical insight from validating against real-world package inventories: the original scoring model conflated "stable/finished" with "abandoned." A project like argon2 or dosfstools — quietly maintained for 15 years with occasional small edits — would score identically to a package whose maintainer disappeared. Both show high concentration and low recent activity, but the risk profiles are fundamentally different.

Ossuary uses a **two-track scoring model**:

- **Non-mature projects**: Standard scoring (described in 4.1–4.2 below)
- **Mature projects**: Modified scoring that uses lifetime contributor history and suppresses the abandonment penalty, while adding takeover detection

#### Maturity Heuristics

A project is classified as **mature** when ALL of these are true:

| Criterion | Threshold | Rationale |
|-----------|-----------|-----------|
| Repository age | ≥ 5 years | Long-lived projects have demonstrated stability |
| Total commits | ≥ 30 | Meaningful development history |
| Last commit | Within 5 years | Not truly dead or deleted |

This classification catches stable infrastructure (argon2, dosfstools, logrotate, cronie, fillup) while correctly leaving young abandoned projects scored as risky.

#### Mature Project Scoring Differences

| Component | Non-Mature | Mature |
|-----------|-----------|--------|
| Base risk | Recent (12-month) concentration | **Lifetime** concentration when <4 commits/year; recent otherwise |
| Activity modifier | -30 to +20 | -30 to **0** (never penalized) |
| Takeover detection | N/A | **+20** if proportion shift detected |

For mature projects, the real risk isn't abandonment — it's unexpected takeover (the xz-utils pattern). A project that sat quietly for 15 years with occasional small edits is safe by default.

The lifetime concentration fallback only applies when the project has fewer than 4 commits per year — the "abandoned" activity tier where concentration from 1-3 commits is unreliable. When a mature project has 4+ recent commits, the recent concentration is used as normal, preserving the governance signal.

### 4.1 Base Risk (Concentration + Bus Factor)

Base risk uses two complementary signals — top-1 **concentration** and CHAOSS **bus factor** (minimum contributors for 50% of commits). The worse signal sets the floor.

**Concentration** (top-1 contributor share):

| Concentration | Risk | Interpretation |
|---------------|------|----------------|
| <30% | 20 | Distributed - healthy |
| 30-49% | 40 | Moderate concentration |
| 50-69% | 60 | Elevated concentration |
| 70-89% | 80 | High concentration |
| ≥90% | 100 | Critical - single maintainer |

**Bus factor** (CHAOSS-aligned):

| Bus Factor | Risk Floor | Interpretation |
|-----------|-----------|----------------|
| 1 | 60 | Single person for 50%+ (concentration already captures) |
| 2 | 40 | Two people control the project |
| 3–5 | 40 | Small group, moderate risk |
| 6+ | 20 | Well-distributed |

**Base Risk = max(concentration_risk, bus_factor_risk)**

This catches cases concentration misses. Example: trivy has 18% top-1 concentration (looks distributed) but bus factor 3 — only 3 people account for 50% of commits. Concentration gives base 20; bus factor raises it to 40.

**Calculation**: Concentration = (commits by top contributor / total commits) × 100, using a tapered window (full weight 0-10 months, linear fade 10-14 months) to smooth week-to-week boundary noise. Bus factor computed from unweighted recent commits, excluding bots (`[bot]` in email/name).

For **non-mature** projects, only recent commits are used. For **mature** projects with <4 commits/year, lifetime concentration is used as fallback.

### 4.2 Activity Modifier

Activity level indicates whether maintainers are engaged and responsive.

| Commits/Year | Modifier | Interpretation |
|--------------|----------|----------------|
| >50 | -30 | Actively maintained |
| 12-50 | -15 | Moderately active |
| 4-11 | 0 | Low activity |
| <4 | +20 | Appears abandoned |

**Rationale**: Abandoned packages are prime targets for takeover attacks (event-stream pattern).

**Mature project exception**: For mature projects, the activity modifier is clamped to ≤0 (reductions only). Active maintenance still earns credit, but low activity is not penalized — a 15-year-old tool with 2 commits/year is stable, not abandoned.

### 4.3 Protective Factors

Protective factors can reduce (or increase) risk based on governance quality signals.

#### Risk Reducers (Negative Points)

| Factor | Points | Threshold | Rationale |
|--------|--------|-----------|-----------|
| **Tier-1 Reputation** | -25 | Score ≥60 | Established maintainers have more to lose |
| **Tier-2 Reputation** | -10 | Score ≥30 | Some track record |
| **GitHub Sponsors** | -15 | Has sponsors | Economic sustainability reduces frustration |
| **Organization (3+ admins)** | -15 | Org with succession | Reduces bus factor |
| **Massive Visibility** | -20 | >50M weekly downloads | High scrutiny |
| **High Visibility** | -10 | >10M weekly downloads | Moderate scrutiny |
| **Distributed Governance** | -10 | <40% concentration | Already healthy |
| **Active Community** | -10 | >20 contributors | Community resilience |
| **CII Best Practices** | -10 | Badge present | Security maturity |
| **Positive Sentiment** | -5 | Score >0.3 | Healthy maintainer mood |
| **Project Maturity** | 0 (informational) | Mature project (see §4.0) | Benefit is activity-penalty suppression + lifetime concentration fallback, not a score bonus |

#### Risk Increasers (Positive Points)

| Factor | Points | Condition | Rationale |
|--------|--------|-----------|-----------|
| **Frustration Detected** | +20 | Keywords found | colors/faker pattern |
| **Negative Sentiment** | +10 | Score <-0.3 | Pre-sabotage warning |
| **Takeover Risk** | +20 | Proportion shift >30pp on mature project | xz-utils/Jia Tan pattern (see §4.4) |

### 4.4 Proportion Shift Takeover Detection

For mature projects, Ossuary detects the **xz-utils attack pattern**: a minor or unknown contributor gradually taking over a quiet project.

#### Methodology

For each contributor who made commits in the last 12 months, compute:

```
proportion_shift = recent_share% - historical_share%
```

Where:
- **recent_share%** = contributor's commits in last 12 months / total recent commits × 100
- **historical_share%** = contributor's commits before last 12 months / total historical commits × 100

If any contributor's proportion shift exceeds **+30 percentage points** on a mature project with ≥5 recent commits, a **takeover risk** flag is raised (+20 points).

#### Guards Against False Positives

Two filters prevent false alarms from established maintainers and automated tooling:

1. **Bot filtering**: Contributors with `[bot]` in their email or name are excluded (e.g., dependabot, renovate). Bots can dominate recent commits on quiet projects without representing a takeover risk.

2. **Historical share threshold**: Only contributors with **<5% of historical commits** are considered as takeover suspects. Established maintainers (e.g., a project creator at 20% historical share) naturally fluctuate in activity — that's not a takeover signal. This threshold catches Jia Tan (0.8% historical) while filtering out long-time contributors like project founders whose share temporarily increases.

#### Design Rationale

This approach detects proportional change, not absolute newcomer status. Jia Tan made a few small patches in 2022 — enough to be "established" — before dominating the project in 2023. A binary newcomer check would miss this pattern. Proportion shift catches it because going from 0.8% to 50% of recent commits is a +49.5pp shift regardless of when the contributor first appeared.

#### Validation Against xz-utils Timeline

| Cutoff Date | Jia Tan Historical % | Jia Tan Recent % | Shift | Detected? |
|-------------|---------------------|-------------------|-------|-----------|
| 2023-01 | 0.2% | 15% | +14.8pp | No (below 30pp) |
| 2023-03 | 0.6% | 31% | +30.4pp | **Yes** |
| 2023-06 | 1.2% | 42% | +40.8pp | **Yes** |
| 2024-01 | 3.5% | 50% | +46.5pp | **Yes** |

**Result**: Ossuary detects the xz-utils takeover pattern by March 2023, **12 months before** the backdoor was discovered (March 2024).

---

## 5. Maintainer Reputation System

Reputation provides a composite assessment of maintainer trustworthiness and investment in the ecosystem.

### 5.1 Reputation Signals

| Signal | Points | Threshold |
|--------|--------|-----------|
| **Account Tenure** | +15 | >5 years on GitHub |
| **Portfolio Quality** | +15 | ≥50 original repos with ≥10 stars each |
| **Total Stars** | +15 | ≥50,000 stars across repos |
| **Sponsor Support** | +15 | ≥10 GitHub sponsors |
| **Packages Published** | +10 | ≥20 packages maintained |
| **Top Package Maintainer** | +15 | Maintains top-1000 ecosystem package |
| **Recognized Org** | +15 | Member of nodejs, python, apache, etc. |

### 5.2 Reputation Tiers

| Tier | Score Range | Risk Reduction |
|------|-------------|----------------|
| TIER_1 | ≥60 | -25 points |
| TIER_2 | 30-59 | -10 points |
| UNKNOWN | <30 | 0 points |

### 5.3 Recognized Organizations

Membership in these organizations confers institutional backing:

- **JavaScript/Node**: nodejs, openjs-foundation, npm, expressjs, eslint, webpack, babel
- **Python**: python, psf, pypa, pallets, django, tiangolo
- **General**: apache, cncf, linux-foundation, mozilla, rust-lang, golang
- **Cloud/Infra**: kubernetes, docker, hashicorp

---

## 6. Sentiment Analysis

### 6.1 Approach

Ossuary analyzes commit messages and issue discussions for:

1. **General Sentiment**: Using VADER sentiment analysis
2. **Frustration Keywords**: Pattern matching for burnout/exploitation signals

### 6.2 Frustration Keywords

High-signal keywords that historically preceded sabotage:

```
"not getting paid", "unpaid work", "free labor", "corporate exploitation",
"burned out", "burnout", "stepping down", "abandoning this project",
"fortune 500", "pay developers", "companies make millions",
"protest", "on strike", "boycott", "resentment", "exploitation"
```

### 6.3 Sentiment Scoring

| Compound Score | Effect |
|----------------|--------|
| < -0.3 | +10 risk points |
| > 0.3 | -5 risk points |
| Otherwise | Neutral |

---

## 7. Risk Levels

| Score | Level | Semaphore | Recommended Action |
|-------|-------|-----------|-------------------|
| 0-19 | VERY_LOW | Green | Routine monitoring |
| 20-39 | LOW | Green | Quarterly review |
| 40-59 | MODERATE | Yellow | Monthly review |
| 60-79 | HIGH | Orange | Weekly review, contingency plan |
| 80-100 | CRITICAL | Red | Immediate action required |

---

## 8. Validation Methodology

### 8.1 Dataset Construction

The validation dataset (v6, n=164) includes:

1. **Known Incidents** (33 packages): Packages with documented supply chain incidents, spanning governance failures, protestware, account compromises, CI/CD exploits, and maintainer sabotage. Includes both in-scope and explicitly out-of-scope incidents.
2. **Governance Risk** (12 packages): Packages with elevated governance risk signals but no incident (yet) — abandoned, single-maintainer, or concentrated projects.
3. **Control Group** (119 packages): Popular packages with healthy governance across all 8 ecosystems.

Total: 164 packages across npm (66), PyPI (46), Cargo (8), RubyGems (11), Packagist (5), NuGet (4), Go (6), GitHub (18).

**Dataset construction principles**:
- Incidents drawn from documented supply chain attacks 2016–2026, cross-referenced against multiple sources (Socket.dev, Snyk, CISA advisories, incident write-ups)
- Controls selected as top packages per ecosystem by download count
- Each incident classified by detectability tier (T1-T5) based on whether governance weakness was observable before the attack
- Out-of-scope incidents explicitly included to validate detection boundaries
- Each case includes attack type, incident date, cutoff date (for T-1 analysis), and rationale

### 8.2 Scoped Validation Framework

Traditional recall penalizes the model for not detecting attacks it was never designed to detect (e.g., credential theft on healthy projects). The scoped framework addresses this:

**Tier classification**: Each incident is assigned to a detectability tier based on whether governance weakness was observable before the attack. In-scope tiers (T1, T2, T3, T_risk) contribute to recall; out-of-scope tiers (T4, T5) do not.

| Tier | Label | In-scope? | Count |
|------|-------|-----------|-------|
| T1 | Governance decay → compromise | Yes | 9 |
| T2 | Protestware / sabotage | Yes | 6 |
| T3 | Account compromise + weak governance | Yes | 4 |
| T_risk | Governance risk, no incident | Yes | 12 |
| T4 | Account compromise + strong governance | No | 8 |
| T5 | CI/CD pipeline exploit | No | 6 |

**Decision procedure**: For borderline cases, two questions determine scope: (1) Would Ossuary's signals have shown elevated risk before the attack? (2) Was governance weakness the enabling condition or merely coincidental? If both yes → in-scope.

### 8.3 Classification Rules

| Expected | Predicted Score | Classification |
|----------|-----------------|----------------|
| Incident/Risk (in-scope) | ≥60 | True Positive (TP) |
| Incident/Risk (in-scope) | <60 | False Negative (FN) |
| Safe | <60 | True Negative (TN) |
| Safe | ≥60 | False Positive (FP) |

Out-of-scope incidents (T4, T5) are tracked separately as "bonus detections" but do not count as TP or FN.

### 8.4 Results (n=164, Scope B)

```
In-scope incidents: 31 (T1=9, T2=6, T3=4, T_risk=12)
Out-of-scope incidents: 14 (T4=8, T5=6)
Controls: 119

Confusion Matrix (Scope B):
  TP: 24  |  FN: 7
  FP: 1   |  TN: 118

Accuracy:   94.7%
Precision:  96.0%
Recall:     77.4%
F1 Score:   0.857
```

**Key results**:

- **1 false positive** (rxjs) across 119 safe packages. rxjs scores 75 HIGH due to 100% maintainer concentration and 0 commits in the last year. The governance signals are genuinely concerning; it may warrant reclassification as `governance_risk`.
- **6 in-scope false negatives**, all explainable: faker (community fork), node-ipc (active development masks risk), polyfill.io (ownership transfer untracked), core-js (high activity offsets bus-factor risk), es5-ext and is-promise (maintainer reputation correctly reduces score).
- **7 in-scope false negatives**, all explainable: faker (community fork), node-ipc (active development masks risk), polyfill.io (ownership transfer untracked), devise (borderline drift), core-js (high activity offsets risk), es5-ext (maintainer reputation), is-promise (reputation correctly reconstructed at cutoff).
- **77.4% in-scope recall** reflects genuine detection capability with honest historical scoring. The previous 80.6% was slightly inflated by incorrectly stripping reputation that existed at cutoff dates.

**Comparison with unscoped metrics**: Across all 45 incidents (including out-of-scope), overall recall is 55.6%. This lower number is expected — 14 out-of-scope incidents are fundamentally undetectable from governance signals.

**Tuning history**: v4.0 initially used a -15 maturity bonus + lifetime concentration for all mature projects, achieving 91.6% accuracy on cached scores but only 81.8% on fresh validation. Parameter sweep across 16 configurations (bonus ∈ {0,-5,-10,-15} × lifetime threshold ∈ {1,4,8,12}) identified the optimal: bonus=0, lifetime fallback when <4 commits/year.

### 8.5 Per-Tier Detection Rates

| Tier | Detected | Rate | Notes |
|------|----------|------|-------|
| **T1: Governance decay** | 8/9 | **89%** | 1 miss: polyfill.io (ownership transfer) |
| **T2: Protestware / sabotage** | 2/6 | **33%** | 4 misses: reputation-protected maintainers |
| **T3: Weak-gov compromise** | 4/4 | **100%** | All detected |
| **T_risk: Governance risk** | 10/12 | **83%** | 2 misses: core-js (very active), devise (borderline) |
| T4: Strong-gov compromise (OOS) | 1/8 | 12% | Expected — out of scope |
| T5: CI/CD exploits (OOS) | 0/6 | 0% | Expected — out of scope |

T1 (governance decay, 88%) and T3 (weak-governance compromise, 100%) are the primary targets. T2 (protestware, 33%) is weakest because protestware maintainers tend to have strong reputations that correctly reduce their risk scores. This is a genuine trade-off: reputation DOES reduce attack probability, but doesn't prevent unilateral action.

### 8.6 In-Scope False Negative Analysis

7 in-scope false negatives, all explainable:

| Package | Score | Tier | Why Missed |
|---------|-------|------|-----------|
| faker | 0 | T2 | Evaluating community fork (faker-js/faker); original repo deleted |
| node-ipc | 50 | T2 | Active development masks bus-factor-1 risk |
| polyfill.io | 40 | T1 | Ownership transfer to malicious CDN is an untracked signal |
| devise | 50 | T_risk | Borderline; concentration drift from minor changes |
| core-js | 40 | T_risk | High activity gives discount despite 92% concentration |
| es5-ext | 40 | T2 | 100% concentration but maintainer (medikoo) has strong reputation |
| is-promise | 45 | T2 | Reputation correctly reconstructed at 2020 cutoff |

Historical reputation reconstruction (v3.2) verifies portfolio and tenure at the cutoff date using repo `created_at` timestamps. This gives honest T-1 scores: is-promise (45) reflects ForbesLindesay's real 2020 reputation rather than stripping it to zero. The cost is 1 fewer TP compared to the stripped version, but the score is more accurate.

### 8.7 Out-of-Scope Incident Analysis

14 out-of-scope incidents are included to validate detection boundaries:

**T4: Account compromise on healthy projects (8 cases)** — ua-parser-js (90, bonus detection), eslint-scope (35), LottieFiles (45), chalk (35), cline (0), solana-web3.js (0), eslint-config-prettier (55), num2words (0).

**T5: CI/CD pipeline exploits (6 cases)** — reviewdog (0), codecov (0), rspack (0), ultralytics (0), tj-actions (50), nrwl/nx (0).

All correctly score below threshold except ua-parser-js (bonus detection at 75). A tool that flagged all credential-based attacks would need to flag every package, producing unacceptable false positive rates.

### 8.8 The Scoped Validation Contribution

A key methodological contribution is the **tier-based scoped evaluation**:

1. **Define detectability tiers** a priori based on whether governance signals were observable
2. **Include out-of-scope incidents** in the dataset to validate boundaries
3. **Report both scoped and unscoped metrics** for transparency
4. **Document per-tier detection rates** to identify specific strengths and weaknesses

This is more honest than either (a) reporting only in-scope incidents (hides the limitations) or (b) counting all incidents equally (artificially depresses recall for a tool with a clear, stated scope).

### 8.7 T-1 Validation (Predictive Power)

To validate **predictive** capability, we scored packages at a cutoff date *before* their incidents occurred:

| Package | Incident Date | Cutoff Date | T-1 Score | Level | Key Signals Detected |
|---------|---------------|-------------|-----------|-------|---------------------|
| event-stream | 2018-09-16 | 2018-09-01 | 100 | CRITICAL | 75% concentration, "free work" frustration |
| colors | 2022-01-08 | 2022-01-01 | 100 | CRITICAL | 100% concentration, "protest", "exploitation" |
| coa | 2021-11-04 | 2021-11-01 | 100 | CRITICAL | 100% concentration, abandoned |

**Control comparison** (same timeframe):

| Package | Cutoff Date | Score | Level | Key Signals |
|---------|-------------|-------|-------|-------------|
| express | 2022-01-01 | 0 | VERY_LOW | Org-backed (30 admins), tier-1 maintainer, 64M downloads/wk |

**Result**: 100% detection rate for governance-detectable incidents at T-1, with clear differentiation from healthy packages.

#### T-1 Analysis Details

**event-stream (before September 2018 compromise)**:
```
Score: 100 CRITICAL
- Base Risk: 75% concentration (+80)
- Activity: 4 commits/year (+0)
- Frustration: "free work" keyword detected (+20)
```
The tool would have flagged this as a prime takeover target with frustration signals.

**colors (before January 2022 sabotage)**:
```
Score: 100 CRITICAL
- Base Risk: 100% concentration (+100)
- Activity: 0 commits/year (+20)
- Frustration: "protest", "exploitation" keywords detected (+20)
- Protective: GitHub Sponsors (-15), downloads (-10)
```
Despite protective factors from visibility and sponsors, the frustration signals and extreme concentration produced a CRITICAL score.

**coa (before November 2021 compromise)**:
```
Score: 100 CRITICAL
- Base Risk: 100% concentration (+100)
- Activity: 0 commits/year (+20)
```
Classic abandonment pattern - single maintainer, no activity, prime target for malicious takeover.

This demonstrates that the methodology could have flagged these packages **before** their incidents occurred, validating the predictive value of governance metrics.

#### xz-utils Proportion Shift Detection (v4.0)

The xz-utils attack (CVE-2024-3094) was the most sophisticated governance attack documented, with a 2.6-year timeline. Using proportion shift takeover detection, Ossuary detects the anomaly by **March 2023**:

```
Cutoff: 2023-03 (12 months before disclosure)
Score: HIGH — Takeover risk detected
- Mature project (22 years, 80+ lifetime contributors)
- Jia Tan: +30.4pp shift in commit share
- Evidence: "Jia Tan: +30pp shift in commit share on mature project (xz-utils pattern)"
```

This validates that governance-based scoring can detect social engineering attacks during the grooming phase, before any malicious code is introduced.

---

## 9. Limitations

### 9.1 Methodological Limitations

1. **GitHub-centric**: Relies on GitHub metadata; other forges have limited support
2. **Historical data**: Git history can be rewritten; metrics reflect current state
3. **English bias**: Sentiment analysis optimized for English text
4. **API rate limits**: Full analysis requires authenticated GitHub API access

### 9.2 Detection Limitations

1. **Cannot detect insider threats** from trusted maintainers with good signals
2. **Cannot detect account compromise** on active, well-governed projects
3. **Cannot detect typosquatting** (new packages have no governance history)
4. ~~**May flag healthy "done" packages** as risks~~ — Resolved in v4.0 by mature project detection (see §4.0)

### 9.3 Temporal Limitations

1. **Reputation data is current-state**: Stars, sponsors, repos reflect present, not historical
2. **Organization membership is current**: Historical org membership not tracked
3. **Download counts are current**: Cannot assess historical visibility

### 9.4 Score Stability

#### The problem (pre-v3.1)

Comparing two validation runs one week apart on the same 157 packages:
- **95% stable** (149/157 scored identically)
- **8 packages changed** by ±10-20 points
- sidekiq crossed the 60-point threshold (40→60), creating a phantom false positive with no actual governance change

Root cause: the hard 12-month cutoff for concentration. A single commit crossing the 365-day boundary shifts the top contributor's commit percentage by 2-3%, which the scoring formula amplifies to ±20 point score changes. For example, sidekiq's concentration moved from 68.9%→71.2% solely because one commit aged past day 365.

#### The fix (v3.1): tapered concentration

Activity count keeps the hard 12-month cutoff (the activity modifier uses coarse buckets — >50, ≥12, ≥4, <4 commits — that are insensitive to ±3 commit boundary noise). Concentration uses a tapered window:

- 0-10 months: weight 1.0 (fully recent)
- 10-14 months: weight fades linearly from 1.0 → 0.0
- 14+ months: weight 0.0 (excluded)

A commit at month 11 gradually fades rather than vanishing when it ages past the boundary. This eliminates the cliff edge that caused phantom threshold crossings.

#### Result

Compared to the hard-cutoff run on the same data:
- sidekiq: 60→40 (false positive eliminated)
- No true positives lost
- Precision: 92.3%→96.0%
- Only 6 scores changed, all by ≤20 points, all classification changes were improvements

The taper is not artificial smoothing — a commit from 11 months ago genuinely shouldn't have the same weight as one from last month, and shouldn't vanish entirely at 12 months and 1 day. The hard cutoff was the artifact.

---

## 10. Threats to Validity

This section discusses potential threats to the validity of the research findings, following standard academic conventions for empirical software engineering research.

### 10.1 Internal Validity

Internal validity concerns whether the methodology correctly measures what it claims to measure.

| Threat | Description | Mitigation |
|--------|-------------|------------|
| **Threshold Selection** | Risk thresholds (60+ = risky) were chosen based on incident analysis, not derived empirically | Validated against 164 packages across 8 ecosystems; threshold sensitivity tested at 50, 55, 60, 65 — ≥60 is optimal (96.0% precision, 77.4% in-scope recall) |
| **Keyword Selection Bias** | Frustration keywords derived from known incidents may overfit to historical cases | Keywords based on general burnout/economic frustration patterns, not incident-specific |
| **Scoring Formula Weights** | Point values for factors are hand-tuned, not learned from data | Weights validated through iterative testing; future work could use ML optimization |
| **Maturity Classification** | 5-year/30-commit threshold is heuristic, not empirically derived | Validated against 94 SLE packages; eliminates false CRITICALs on known-stable infrastructure |
| **Confounding Variables** | High scores might correlate with other unmeasured factors (e.g., project age, domain) | Controlled for by including diverse package types in validation set |

### 10.2 External Validity

External validity concerns whether findings generalize beyond the study context.

| Threat | Description | Mitigation |
|--------|-------------|------------|
| **Ecosystem Bias** | Initial validation limited to npm and PyPI | v2+ validation covers 8 ecosystems (npm, PyPI, Cargo, RubyGems, Packagist, NuGet, Go, GitHub) with consistent results |
| **Survivorship Bias** | Can only analyze repositories that still exist; deleted repos (like Marak/Faker.js) are invisible | Acknowledged as limitation; 2 incidents excluded because repos deleted (phpass, electron-native-notify) |
| **Selection Bias in Incidents** | Known incidents may be biased toward governance-detectable cases | Deliberately included 14 out-of-scope incidents (T4: account compromise, T5: CI/CD) to validate detection boundaries |
| **Temporal Generalization** | Validated on 2016-2026 incidents; attack patterns may evolve | T-1 validation confirms historical effectiveness; 2025 incidents (chalk, tj-actions, Nx) confirm boundary holds for recent attacks |
| **Cultural/Language Bias** | English-language sentiment analysis; non-English projects may score differently | Acknowledged limitation; VADER optimized for English social media text |

### 10.3 Construct Validity

Construct validity concerns whether the theoretical constructs are correctly operationalized.

| Threat | Description | Mitigation |
|--------|-------------|------------|
| **"Governance Risk" Definition** | Governance risk is a latent construct; operationalization may not capture all dimensions | Definition grounded in incident analysis; validated by predictive accuracy |
| **Maintainer Concentration Proxy** | Commit count used as proxy for "control"; doesn't capture npm publish rights, code review authority | Git commits are observable and historically correlate with incidents |
| **Frustration Measurement** | Keyword matching is crude; may miss subtle frustration or produce false positives | Combined with VADER sentiment; keywords chosen for high precision |
| **Reputation Conflation** | GitHub stars/repos conflate popularity with trustworthiness | Reputation is one factor among many; not solely determinative |

### 10.4 Conclusion Validity

Conclusion validity concerns whether the statistical conclusions are justified.

| Threat | Description | Mitigation |
|--------|-------------|------------|
| **Small Incident Sample** | 45 incident/risk packages in validation set (31 in-scope) | This is a near-census, not a sample. Cross-referencing CNCF (89), IQT Labs (182), and Ladisa et al. (94) catalogs identified ~50 total scorable governance-relevant incidents across our 8 ecosystems; we include 45 (remainder have deleted repos). The population IS small — governance-detectable attacks are rare events. |
| **Class Imbalance** | 45 incidents vs 119 controls (1:2.6 ratio) | Reported precision and recall separately; F1 accounts for imbalance; metrics reported both scoped (Scope B) and unscoped |
| **No Cross-Validation** | Single train/test split, not k-fold | Dataset is the near-complete population, not a sample from a larger one; temporal holdout analysis performed (≤2022 dev / 2023+ holdout) but holdout has only 3 in-scope incidents |
| **Confidence Intervals** | Point estimates reported without confidence intervals | Bootstrap CIs reported: precision 86–100%, recall 62–91%, F1 74–94%. Wide recall CI reflects genuine uncertainty from small population |
| **ML Comparison** | Hand-tuned formula not validated against learned alternatives | Five ML models tested (LR, SVM, RF, Gradient Boosting, XGBoost). Best ML achieves F1 0.787 vs hand-tuned 0.857. ML validates feature selection and threshold (PR-optimal = 60) but cannot match precision (80% vs 96%) due to small n and nonlinear interactions |

### 10.5 Mitigations Summary

Despite these threats, several factors support the validity of findings:

1. **96.0% Precision**: 1 false positive (rxjs) across 164 packages and 8 ecosystems
2. **77.4% In-Scope Recall**: Scoped framework with honest historical reputation reconstruction
3. **Per-Tier Transparency**: T1 89%, T2 33%, T3 100%, T_risk 83% — specific strengths and weaknesses documented
4. **Near-Census Coverage**: Dataset covers 45 of ~50 known scorable incidents (90%), cross-referenced against CNCF, IQT Labs, and Ladisa et al. catalogs
5. **CHAOSS Bus Factor**: Contributor diversity metric catches patterns missed by top-1 concentration (e.g. trivy: 18% top-1 but bus factor 3)
4. **100% T-1 Detection**: All governance-detectable incidents scored CRITICAL before they occurred
5. **Explicit Boundary Validation**: 14 out-of-scope incidents included and documented
6. **Cross-Ecosystem Generalization**: Consistent results across npm, PyPI, Cargo, RubyGems, Packagist, NuGet, Go, and GitHub
7. **Temporal Range**: Incidents spanning 2016–2026, including the 2025 npm phishing wave and CI/CD exploit wave
8. **Score Stability**: Tapered concentration window eliminates phantom threshold crossings from boundary noise
9. **Grounded in Real Incidents**: Methodology derived from analysis of actual supply chain attacks
10. **Transparent Limitations**: Explicitly documents what the tool cannot detect and proves this empirically

---

## 11. Recommendations for Use

### 11.1 Integration Patterns

1. **CI/CD Pipeline**: Score dependencies on PR, fail on CRITICAL
2. **Scheduled Audits**: Weekly scans of dependency tree
3. **Acquisition Diligence**: Score target's OSS dependencies
4. **Vendor Assessment**: Evaluate third-party software stacks

### 11.2 Score Interpretation

| Score Range | Action |
|-------------|--------|
| 0-39 | Standard dependency management |
| 40-59 | Add to watchlist, review quarterly |
| 60-79 | Investigate alternatives, prepare fork |
| 80-100 | **Immediate review**, consider removal |

### 11.3 Combining with Other Tools

Ossuary complements but does not replace:

| Tool Type | Purpose | Ossuary Relationship |
|-----------|---------|---------------------|
| **SBOM tools** | Inventory what you have | Provides risk context |
| **Vulnerability scanners** | Known CVEs | Different risk dimension |

### 11.4 Empirical Comparison with OpenSSF Scorecard

OpenSSF Scorecard is the most widely-used automated security assessment tool for open source projects. To test the hypothesis that governance scoring and security-practice scoring measure orthogonal dimensions, we ran Scorecard against the same 158 packages in our validation set and compared results systematically.

#### Methodology

| Aspect | OpenSSF Scorecard | Ossuary |
|--------|-------------------|---------|
| **Focus** | Security best practices | Governance risk |
| **Checks** | CI/CD, branch protection, fuzzing, SAST, signed releases | Concentration, activity, frustration signals, reputation |
| **Question answered** | "Does this project follow security hygiene?" | "Could this project be abandoned or compromised?" |
| **Scale** | 0–10 (higher = better practices) | 0–100 (higher = more risk) |

We queried the Scorecard public API (`api.securityscorecards.dev`) for all 158 packages, resolving each to its GitHub repository via ecosystem-specific registry APIs (npm, PyPI, RubyGems, Cargo, Packagist, NuGet, Go). **132 of 158 packages** returned valid Scorecard data (26 had no indexed data, typically because the repository was deleted or not yet scanned).

#### Correlation Analysis

| Metric | Value |
|--------|-------|
| Packages with both scores | 132 |
| **Pearson correlation** | **−0.678** |
| **Spearman rank correlation** | **−0.625** |

The moderate negative correlation is expected: high Ossuary scores (risky governance) tend to occur on packages with low Scorecard scores (poor security practices), and vice versa. However, the correlation is far from −1.0, confirming the tools measure substantially different dimensions. The unexplained variance (r² = 0.46) represents the orthogonal information each tool uniquely captures.

#### Quadrant Analysis

Using thresholds of Ossuary ≥ 60 (HIGH+ risk) and Scorecard ≥ 5.0/10:

```
                          Scorecard ≥ 5.0    Scorecard < 5.0
  Ossuary ≥ 60 (risky)         2                  19
  Ossuary < 60 (healthy)      84                  27
```

**Quadrant I — High Ossuary Risk + High Scorecard (2 packages)**

These are the most thesis-relevant cases: projects that follow good security practices but have governance vulnerabilities that only Ossuary detects.

| Package | Ossuary | Scorecard | Attack Type |
|---------|---------|-----------|-------------|
| **xz-utils** | 80 CRITICAL | 6.3 | governance_failure |
| **ua-parser-js** | 75 HIGH | 8.1 | account_compromise |

*xz-utils* is the canonical example: Jia Tan deliberately maintained excellent security practices (signed releases, code review, CI/CD) specifically to build trust before inserting the backdoor. Scorecard gave 6.3/10 — reasonable security hygiene. Ossuary scored 80 CRITICAL because it detected the underlying governance vulnerability: extreme contributor concentration in a single newcomer who had rapidly gained commit access.

*ua-parser-js* scored 8.1/10 on Scorecard — excellent security practices — yet was compromised via account takeover. Ossuary scored 75 HIGH due to single-maintainer concentration risk that made the account a high-value target.

**Quadrant II — High Ossuary Risk + Low Scorecard (19 packages)**

Both tools flag problems, though for different reasons. Examples:

| Package | Ossuary | Scorecard | Type |
|---------|---------|-----------|------|
| colors | 100 | 2.2 | maintainer_sabotage |
| event-stream | 90 | 2.1 | governance_failure |
| rest-client | 100 | 3.3 | account_compromise |
| moment | 85 | 2.8 | governance_risk |
| boltdb/bolt | 100 | 3.0 | governance_risk |

This is the "agreement" quadrant — both tools correctly identify these as problematic, though Scorecard flags poor practices while Ossuary flags governance risk.

**Quadrant III — Low Ossuary Risk + High Scorecard (84 packages)**

Both tools agree these are healthy. This is the largest quadrant, dominated by well-maintained popular packages:

| Package | Ossuary | Scorecard |
|---------|---------|-----------|
| express | 0 | 8.8 |
| requests | 0 | 8.6 |
| typescript | 0 | 8.1 |
| django | 0 | 6.8 |
| react | 0 | 6.5 |
| tokio | 0 | 7.2 |

**Quadrant IV — Low Ossuary Risk + Low Scorecard (27 packages)**

Poor security practices but acceptable governance — a dimension Ossuary doesn't penalize:

| Package | Ossuary | Scorecard |
|---------|---------|-----------|
| debug | 30 | 2.6 |
| inherits | 35 | 3.6 |
| nanoid | 5 | 3.8 |

Many small, stable npm packages fall here: single-purpose modules with no CI/CD or branch protection, but functional governance (responsive maintainer, no concentration risk).

#### Incident Detection Comparison

Of the 32 incident packages with valid Scorecard data:

| Detection Method | Flagged | Rate |
|------------------|---------|------|
| Ossuary ≥ 60 | 21/32 | 66% |
| Scorecard < 5.0 | 27/32 | 84% |
| **Either tool** | **30/32** | **94%** |

The two tools that missed detection overlap on only 2 packages — combining them catches 94% of incidents vs. 66% or 84% alone. This demonstrates their complementary value.

Critically, the 5 incidents where Scorecard ≥ 5.0 (indicating good security practices) were all detected by Ossuary:

| Package | Ossuary | Scorecard | Attack |
|---------|---------|-----------|--------|
| ua-parser-js | 75 HIGH | 8.1 | account_compromise |
| xz-utils | 80 CRITICAL | 6.3 | governance_failure |
| tj-actions/changed-files | 50 | 5.9 | account_compromise |
| codecov/codecov-action | 0 | 7.0 | account_compromise |
| nrwl/nx | 0 | 6.4 | account_compromise |

Of these 5, Ossuary flagged 2 (xz-utils and ua-parser-js) — precisely the governance-detectable ones. The remaining 3 (tj-actions, codecov, nx) are CI/CD or credential compromises that neither governance nor security-practice scoring can predict, confirming the detection boundary analysis from §8.6.

#### Key Findings

1. **Moderate negative correlation (r = −0.68)** confirms the tools measure related but distinct dimensions. They are neither redundant (r ≈ −1.0) nor fully independent (r ≈ 0).

2. **The "blind spot" quadrant** (High Ossuary + High Scorecard) contains 2 packages — both real incidents where good security practices masked governance vulnerability. This is precisely where Ossuary adds unique value.

3. **Combined detection** reaches 94% of incidents, substantially better than either tool alone.

4. **Different failure modes**: Scorecard misses well-practiced projects with governance risk (xz-utils, ua-parser-js). Ossuary misses credential/CI/CD compromises at well-governed projects (codecov, nx). The tools are complementary by design.

### 11.5 Comparison with CHAOSS Metrics (Augur/GrimoireLab)

The [CHAOSS project](https://chaoss.community/) (Community Health Analytics for Open Source Software) defines standardized metrics for open source health. Tools like [Augur](https://github.com/chaoss/augur) and [GrimoireLab](https://chaoss.github.io/grimoirelab/) implement these metrics.

#### Relevant CHAOSS Metrics

| CHAOSS Metric | Description | Ossuary Equivalent |
|---------------|-------------|-------------------|
| **Contributor Absence Factor** (Bus Factor) | Minimum contributors for 50% of commits | Maintainer Concentration |
| **Elephant Factor** | Minimum *organizations* for 50% of commits | Not measured (individual focus) |
| **Activity Dates and Times** | Commit frequency patterns | Activity Modifier |
| **Change Request Closure Ratio** | PR/issue responsiveness | Not directly measured |

#### Key Differences

| Aspect | CHAOSS/Augur/GrimoireLab | Ossuary |
|--------|--------------------------|---------|
| **Output** | Individual metrics and dashboards | Single risk score (0-100) |
| **Focus** | Community health broadly | Supply chain attack risk specifically |
| **Sentiment** | Not included | Frustration/burnout detection |
| **Actionability** | Requires interpretation | Direct risk level + recommendations |
| **Historical** | Strong time-series support | Cutoff-based T-1 analysis |

#### Complementary Relationship

CHAOSS metrics provide **deep diagnostic data** for understanding community dynamics. Ossuary provides **rapid risk triage** for supply chain security decisions.

Example workflow:
1. **Ossuary** flags package as HIGH risk (score 75)
2. **GrimoireLab** dashboard reveals *why*: declining contributor diversity, increasing response times
3. Security team makes informed decision with full context

#### Academic Foundation

Ossuary's concentration metric aligns with CHAOSS's [Contributor Absence Factor](https://chaoss.community/kb/metric-contributor-absence-factor/), providing academic grounding for the approach. The key innovation is combining this with:
- Sentiment analysis for frustration detection
- Reputation scoring for maintainer assessment
- Weighted scoring formula calibrated against known incidents

### 11.6 Real-World Dependency Scanning

To demonstrate practical applicability, we scanned the dependency trees of two major open source projects using `ossuary scan`.

#### Express.js (npm, 28 runtime dependencies)

Express is the most widely-used Node.js web framework. Scanning its `package.json` runtime dependencies:

| Risk Level | Count | Examples |
|------------|-------|---------|
| CRITICAL | 2 | escape-html (100% conc., 0 commits/yr), depd (100% conc., 0 commits/yr) |
| HIGH | 3 | once (100% conc., 1 commit/yr), merge-descriptors (100% conc., 0 commits/yr), cookie-signature (100% conc., 0 commits/yr) |
| MODERATE | 1 | etag (67% conc., 3 commits/yr) |
| LOW | 10 | debug, qs, statuses, proxy-addr, http-errors, ... |
| VERY_LOW | 12 | body-parser, accepts, router, mime-types, ... |

**18% of Express's runtime dependencies score HIGH or CRITICAL.** These are single-maintainer micro-packages with no recent activity — exactly the governance profile that preceded the event-stream compromise. A security team reviewing this output would prioritize `escape-html` and `depd` for governance review or vendoring.

#### Home Assistant (PyPI, 58 core dependencies)

Home Assistant is one of the largest Python open source projects, with over 2,000 integrations. Scanning its core `requirements.txt`:

| Risk Level | Count | Examples |
|------------|-------|---------|
| CRITICAL | 9 | standard-aifc, standard-telnetlib, atomicwrites-homeassistant, ifaddr, astral, cronsim, ... |
| HIGH | 6 | ciso8601, orjson, audioop-lts, psutil-home-assistant, ha-ffmpeg, ... |
| MODERATE | 3 | python-slugify, webrtc-models, lru-dict |
| LOW | 11 | PyYAML, aiohttp_cors, packaging, bcrypt, Jinja2, ... |
| VERY_LOW | 28 | aiohttp, requests, cryptography, Pillow, SQLAlchemy, ... |

**26% of Home Assistant's core dependencies score HIGH or CRITICAL.** Several CRITICAL packages are HA-specific forks (`atomicwrites-homeassistant`, `psutil-home-assistant`) or niche single-maintainer libraries (`ifaddr`, `astral`) — packages unlikely to appear in vulnerability scanners but carrying real governance risk.

#### Practical Implications

These scans reveal a pattern common across large projects: the core framework dependencies (aiohttp, requests, cryptography) are well-governed, but the **long tail of utility dependencies** carries significant governance risk. This aligns with the "Roads and Bridges" thesis (Eghbal, 2016) — critical infrastructure depends on small, undermaintained packages that receive no attention until they fail.

The `ossuary scan` command makes this risk visible in seconds, enabling:
1. **Dependency review** during security audits
2. **CI/CD integration** to flag new high-risk dependencies in pull requests
3. **Vendoring decisions** — CRITICAL-scored packages are candidates for vendoring or replacement
4. **Maintainer outreach** — identifying which upstream packages need community investment

---

## 12. Future Work

1. ~~**Expand ecosystem support**~~: Done - 8 ecosystems (npm, PyPI, Cargo, RubyGems, Packagist, NuGet, Go, GitHub)
2. **Historical snapshots**: Archive reputation/org data for better T-1 analysis
3. **ML enhancement**: Train classifier on larger incident corpus
4. ~~**Dependency file scanning**~~: Done - `ossuary scan` supports requirements.txt, package.json, Cargo.toml, go.mod, Gemfile, composer.json, .csproj
5. **Dependency graph analysis**: Transitive risk aggregation
5. **Maintainer network analysis**: Identify shared maintainer risks across packages
6. ~~**PyPI repository URL discovery**~~: Done - case-insensitive URL extraction with multi-priority fallback
7. ~~**Mature project detection**~~: Done - two-track scoring for projects >5 years old with established history
8. ~~**Takeover detection**~~: Done - proportion shift analysis catches xz-utils pattern 12 months early

---

## References

### Essential Reading

These papers directly inform the methodology and should be read in full:

1. **Ladisa, P., et al. (2023). "SoK: Taxonomy of Attacks on Open-Source Software Supply Chains."** IEEE S&P 2023. https://arxiv.org/abs/2204.04008
   — *The* foundational taxonomy. Identifies 12 attack categories; governance-based attacks are a distinct class. Start here.

2. **Eghbal, N. (2016). "Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure."** Ford Foundation.
   — Established that most OSS is maintained by 1-2 people. The "why bus factor matters" argument for any thesis on this topic.

3. **Raman, N., et al. (2020). "Stress and Burnout in Open Source."** ICSE-NIER 2020. https://dl.acm.org/doi/10.1145/3377816.3381732
   — SVM classifier for toxic discussions; directly influenced Ossuary's sentiment/frustration analysis.

4. **Ohm, M., et al. (2020). "Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks."** DIMVA 2020. https://dasfreak.github.io/Backstabbers-Knife-Collection/
   — Catalog of real-world attacks with patterns. Good for incident classification in lit review.

5. **"A Software Engineering Analysis of the XZ Utils Supply Chain Attack."** arXiv 2504.17473. https://arxiv.org/abs/2504.17473
   — Detailed analysis of the most sophisticated governance attack. The 2.6-year timeline is the key case study.

6. **Guo, Y., et al. (2024). "Sustaining Maintenance Labor for Healthy Open Source Software Projects."** arXiv. https://arxiv.org/abs/2408.06723
   — Argues depleted maintainer capacity → unmaintained projects → security consequences. Supports the activity modifier.

### Additional References

7. "Research Directions in Software Supply Chain Security." ACM TOSEM 2024. https://dl.acm.org/doi/10.1145/3714464
8. Sonatype. "State of the Software Supply Chain." Annual Report. https://www.sonatype.com/state-of-the-software-supply-chain
9. Synopsys. (2022). "Open Source Security and Risk Analysis Report."
10. OpenSSF Scorecard - https://securityscorecards.dev/
11. **Goggins, S., Germonprez, M., & Lumbard, K. (2021). "Making Open Source Project Health Transparent."** IEEE Computer, 54(8), 104–111. https://doi.org/10.1109/MC.2021.3084015 — Key paper on the CHAOSS project and its approach to community health metrics.
12. **Avelino, G., Passos, L., Hora, A., & Valente, M. T. (2016). "A Novel Approach for Estimating Truck Factors."** 24th IEEE International Conference on Program Comprehension (ICPC), 1–10. https://arxiv.org/abs/1604.06766 — Defines the algorithm Ossuary's bus factor metric is based on: minimum contributors whose departure causes >50% of files to become orphaned.
13. CHAOSS Contributor Absence Factor metric definition - https://chaoss.community/kb/metric-contributor-absence-factor/
14. CISA Alert AA25-266A. "Widespread Supply Chain Compromise Impacting npm Ecosystem." September 23, 2025. https://www.cisa.gov/news-events/alerts/2025/09/23/widespread-supply-chain-compromise-impacting-npm-ecosystem
15. CERT/CC VU#534320. "npm ecosystem design weaknesses enabling supply chain compromise." https://kb.cert.org/vuls/id/534320
16. Check Point Research. "The Great NPM Heist: September 2025." https://blog.checkpoint.com/crypto/the-great-npm-heist-september-2025/
17. Unit 42 / Palo Alto Networks. "Shai-Hulud: npm Supply Chain Worm." https://unit42.paloaltonetworks.com/npm-supply-chain-attack/
18. **Cosentino, V., Canovas Izquierdo, J. L., & Cabot, J. (2015). "Assessing the Bus Factor of Git Repositories."** 22nd IEEE SANER, 499–503. https://doi.org/10.1109/SANER.2015.7081864 — Earlier bus factor computation work.
19. LFX Insights / Linux Foundation Open Source Index - https://insights.linuxfoundation.org/open-source-index
20. OSSInsight (PingCAP) - https://ossinsight.io/

---

*Document version: 6.2*
*Last updated: March 2026*
*Validation dataset: 164 packages across 8 ecosystems (Scope B: 96.0% precision, 77.4% recall, F1 0.857)*
*Run validation: `python scripts/validate.py -o validation_results.json`*
