# Design-005: Metatags in Code Comments for Graph Linkage

**Date:** 2026-02-13
**Status:** Proposal
**Author:** (auto-generated feasibility study)

## 1. Problem Statement

Clew Engine currently derives graph relationships exclusively from **structural AST analysis** — parsing `IMPORTS`, `CALLS`, `DEFINES`, and `CONTAINS` edges from tree-sitter. This captures the mechanical shape of code but misses **semantic intent**: the relationships that developers express in documentation comments.

A function's docstring might say `@see AuthValidator` or `:class:\`SessionManager\``, signaling an architectural relationship that no amount of AST analysis can infer. These annotations are a rich, untapped source of graph edges that would strengthen Clew's context retrieval for AI agents.

**Goal:** Determine whether structured metatags in code comments (analogous to JSDoc `@see`, `@returns`, `@link`) can be parsed during ingestion and materialized as first-class graph edges in Neo4j.

## 2. Feasibility Assessment

**Verdict: Feasible.** The current architecture accommodates this with no breaking changes.

### 2.1 Tree-Sitter Comment Extraction

Tree-sitter represents comments as **"extra" nodes** in the AST — they appear in the tree and can be visited during traversal, but are currently ignored by `CodeParser._traverse()`.

| Language | Comment Node Type(s) | Notes |
|---|---|---|
| Python | `comment` | `#`-prefixed lines only; docstrings are `expression_statement > string` |
| JavaScript/TypeScript | `comment` | Both `//` and `/* */` share one node type |
| Java | `comment`, `block_comment`, `line_comment` | Grammar-dependent |
| Go | `comment` | Both `//` and `/* */` |
| Rust | `line_comment`, `block_comment` | Two distinct node types |
| C/C++ | `comment` | Both `//` and `/* */` |
| Ruby | `comment` | `#`-prefixed |
| PHP | `comment` | All styles |
| Kotlin | `comment`, `multiline_comment` | Grammar-dependent |

**Key finding:** Comments are already traversed by `_traverse()` (the recursive `for child in node.children` loop at `parser.py:99`). They simply don't match any extraction rule and are skipped. Adding a `comment_types` frozenset to `LanguageConfig` and an `elif` branch in `_traverse()` is the only change needed in the parser.

### 2.2 Associating Comments with Symbols

Comments are "extra" nodes in tree-sitter — they don't have a parent-child AST relationship with the function or class they document. Association requires a heuristic.

**Recommended approach:** Line-proximity matching.
- A comment block on lines N..M is associated with the first definition starting on line M+1 or M+2 (allowing one blank line).
- Python docstrings (the first `expression_statement > string` inside a function/class body) are handled separately: they are children of the definition node itself.

This heuristic is used by every major doc generator (JSDoc, Javadoc, Sphinx) and is reliable in practice.

### 2.3 Compatibility with Existing Pipeline

| Component | Impact | Detail |
|---|---|---|
| `LanguageConfig` | Add one field | `comment_types: frozenset[str]` |
| `CodeParser._traverse()` | Add one `elif` branch | Extract comment text + line range |
| `ParsedNode` | No change | `type="comment"` fits the existing schema |
| `ProcessingStage` (parsing) | No change | Comments become `ParsedNode` entries |
| Neo4j `GraphStore` | Add methods | New relationship types for metatag edges |
| Qdrant `VectorStore` | Optional | Embed comment text for semantic search |
| API `/v1/graph/traverse` | Extend allowlist | Add new relationship types |

No existing behavior changes. All additions are purely additive.

## 3. Prior Art Survey

### 3.1 Existing Standards

| Ecosystem | Format | Relationship Tags | Reference Syntax |
|---|---|---|---|
| **JSDoc** (JS/TS) | `@tag` in `/** */` | `@see`, `@link`, `@extends`, `@implements` | `@see ClassName`, `{@link ClassName#method}` |
| **Javadoc** (Java) | `@tag` in `/** */` | `@see`, `@link`, `@linkplain` | `@see pkg.Class#method` |
| **Sphinx** (Python) | `:role:\`ref\`` in docstrings | `:class:`, `:func:`, `:mod:`, `:meth:` | `:class:\`~auth.SessionManager\`` |
| **Google-style** (Python) | `See Also:` section | Convention only | Free-form text |
| **Rustdoc** (Rust) | `[text](path)` in `///` | Intra-doc links | `[SessionValidator](crate::auth::SessionValidator)` |
| **Doxygen** (C/C++) | `@tag` in `/** */` or `///` | `@see`, `@relates`, `@extends` | `@see ClassName` |
| **KDoc** (Kotlin) | `@tag` in `/** */` | `@see`, `@sample`, `@constructor` | `@see [ClassName]` |
| **PHPDoc** (PHP) | `@tag` in `/** */` | `@see`, `@link`, `@uses` | `@see ClassName::method()` |

### 3.2 Key Observation

Despite surface-level syntax differences, every ecosystem converges on the same **core relationship verbs**: `see` (cross-reference), `link` (inline reference), `extends`/`implements` (type hierarchy), and `returns`/`param` (type flow). The `@`-prefix tag syntax originated in Javadoc and has become the de facto standard in most languages.

## 4. Proposed Conventions

### 4.1 Design Principles

1. **Don't invent a new standard.** Reuse existing conventions. Developers should not need to learn anything new.
2. **Language-native first.** Parse each language's own doc-comment format. A Python developer writes Sphinx roles; a TypeScript developer writes JSDoc tags.
3. **Normalize to a common graph vocabulary.** Regardless of surface syntax, map to a small set of Neo4j relationship types.
4. **Graceful degradation.** Unresolved references are stored as strings, not dropped. Resolution is best-effort.

### 4.2 Minimal Metatag Set (Graph Relationship Types)

These are the **output** relationship types materialized in Neo4j. The input syntax varies by language.

| Relationship | Meaning | Source Tags |
|---|---|---|
| `REFERENCES` | Generic cross-reference | `@see`, `:class:`, `:func:`, `@link`, Rustdoc `[...]` |
| `EXTENDS` | Inheritance / subclassing | `@extends`, `@augments` |
| `IMPLEMENTS` | Interface / protocol conformance | `@implements` |
| `RETURNS_TYPE` | Return type linkage | `@returns {Type}`, `@return`, `:rtype:` |
| `USES` | Dependency / consumption | `@uses`, `@requires` |

Five relationship types. That's the entire vocabulary for v1.

### 4.3 Input Syntax by Language Family

**Family A: @-tag languages** (JavaScript, TypeScript, Java, Kotlin, PHP, C/C++, C#, Swift)

Parsed from `/** */` or `///` comment blocks:

```javascript
/**
 * Validates an incoming session token.
 *
 * @see SessionManager           → REFERENCES edge to "SessionManager"
 * @extends BaseValidator        → EXTENDS edge to "BaseValidator"
 * @implements Validator         → IMPLEMENTS edge to "Validator"
 * @returns {AuthResult}         → RETURNS_TYPE edge to "AuthResult"
 * @uses TokenCache              → USES edge to "TokenCache"
 */
function validateSession(token) { ... }
```

Parsing: regex `@(see|extends|implements|returns|uses)\s+\{?(\w[\w.#]*)` applied to comment text.

**Family B: Sphinx-role languages** (Python)

Parsed from docstrings (triple-quoted strings as first statement in a body):

```python
def validate_session(token: str) -> AuthResult:
    """Validate an incoming session token.

    See Also:
        :class:`SessionManager`       → REFERENCES edge to "SessionManager"
        :func:`check_expiry`           → REFERENCES edge to "check_expiry"
        :meth:`TokenCache.lookup`      → USES edge to "TokenCache.lookup"

    Returns:
        :class:`AuthResult`            → RETURNS_TYPE edge to "AuthResult"
    """
```

Python also supports `@`-tag syntax in `#`-comments and some codebases use it in docstrings. The parser should handle both.

Parsing: regex `:(class|func|meth|mod|attr):\`~?([^`]+)\`` for Sphinx roles, plus `@(see|extends|uses)\s+(\w[\w.]*)` for @-tag fallback.

**Family C: Intra-doc link languages** (Rust)

Parsed from `///` and `//!` doc comments:

```rust
/// Validates an incoming session token.
///
/// See also [`SessionManager`] for session lifecycle.
/// Extends [`BaseValidator`].
///
/// # Returns
/// [`AuthResult`] on success.
pub fn validate_session(token: &str) -> AuthResult { ... }
```

Parsing: regex `` \[`([^`]+)`\] `` for Rustdoc intra-doc links. Contextual keywords ("Extends", "Implements", "Returns") in surrounding text determine the relationship type; default is `REFERENCES`.

### 4.4 Reference Format

References extracted from metatags are stored as **strings** using dot-separated qualified names:

| Raw Tag | Stored Reference |
|---|---|
| `@see SessionManager` | `SessionManager` |
| `@see pkg.auth.SessionManager` | `pkg.auth.SessionManager` |
| `{@link Class#method}` | `Class.method` |
| `:class:\`~auth.SessionManager\`` | `auth.SessionManager` |
| `[crate::auth::SessionManager]` | `auth.SessionManager` |

Resolution to actual Neo4j `CodeBlock` nodes happens at query time (not ingestion time), using name matching with fallback to fuzzy search. This avoids ingestion failures when references point to code that hasn't been indexed yet or lives in external dependencies.

## 5. Proposed Neo4j Schema Extension

### 5.1 New Relationship Types

```cypher
-- Add to ALLOWED_RELATIONSHIPS in graph.py route
ALLOWED_RELATIONSHIPS = {
    "IMPORTS", "CALLS", "CONTAINS", "DEFINES",   -- existing
    "REFERENCES", "EXTENDS", "IMPLEMENTS",         -- new: metatag-derived
    "RETURNS_TYPE", "USES"                         -- new: metatag-derived
}
```

### 5.2 Relationship Properties

Each metatag-derived edge carries metadata about its source:

```cypher
(source:CodeBlock)-[:REFERENCES {
    source: "comment",          -- origin: "comment" vs "ast" (future)
    tag: "@see",                -- original tag name
    raw: "@see SessionManager", -- original text
    line: 42                    -- line number of the comment
}]->(target:CodeBlock)
```

### 5.3 Unresolved References

When a metatag references a symbol not found in the graph, create a placeholder node:

```cypher
MERGE (u:UnresolvedRef {name: $ref_name, repo_id: $repo_id})
CREATE (source)-[:REFERENCES {source: "comment", resolved: false}]->(u)
```

On subsequent ingestion runs, attempt to resolve `UnresolvedRef` nodes against newly indexed `CodeBlock` nodes and replace the placeholder edge.

## 6. Implementation Approach

### 6.1 Phase 1 — Comment Extraction (Minimal)

**Changes:**

1. **`language_registry.py`**: Add `comment_types` field to `LanguageConfig`. Register comment node types for all languages.

2. **`parser.py`**: Add `elif node_type in config.comment_types` branch in `_traverse()`. Extract comment text, line range, and associated definition (by proximity).

3. **`context.py`**: No changes needed — `ParsedNode(type="comment", ...)` works as-is.

4. **New module `metatag_parser.py`**: Regex-based extraction of metatags from comment text. Returns list of `(relationship_type, target_ref)` tuples. Language-family dispatch (A/B/C from section 4.3).

### 6.2 Phase 2 — Graph Materialization

**Changes:**

1. **New processor `comment_processor.py`**: Analogous to `definition.py`, `call.py`, `import_processor.py`. For each comment's metatags, create Neo4j relationship edges from the associated definition to the referenced symbol.

2. **`graph.py` (ingestion)**: Add `create_metatag_relationship()` method. MERGE relationship with properties.

3. **`graph.py` (API route)**: Extend `ALLOWED_RELATIONSHIPS` set.

4. **`neo4j.py` (adapter)**: No changes — the Cypher traversal query already handles arbitrary relationship types via the `$rel_types` parameter.

### 6.3 Phase 3 — Semantic Enhancement (Optional)

- Embed comment text in Qdrant for semantic search ("find code related to session management" matches comments).
- MCP tool enrichment: include metatag relationships in `explore_module` responses.
- Broken-reference reporting: warn when `@see Foo` points to nothing after full ingestion.

## 7. Use Cases

### 7.1 Architectural Impact Analysis (Clew-Bot)

**Current:** Clew-Bot detects that a PR modifies `AuthHandler` and traverses `CALLS`/`IMPORTS` edges to find direct dependents.

**With metatags:** Clew-Bot also finds that `SessionController` has `@see AuthHandler` in its docstring — a conceptual dependency invisible to AST analysis. The PR comment now includes: *"Note: `SessionController` references `AuthHandler` via documentation (line 15). Verify semantic compatibility."*

### 7.2 Type Hierarchy Discovery

```python
class RedisCache:
    """Redis-backed cache implementation.

    @implements CacheProtocol
    @extends BaseCache
    """
```

Query: *"What implements CacheProtocol?"* → Traverse `IMPLEMENTS` edges → find `RedisCache`, `MemoryCache`, etc. — even when Python's duck-typed protocols make AST-based detection unreliable.

### 7.3 Cross-Module Navigation for AI Agents

An MCP `search_codebase("authentication flow")` returns `validate_session()`. The agent then calls `explore_module("validate_session")` and receives not just `CALLS`/`IMPORTS` but also `REFERENCES` edges from metatags, giving it a broader map of related code to read.

### 7.4 Breaking Change Detection

A PR deletes `TokenValidator`. Graph query finds all incoming `REFERENCES`/`USES` edges from metatags across the codebase. The CI check reports: *"Warning: 3 files reference `TokenValidator` in documentation comments — update or remove these references."*

### 7.5 Onboarding and Documentation Quality

Query all `CodeBlock` nodes with zero outgoing `REFERENCES` edges from metatags → identify under-documented code. Or query `UnresolvedRef` nodes → identify stale documentation.

## 8. Risks and Mitigations

| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Comment format fragmentation across codebases | High | Low | Parse multiple formats per language family; unknown tags are silently skipped |
| False positives (matching `@see` in non-doc comments) | Medium | Low | Only parse comments associated with definitions (proximity heuristic filters noise) |
| Unresolved references cluttering the graph | Medium | Medium | Store as `UnresolvedRef` nodes; exclude from traversal by default; resolve lazily |
| Performance regression during ingestion | Low | Low | Comment extraction adds ~5% to parse time (regex on small strings); no database round-trip change |
| Developers not using metatags | High | None | Zero downside — feature is purely additive. Value grows organically as codebases adopt conventions |

## 9. What This Proposal Does NOT Cover

- **Inventing a universal comment metatag standard.** We parse existing conventions.
- **Enforcing metatag usage.** No linting or required tags.
- **Parsing natural language in comments.** Only structured tags with known patterns.
- **Multi-language files** (e.g., JSX with CSS-in-JS). Single-language-per-file assumption holds.

## 10. Decision

**Recommendation:** Proceed with Phase 1 (comment extraction + metatag parsing) after the current Pre-Release Hardening milestone completes. The feature is low-risk, purely additive, and directly strengthens Clew's core differentiator: graph-based context that goes beyond what vector search alone can provide.

Phase 2 (graph materialization) should follow immediately, as the metatags only deliver value when they become traversable edges.

Phase 3 (semantic enhancement) can be deferred based on user feedback from the Internal Alpha.
