# newsfromthefront — Design Spec

**Date:** 2026-03-12
**Status:** Approved

---

## Overview

`newsfromthefront` is a CogOS application that continuously monitors the competitive landscape for a given project. It ingests a GitHub repository plus user-written context, runs daily web research via the Perplexity API, tracks findings over time, and delivers delta reports to a Discord channel as threaded posts. Users refine the research brief by replying to report threads. A backfill capability allows historical analysis to seed the knowledge base before daily incremental operation begins.

---

## Architecture

### Image

A new CogOS image located at `images/apps/newsfromthefront/`. This is a standalone image directory, separate from `cogent-v1`, under an `images/apps/` convention for application-level cogents. All process names are prefixed `newsfromthefront-` for clarity.

### Processes

| Process | Mode | Runner | Trigger |
|---|---|---|---|
| `newsfromthefront-researcher` | daemon | lambda | daily cron channel message |
| `newsfromthefront-analyst` | daemon | lambda | `newsfromthefront:findings-ready`, `newsfromthefront:discord-feedback` |
| `newsfromthefront-test` | daemon | lambda | `newsfromthefront:run-requested` (mode: "test") |
| `newsfromthefront-backfill` | daemon | lambda | `newsfromthefront:run-requested` (mode: "backfill") |

`newsfromthefront-backfill` is re-entrant: each Lambda invocation processes one interval, then re-triggers itself for the next, so no single run is long-lived.

### New Capability

`web_search` — a first-class CogOS capability backed by the Perplexity API. Registered in `BUILTIN_CAPABILITIES` and reusable by any future cogent.

### Channels

| Channel | Producer | Consumer | Purpose |
|---|---|---|---|
| `newsfromthefront:findings-ready` | researcher, test | analyst | Signals that raw findings are ready for analysis |
| `newsfromthefront:discord-feedback` | discord-handle-message | analyst | Routes Discord thread replies back to the analyst |
| `newsfromthefront:run-requested` | discord-handle-message | test, backfill | Triggers on-demand runs |

### File Store Layout

```
newsfromthefront/
  brief.md                  — GitHub URL + user goals/context (evolves via thread replies)
  knowledge-base.json       — structured history of all findings seen to date
  state.json                — Discord thread IDs for active report threads
  backfill-state.json       — re-entrant backfill progress (present only during an active backfill)
  findings/<date>.md        — raw Perplexity output per researcher run
  reports/<date>.md         — generated analyst report per run
```

---

## Processes

### `newsfromthefront-researcher`

**Trigger:** Daily cron channel message.

**Behavior:**
1. Read `newsfromthefront/brief.md` — GitHub URL, user goals, accumulated context, known competitors list.
2. Fetch the repo's README and key documentation files via the GitHub REST API (using `secrets.get("cogent/github_token")`).
3. Derive a set of Perplexity search queries from the project description — the LLM generates these from the brief, not a hardcoded list, so query quality improves as the brief grows richer. Use `recency: "day"` to focus on fresh content.
4. Run each query via the `web_search` capability.
5. Write all raw findings to `newsfromthefront/findings/<date>.md`.
6. Send `{run_id, findings_key, date, is_test: false, is_backfill: false}` on `newsfromthefront:findings-ready`.

**Capabilities:** `web_search`, `dir`, `channels`, `secrets`

**Retry:** `max_retries=3, retry_backoff_ms=60000`. On GitHub API unavailability, falls back to the cached project description in `brief.md` and proceeds with search only.

---

### `newsfromthefront-analyst`

**Trigger:** `newsfromthefront:findings-ready` or `newsfromthefront:discord-feedback`.

**On `findings-ready`:**
1. Read the findings file at `findings_key`.
2. Read `newsfromthefront/knowledge-base.json`.
3. Identify genuinely new findings — deduplicate by URL and title against the knowledge base.
4. Classify new findings by type: `competitor`, `product_update`, `funding`, `launch`, `other`.
5. Write the delta report to `newsfromthefront/reports/<date>.md`.
6. If `is_test: false` and `is_backfill: false`: post the report to the configured Discord channel as a new thread titled "Newsfromthefront — `<date>`". Store the thread ID in `newsfromthefront/state.json`.
7. If `is_backfill: true`: not applicable — backfill handles its own research and KB updates directly, and does not route through the analyst.
8. If `is_test: true`: post to a clearly labeled test thread. Do not update the knowledge base or state.
9. Update `newsfromthefront/knowledge-base.json` with new findings (unless `is_test: true`).

**On `discord-feedback`:**
1. Read the feedback content and author.
2. Read `newsfromthefront/brief.md`.
3. Incorporate the feedback — update goals, add context, refine competitors list, adjust search focus.
4. Write the updated brief back to `newsfromthefront/brief.md`.
5. Reply in the Discord thread to confirm: "Brief updated." so the user knows their feedback was received.

**Capabilities:** `web_search`, `dir`, `channels`, `discord`, `secrets`

**Error handling:** If Discord posting fails after retries, post a brief error note to the channel so failures are never silent.

---

### `newsfromthefront-test`

**Trigger:** `newsfromthefront:run-requested` with `mode: "test"`.

**Behavior:** Runs the full researcher + analyst loop from scratch. Passes `is_test: true` throughout — does not update the knowledge base or state. Posts report to a clearly labeled test thread (title: "Newsfromthefront TEST — `<date>`"). Intended for tuning runs where the user wants to see end-to-end output without affecting production state.

**Capabilities:** `web_search`, `dir`, `channels`, `discord`, `secrets`

---

### `newsfromthefront-backfill`

**Trigger:** `newsfromthefront:run-requested` with `mode: "backfill"`. On first trigger, payload includes `after_date` and `before_date`; on subsequent self-triggers those are omitted (state is read from file store).

**Re-entrancy:** Backfill is re-entrant — each Lambda invocation processes exactly one interval, then self-triggers for the next. Progress is tracked in `newsfromthefront/backfill-state.json` so a crash or restart resumes from the last completed interval rather than restarting from scratch.

**Behavior:**
1. If `after_date`/`before_date` present in payload: initialize `backfill-state.json` with `{after_date, before_date, current_date: after_date, intervals_done: 0, findings_count: 0}`. Determine granularity — week-by-week for ranges > 30 days, day-by-day otherwise.
2. Read `backfill-state.json`. Process the next interval starting from `current_date`: run researcher-style Perplexity searches using `after_date`/`before_date` params for the interval. Write findings to `newsfromthefront/findings/<date>.md`. Run analyst logic (dedup against KB, update KB). Increment `current_date` and `intervals_done`. Write updated state.
3. If more intervals remain: send `{mode: "backfill"}` on `newsfromthefront:run-requested` to trigger the next invocation.
4. If all intervals complete: post a summary thread to Discord — "Backfill complete: `<after_date>` → `<before_date>`. Knowledge base initialized with N findings." Delete `backfill-state.json`.

**Note:** Because each invocation updates the knowledge base directly (rather than routing through the analyst channel), there are no concurrent KB write races.

**Capabilities:** `web_search`, `dir`, `channels`, `discord`, `secrets`

---

## `web_search` Capability

### Location

`src/cogos/capabilities/web_search.py` — registered in `BUILTIN_CAPABILITIES`.

### Methods

Three search backends, all on the same capability:

```
web_search.search(
  query:       string   — the search query
  recency:     string?  — "day" | "week" | "month" (relative window, for normal runs)
  after_date:  string?  — ISO date, lower bound (for backfill)
  before_date: string?  — ISO date, upper bound (for backfill)
) → {
  summary: string,
  sources: list[{title: string, url: string, snippet: string}]
}

web_search.search_github(
  query:       string   — GitHub search query (supports qualifiers: org:, repo:, language:, etc.)
  type:        string?  — "repositories" | "issues" | "discussions" | "code" (default: "repositories")
  after_date:  string?  — ISO date, filter by updated_at
  before_date: string?  — ISO date, filter by updated_at
) → {
  items: list[{title: string, url: string, description: string, stars: number?, updated_at: string}]
}

web_search.search_twitter(
  query:       string   — X/Twitter search query (supports operators: from:, lang:, -filter:retweets, etc.)
  recency:     string?  — "day" | "week" | "month" (approximate, maps to max_results + time window)
  after_date:  string?  — ISO date (maps to X API start_time)
  before_date: string?  — ISO date (maps to X API end_time)
) → {
  tweets: list[{id: string, text: string, author: string, url: string, created_at: string, likes: number, retweets: number}]
}
```

### Implementation

**`search`** — Calls Perplexity's `/chat/completions` endpoint with `model: "sonar"`. Reads API key from `secrets.get("cogent/perplexity_api_key")`. Maps `recency` to Perplexity's `search_recency_filter` param; maps `after_date`/`before_date` to Perplexity's date filter params.

**`search_github`** — Calls GitHub's `/search/{type}` REST API. Reads token from `secrets.get("cogent/github_token")`. Constructs date qualifiers (`pushed:>YYYY-MM-DD`) from `after_date`/`before_date`. Returns up to 30 items per call.

**`search_twitter`** — Calls X API v2 `/tweets/search/recent` (or `/tweets/search/all` for backfill with date ranges). Reads Bearer token from `secrets.get("cogent/twitter_bearer_token")`. Returns up to 100 tweets per call, sorted by recency.

Default scoping: unrestricted. Scoping can restrict to allowed `ops` (which methods are permitted).

---

## Channel Schemas

### `newsfromthefront:findings-ready`

```yaml
fields:
  run_id:       string
  findings_key: string
  date:         string
  is_test:      bool
  is_backfill:  bool
```

### `newsfromthefront:discord-feedback`

```yaml
fields:
  thread_id: string
  content:   string
  author:    string
```

### `newsfromthefront:run-requested`

```yaml
fields:
  mode:        string   — "test" | "backfill"
  after_date:  string?
  before_date: string?
```

---

## Knowledge Base Structure

`newsfromthefront/knowledge-base.json`:

```json
{
  "findings": [
    {
      "id": "<uuid>",
      "date": "2026-03-12",
      "type": "competitor | product_update | funding | launch | other",
      "title": "...",
      "summary": "...",
      "url": "...",
      "relevance": "brief note on why this matters to the project"
    }
  ],
  "competitors": ["..."],
  "last_run": "2026-03-12"
}
```

The `competitors` list is fed back into the researcher's search query generation on each run, so coverage deepens automatically over time. Deduplication uses URL and title — if a finding has been seen before it is not reported as new.

---

## Discord Feedback Loop

When a user @mentions the bot in a report thread:

1. The Discord bridge captures the mention via `io:discord:mention`. The message payload includes `channel_id` (the Discord thread's channel ID) — this is already supported by the bridge.
2. `discord-handle-message` checks whether `channel_id` is present in `newsfromthefront/state.json`.
3. If yes: forwards to `newsfromthefront:discord-feedback` with `{thread_id, content, author}`.
4. If no: handles normally as a chat message.
5. `newsfromthefront-analyst` wakes on `discord-feedback`, reads the feedback, and updates `brief.md`.

---

## Testing Workflow

Three modes, each faster than the last:

| Mode | How | What it tests | Touches KB? |
|---|---|---|---|
| Full fresh test | `@cogent test` in Discord | End-to-end: ingest + search + analysis + report | No |
| Analyst-only | `cogent dr.alpha cogos channel send newsfromthefront:findings-ready ...` with `is_test: true` | Report formatting, analysis quality | No |
| Search-only | `cogent dr.alpha cogos channel send` to trigger researcher, then read findings file | Query generation, Perplexity result quality | No |

Use full fresh tests for tuning end-to-end output. Use analyst-only to iterate on report formatting without burning Perplexity credits. Use search-only to tune query generation independently.

---

## Required Secrets

| Key | Purpose |
|---|---|
| `cogent/perplexity_api_key` | Perplexity API — `web_search.search()` |
| `cogent/github_token` | GitHub REST API — `web_search.search_github()` and repo ingestion |
| `cogent/twitter_bearer_token` | X API v2 Bearer Token — `web_search.search_twitter()` |

---

## CogOS Enhancements (Desired)

These are not blockers — the application can be built without them — but they are real gaps that will recur with every new CogOS application.

### Enhancement #1: HTTP capability

GitHub repo ingestion currently requires writing raw Python HTTP code in `run_code()` using the `secrets` capability for the API key. A general-purpose `http` capability would make external API calls a clean, typed, auditable primitive rather than raw code:

```
http.get(url: string, headers: dict?) → {status: number, body: string}
http.post(url: string, body: dict, headers: dict?) → {status: number, body: string}
```

This would be useful for any cogent that needs to interact with external REST APIs (GitHub, Linear, Stripe, etc.) and avoids reimplementing HTTP boilerplate in every process.

---

## Out of Scope

- Multi-project support (single project per instance for now; `brief.md` is singular)
- Email delivery of reports
- Configurable per-project cadence (daily is hardcoded)
- Automatic competitor discovery without user confirmation
