# FenLiu (分流)

*Created by marvin8 with assistance from Claude and DeepSeek AI assistants.*

> **⚠️ DISCLAIMER / PROVISO**: This project is a **work in progress** with major changes still happening. It is in no way anywhere close to finished and is only borderline useful for actual production use. Expect breaking changes, incomplete features, and significant architectural evolution as development continues.

**Divide the Fediverse content flow**

FenLiu is a web application that monitors Fediverse hashtags, filters spam, allows human review, learns from feedback, and exports quality content for boosting. Inspired by the ancient Chinese Dujiangyan irrigation system (256 BC) that separated silt from water, FenLiu applies 2,300-year engineering wisdom to modern digital content streams.

## Current Status — v0.7.1

Fully functional spam filtering and content management system with complete Curated Queue integration, flexible pattern-based user blocking, automated queue lifecycle management, production-ready containerization, and ML training data collection.

**Latest (v0.7.1)**: Random post selection for the Curated Queue API. 411 tests passing.

## Documentation

**📚 Live Docs**: https://marvinsmastodontools.codeberg.page/dujiangyan/fenliu/

The `docs/` folder contains full MkDocs documentation covering installation, API reference, pattern blocking, Curated Queue integration, and more.

```bash
mkdocs serve   # serve locally with hot reload
mkdocs build   # build static site
```

## Features

### Core Functionality
- **Hashtag Monitoring**: Monitor multiple Fediverse hashtags with customizable instance sources and scheduling
- **Spam Scoring**: Rule-based detection (0-100 scale) with 7 intelligent detection rules
- **Manual Review Interface**: Approve/reject posts with scoring; pagination, bulk actions, and back-to-top link
- **Curated Queue Export**: Reliable API-driven queue with ack/nack/error pattern

### Reblog Controls (Export Filters)
- **Pattern-Based User Blocking**: exact, suffix, prefix, and contains matching modes
- **Hashtag Blocklist**: Exclude posts containing blocked hashtags
- **Attachments-Only Mode**: Export only posts with media attachments
- **Auto-Reject on Fetch**: Automatically reject blocked content before review
- **Blocklist Refresh**: Apply Settings changes to the review page instantly

### Web Interface
- Dashboard, Streams Management, Review Workflow, Pattern Blocking Settings, Queue Preview, Statistics
- Responsive design — no external JavaScript dependencies

### REST API
- Hashtag streams, posts, curated queue, reblog controls, statistics, health
- API key authentication for queue endpoints

### Technical Quality
- 411 tests, 100% passing
- Comprehensive type hints; zero type errors under `ty check`
- All functions pass ruff and complexipy checks
- Alembic migrations run automatically on startup

## Quick Start

**Prerequisites**: Python 3.12+, `uv` package manager

```bash
uv sync -U --all-groups
fenliu --reload --debug
```

Open `http://localhost:8000`, create a hashtag stream, fetch posts, and review them.

### Container Deployment

```bash
podman build -t fenliu -f Containerfile .
cp .env.example .env   # edit with your settings
podman run -d -p 8000:8000 --env-file .env \
  -v fenliu-data:/app/data -v fenliu-logs:/app/logs fenliu
```

See the [Container Deployment guide](https://marvinsmastodontools.codeberg.page/dujiangyan/fenliu/getting-started/container-deployment/) for full details.

## API Endpoints

All curated queue endpoints require `X-API-Key` header (generate in Settings).

**Streams & Posts:**
- `GET /api/v1/streams` — List streams
- `POST /api/v1/streams` — Create stream
- `GET/PUT/DELETE /api/v1/streams/{id}` — Stream operations
- `POST /api/v1/streams/{id}/fetch` — Fetch posts for stream
- `POST /api/v1/streams/fetch-all` — Fetch all active streams
- `GET /api/v1/posts` — List posts with filtering
- `PATCH /api/v1/posts/{id}` — Update post (review, approve, score)
- `GET /api/v1/stats` — Application statistics

**Curated Queue:**
- `GET /api/v1/curated/next` — Next post (204 if empty); `?random=true` for random selection
- `POST /api/v1/curated/{post_id}/ack` — Confirm successful reblog
- `POST /api/v1/curated/{post_id}/nack` — Return to queue (transient failure)
- `POST /api/v1/curated/{post_id}/error` — Mark permanently failed
- `POST /api/v1/curated/{post_id}/requeue` — Return errored post to queue
- `POST /api/v1/curated/cleanup` — Delete old delivered posts
- `POST /api/v1/curated/trim-pending` — Trim excess pending posts

**Reblog Controls:**
- `GET/PUT /api/v1/reblog-controls/settings` — Reblog filter settings
- `GET/POST /api/v1/reblog-controls/blocked-users` — Blocked users (pattern-based)
- `DELETE /api/v1/reblog-controls/blocked-users/{id}` — Remove blocked user
- `GET/POST /api/v1/reblog-controls/blocked-hashtags` — Blocked hashtags
- `DELETE /api/v1/reblog-controls/blocked-hashtags/{id}` — Remove blocked hashtag
- `POST /api/v1/reblog-controls/reject-blocked` — Bulk reject matching posts

**System:** `GET /health`, `GET /info`

See the [API docs](https://marvinsmastodontools.codeberg.page/dujiangyan/fenliu/api/) for full reference.

## Configuration

Key environment variables (see `.env.example` for full list):

| Variable | Default | Description |
|----------|---------|-------------|
| `DATABASE_URL` | `sqlite:///./fenliu.db` | Database connection |
| `DEFAULT_INSTANCE` | `mastodon.social` | Default Fediverse instance |
| `RESERVE_TIMEOUT_SECONDS` | `300` | Queue reservation timeout |
| `VERY_HIGH_THRESHOLD` | `76` | Spam score: very high |
| `LOW_MAX_THRESHOLD` | `25` | Spam score: low |
| `DEBUG` | `false` | Enable debug logging |

## Development

```bash
uv run pytest                    # run tests
uv run ruff check .              # lint
uv run complexipy .              # complexity check
uv run ty check                  # type check
nox                              # full CI simulation

alembic upgrade head             # apply migrations
alembic revision --autogenerate -m "description"  # new migration
```

## Project Structure

```
fenliu/
├── src/fenliu/
│   ├── main.py                  # PyView application and LiveViews
│   ├── models.py                # SQLAlchemy models
│   ├── schemas.py               # Pydantic validation
│   ├── api/                     # REST API (curated, reblog_controls, api_keys)
│   ├── services/                # Business logic (spam scoring, fediverse, scheduler)
│   ├── templates/               # HTML templates
│   └── static/                  # CSS and assets
├── alembic/                     # Database migrations
├── tests/                       # Test suite (402 tests)
└── docs/                        # MkDocs documentation
```

## Technical Stack

- **Framework**: PyView (Starlette-based LiveView)
- **Database**: SQLAlchemy + SQLite, Alembic migrations
- **Validation**: Pydantic v2
- **Fediverse**: minimal-activitypub
- **Frontend**: Jinja2 + Tailwind CSS, no external JS
- **Testing**: pytest (411 tests)
- **Tooling**: ruff, ty, complexipy, uv


## What's New in v0.7.1

- **Random queue selection**: `GET /api/v1/curated/next?random=true` returns a randomly chosen eligible post instead of the oldest. If the chosen author has more than one pending post, their oldest is returned to avoid consecutive same-author posts
- **9 new tests** for random selection behaviour; 411 total

## Previous Release — v0.7.0

- **Review pagination**: 20 posts/page with prev/next navigation
- **Bulk actions**: Approve All / Reject All for the current page
- **Auto-refresh**: Empty page reloads automatically when more posts exist
- **ML training snapshots**: `ReviewFeedback` captures 12 feature fields at review time — survives queue cleanup
- **Bug fix**: Stream deletion cascade fixed (`Post → ReviewFeedback`)

## Previous Release — v0.6.0

- **Queue Lifecycle Management**: Auto-delete delivered posts (7-day retention), trim excess pending with weighted random deletion, `cleanup` and `trim-pending` API endpoints
- **Production Containerization**: Multi-stage build (~207 MB), non-root user, persistent volumes, automatic migrations on startup

## Previous Release — v0.5.3

- **Pattern-Based User Blocking**: exact, suffix, prefix, and contains matching modes — see [PATTERN_BLOCKING_FEATURE.md](PATTERN_BLOCKING_FEATURE.md)
- **Blocklist Refresh**: Apply Settings changes to the review page without losing progress

## Cultural Context

"FenLiu" (分流) means "divide the flow" in Chinese, inspired by the Dujiangyan irrigation system (256 BC). This project applies the same engineering wisdom to digital content streams.

## Key Resources
- [Live Documentation](https://marvinsmastodontools.codeberg.page/dujiangyan/fenliu/) — Full docs
- [Roadmap](ROADMAP.md) — Development plans
- [Pattern Blocking Guide](PATTERN_BLOCKING_FEATURE.md) — Pattern matching details
- [API Reference](docs/api/) — Endpoint documentation

## License
AGPL-3.0 — see LICENSE file.

## Contributing
1. Follow existing code style (ruff, comprehensive type hints)
2. Write tests for new functionality
3. Run `nox` before submitting changes
4. Run `alembic upgrade head` after pulling changes with new migrations

---
**v0.7.1** · Phase 4 In Progress · 411 tests ✅ · [Codeberg](https://codeberg.org/marvinsmastodontools/fenliu)
