# RSS Feed Processing Guide

This guide explains how to use the RSS Feed module for fetching, parsing, and managing RSS feeds from various sources.

## Overview

The RSS Feed module provides comprehensive functionality for:
- Fetching RSS feeds from URLs
- Parsing feed metadata and entries
- Async batch processing with concurrency control
- Intelligent caching with expiration
- Retry logic with exponential backoff
- Export to multiple formats (JSON, CSV)

## Installation

The RSS module is part of the Information Composer package:

```bash
pip install information-composer
```

### Dependencies

The module requires:
- `feedparser` - RSS/Atom feed parsing
- `aiohttp` - Async HTTP requests
- `python-dateutil` - Date parsing
- `pyyaml` - Configuration support

## Basic Usage

### Fetching a Single Feed

```python
from information_composer.rss.fetcher import RSSFetcher

# Create fetcher
fetcher = RSSFetcher()

# Fetch a feed
result = fetcher.fetch_single("https://example.com/feed.xml")

# Access feed data
print(f"Title: {result.feed_title}")
print(f"Description: {result.feed_description}")
print(f"Entries: {len(result.entries)}")

for entry in result.entries[:5]:
    print(f"  - {entry.title}")
    print(f"    URL: {entry.link}")
    print(f"    Date: {entry.published}")
```

### Batch Fetching Multiple Feeds

```python
from information_composer.rss.fetcher import RSSFetcher

# Create fetcher with custom config
config = {
    "max_concurrent": 10,
    "timeout": 30,
    "cache_enabled": True
}
fetcher = RSSFetcher(config)

# List of feed URLs
feeds = [
    "https://example.com/feed1.xml",
    "https://example.com/feed2.xml",
    "https://example.com/feed3.xml"
]

# Fetch all feeds
results = fetcher.fetch_batch(feeds)

# Process results
for result in results:
    if result.success:
        print(f"{result.feed_title}: {len(result.entries)} entries")
    else:
        print(f"Error fetching {result.feed_url}: {result.error_message}")
```

### Async Batch Fetching

```python
import asyncio
from information_composer.rss.fetcher import RSSFetcher

async def fetch_feeds_async():
    # Create fetcher
    fetcher = RSSFetcher()

    # Feed URLs
    feeds = [
        "https://example.com/feed1.xml",
        "https://example.com/feed2.xml"
    ]

    # Fetch asynchronously
    results = await fetcher.fetch_batch_async(feeds)

    # Process results
    for result in results:
        if result.success:
            print(f"Fetched: {result.feed_title}")

# Run async function
asyncio.run(fetch_feeds_async())
```

## CLI Usage

The RSS module includes a comprehensive command-line interface:

### Fetch Command

```bash
# Fetch single feed
rss-fetcher -u "https://example.com/feed.xml" -o output.json

# Fetch with custom max entries
rss-fetcher -u "https://example.com/feed.xml" -o output.json --max-entries 50

# Fetch with cache disabled
rss-fetcher -u "https://example.com/feed.xml" -o output.json --no-cache

# Fetch multiple feeds from file
rss-fetcher --input feeds.txt -o output.json

# Export to CSV
rss-fetcher -u "https://example.com/feed.xml" -o output.csv --format csv
```

### Batch Command

```bash
# Batch fetch with custom concurrency
rss-fetcher --input feeds.txt -o results.json --concurrent 15

# Batch fetch with custom timeout
rss-fetcher --input feeds.txt -o results.json --timeout 60

# Batch fetch with retry
rss-fetcher --input feeds.txt -o results.json --max-retries 5
```

### Cache Management

```bash
# Clear cache
rss-fetcher cache clear --cache-dir ./rss_cache

# Set cache expiration
rss-fetcher cache expire --cache-dir ./rss_cache --days 30
```

## Configuration

### YAML Configuration File

Create `rss_config.yaml`:

```yaml
# Cache settings
cache_enabled: true
cache_dir: "./rss_cache"
cache_expire_days: 7

# Request settings
timeout: 30
max_retries: 3
retry_delay: 2

# Concurrency settings
max_concurrent: 10

# User agent
user_agent: "information-composer/0.2.1"

# Output settings
default_output_format: "json"
include_raw_xml: false
```

### Environment Variables

```bash
# Cache directory
export RSS_CACHE_DIR="./rss_cache"

# Request timeout
export RSS_TIMEOUT=30

# Max concurrent requests
export RSS_MAX_CONCURRENT=10

# Cache expiration
export RSS_CACHE_EXPIRE_DAYS=7
```

### Python Configuration

```python
from information_composer.rss.fetcher import RSSFetcher

config = {
    "cache_enabled": True,
    "cache_dir": "./rss_cache",
    "cache_expire_days": 7,
    "timeout": 30,
    "max_retries": 3,
    "retry_delay": 2,
    "max_concurrent": 10,
    "user_agent": "custom-user-agent"
}

fetcher = RSSFetcher(config)
```

## Data Models

### FeedResult

```python
@dataclass
class FeedResult:
    """Result of fetching an RSS feed."""

    feed_url: str
    feed_title: str
    feed_description: str
    feed_link: str
    entries: list[FeedEntry]
    status: FeedStatus
    success: bool
    error_message: str | None
    fetch_time: datetime
    cache_hit: bool
```

### FeedEntry

```python
@dataclass
class FeedEntry:
    """Represents an entry in an RSS feed."""

    title: str
    link: str
    description: str
    published: datetime | None
    author: str | None
    categories: list[str]
    content: str | None
    entry_id: str | None
```

### FeedStatus

```python
class FeedStatus(Enum):
    """Status of feed fetch operation."""

    SUCCESS = "success"
    ERROR = "error"
    TIMEOUT = "timeout"
    PARSE_ERROR = "parse_error"
```

## Advanced Usage

### Custom Parser

```python
from information_composer.rss.parser import RSSParser

# Create parser
parser = RSSParser()

# Parse feed content
with open("feed.xml", "r") as f:
    content = f.read()

metadata, entries = parser.parse(content, "https://example.com/feed.xml")

# Access parsed data
print(f"Title: {metadata['title']}")
print(f"Entries: {len(entries)}")
```

### Cache Management

```python
from information_composer.rss.cache import CacheManager

# Create cache manager
cache = CacheManager("./rss_cache", expire_days=7)

# Check cache
cached_data = cache.get_cached("https://example.com/feed.xml")
if cached_data:
    print("Using cached data")

# Update cache
cache.update_cache("https://example.com/feed.xml", data, entries_count=10)

# Clear cache
cache.clear_cache()

# Clear expired entries
cache.clear_expired()
```

### Error Handling

```python
from information_composer.rss.fetcher import RSSFetcher
from information_composer.rss.models import FeedStatus

fetcher = RSSFetcher()

result = fetcher.fetch_single("https://example.com/feed.xml")

if result.success:
    # Process successful fetch
    for entry in result.entries:
        print(entry.title)
else:
    # Handle errors
    if result.status == FeedStatus.TIMEOUT:
        print("Request timed out")
    elif result.status == FeedStatus.PARSE_ERROR:
        print(f"Parse error: {result.error_message}")
    else:
        print(f"Error: {result.error_message}")
```

### Filtering Entries

```python
from information_composer.rss.fetcher import RSSFetcher
from datetime import datetime, timedelta

fetcher = RSSFetcher()
result = fetcher.fetch_single("https://example.com/feed.xml")

# Filter by date
recent_date = datetime.now() - timedelta(days=7)
recent_entries = [
    e for e in result.entries
    if e.published and e.published > recent_date
]

# Filter by category
tech_entries = [
    e for e in result.entries
    if "technology" in [c.lower() for c in e.categories]
]

# Filter by keywords
keywords = ["AI", "machine learning"]
filtered_entries = [
    e for e in result.entries
    if any(kw.lower() in e.description.lower() for kw in keywords)
]
```

## Integration Examples

### With PubMed

```python
from information_composer.rss.fetcher import RSSFetcher
from information_composer.pubmed import pmid_to_crossref_work

# Fetch PubMed RSS feed
fetcher = RSSFetcher()
result = fetcher.fetch_single("https://pubmed.ncbi.nlm.nih.gov/rss/trending/")

# Process entries
for entry in result.entries[:10]:
    print(f"Title: {entry.title}")
    print(f"Link: {entry.link}")

    # Extract DOI if available
    # DOI is often in the link or description
```

### With LLM Filter

```python
from information_composer.rss.fetcher import RSSFetcher
from information_composer.llm_filter.core.filter import MarkdownFilter

# Fetch feed
fetcher = RSSFetcher()
result = fetcher.fetch_single("https://example.com/feed.xml")

# Convert to markdown
markdown_content = "\n\n".join([
    f"## {e.title}\n\n{e.description}"
    for e in result.entries
])

# Filter with LLM
filter_config = {
    "model": "qwen-plus",
    "api_key": "your-api-key",
    "criteria": "Technology and AI related articles"
}

llm_filter = MarkdownFilter(filter_config)
filtered_content = llm_filter.filter_text(markdown_content)
```

### Export to Database

```python
import sqlite3
from information_composer.rss.fetcher import RSSFetcher

# Fetch feed
fetcher = RSSFetcher()
result = fetcher.fetch_single("https://example.com/feed.xml")

# Store in database
conn = sqlite3.connect("feeds.db")
cursor = conn.cursor()

# Create table
cursor.execute("""
    CREATE TABLE IF NOT EXISTS feed_entries (
        id TEXT PRIMARY KEY,
        title TEXT,
        link TEXT,
        description TEXT,
        published TIMESTAMP,
        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
    )
""")

# Insert entries
for entry in result.entries:
    cursor.execute("""
        INSERT OR REPLACE INTO feed_entries
        (id, title, link, description, published)
        VALUES (?, ?, ?, ?, ?)
    """, (
        entry.entry_id or entry.link,
        entry.title,
        entry.link,
        entry.description,
        entry.published
    ))

conn.commit()
conn.close()
```

## Best Practices

1. **Always use cache** for frequently accessed feeds
2. **Set appropriate timeouts** for slow feeds
3. **Limit concurrent requests** to avoid overwhelming servers
4. **Handle errors gracefully** with proper retry logic
5. **Validate feed data** before processing
6. **Clean up cache** regularly to avoid disk space issues
7. **Use async operations** for batch processing
8. **Respect robots.txt** and feed terms of service

## Troubleshooting

### Common Issues

1. **Timeout Errors**: Increase timeout value or check network connectivity
2. **Parse Errors**: Feed might be malformed, check feed URL
3. **Cache Issues**: Clear cache and retry
4. **Rate Limiting**: Reduce concurrent requests

### Debug Mode

```python
import logging

# Enable debug logging
logging.basicConfig(level=logging.DEBUG)

# Your RSS operations
fetcher = RSSFetcher()
result = fetcher.fetch_single("https://example.com/feed.xml")
```

## Examples Repository

For more examples, see the `examples/` directory:

- `rss_fetch_example.py` - Basic fetching examples
- `rss_batch_example.py` - Batch processing examples
- `rss_async_example.py` - Async operations
- `rss_filter_example.py` - Entry filtering
- `rss_integration_example.py` - Integration with other modules
