# Information Composer Documentation

Welcome to Information Composer! A comprehensive toolkit for collecting, composing, and filtering information from various web resources.

## Documentation Navigation

### Quick Start
- [Installation Guide](installation.md) - How to install and configure the project
- [Quick Start](quickstart.md) - Get started in 5 minutes
- [Configuration](configuration.md) - Detailed configuration options
- [Project Overview](overview.md) - Project architecture and features

### Feature Guides
- [PDF Validation](guides/pdf-validator.md) - PDF file format validation
- [Markdown Processing](guides/markdown-processing.md) - Markdown document processing
- [PubMed Integration](guides/pubmed-integration.md) - Academic literature search and processing
- **NEW:** [Crossref Integration](guides/crossref-integration.md) - Crossref API queries and bibliographic data
- **NEW:** [RSS Processing](guides/rss-integration.md) - RSS feed fetching and parsing
- [DOI Download](guides/doi-download.md) - Academic paper download by DOI
- [LLM Filtering](guides/llm-filtering.md) - AI-powered document filtering
- [Google Scholar Integration](guides/google-scholar-integration.md) - Google Scholar web scraping
- [RiceDataCN Integration](guides/ricedatacn-integration.md) - Rice gene data parsing

### API Reference
- [Core API](api/core.md) - Core functionality API
- [PDF API](api/pdf.md) - PDF processing API
- [Markdown API](api/markdown.md) - Markdown processing API
- [PubMed API](api/pubmed.md) - PubMed query API
- [Crossref API](api/crossref.md) - Crossref integration API
- [RSS API](api/rss.md) - RSS processing API
- [Sites API](api/sites.md) - Web scraping API
- [LLM Filter API](api/llm-filter.md) - LLM filtering API

### Examples and Tutorials
- [Basic Examples](examples/basic-usage.md) - Basic usage examples
- [Advanced Examples](examples/advanced-usage.md) - Advanced features
- [Integration Examples](examples/integration-examples.md) - Integration with other tools

### Development Guide
- [Development Setup](development/setup.md) - Development environment configuration
- [Code Quality](development/code-quality.md) - Code quality tools
- [Testing](development/testing.md) - Testing and debugging
- [Contributing](development/contributing.md) - How to contribute

### Deployment and Operations
- [Deployment Guide](deployment/deployment.md) - Production deployment
- [CI/CD Configuration](deployment/ci-cd.md) - Continuous integration
- [Monitoring](deployment/monitoring.md) - System monitoring

## Main Features

### Core Modules
- **PDF Validation**: Verify PDF file format and integrity
- **Markdown Processing**: Advanced markdown processing with LLM filtering
- **DOI Management**: Download and manage academic papers by DOI
- **PubMed Integration**: Query and process PubMed data

### AI-Powered Features
- **LLM Filtering**: Support for DashScope, Ollama, and OpenAI
- **PubMed Analyzer**: AI-powered literature analysis
- **Markdown Filter**: Intelligent content extraction

### Web Scraping & Data Collection
- **Crossref Integration**: Query Crossref API for bibliographic data
- **Google Scholar**: Crawl and process Google Scholar citations
- **RSS Feeds**: Parse and manage scientific RSS feeds
- **RiceDataCN**: Extract gene data from RiceDataCN database

### Developer Tools
- **Code Quality**: Ruff linter and formatter (primary tool)
- **Testing**: Pytest with 51%+ coverage (570 tests passed)
- **Multi-format Support**: PDF, Markdown, JSON, XML, TXT

## Quick Start

```bash
# Install project
pip install -e .

# Activate environment
source activate.sh  # Linux/macOS
activate.bat        # Windows

# Validate PDF
pdf-validator document.pdf

# Filter markdown with LLM
md-llm-filter -i input.md -o output.md

# Search PubMed
pubmed-cli search "cancer research" -e user@example.com

# Query Crossref
crossref-cli query --doi "10.1038/nature12373"

# Crawl Google Scholar
google-scholar-crawler -q "machine learning" -n 20

# Fetch RSS feeds
rss-fetcher -u "https://example.com/feed.xml" -o output.json
```

## Available CLI Commands

| Command | Description |
|---------|-------------|
| `pdf-validator` | Validate PDF files |
| `md-llm-filter` | Filter markdown with LLM |
| `pubmed-cli` | Search and fetch PubMed data |
| `google-scholar-crawler` | Crawl Google Scholar citations |
| `rss-fetcher` | Fetch and process RSS feeds |
| `crossref-cli` | Query Crossref API |

## Python API Usage

```python
from information_composer import (
    PDFValidator,
    DOIDownloader,
    CrossrefClient,
    jsonify,
    markdownify,
)

# Validate PDF
validator = PDFValidator()
is_valid, error = validator.validate_single_pdf("document.pdf")

# Download by DOI
downloader = DOIDownloader()
result = downloader.download_doi("10.1038/nature12373")

# Query Crossref
client = CrossrefClient()
works = client.query_works(query="machine learning", limit=10)

# Process markdown
json_data = jsonify(markdown_content)
markdown_content = markdownify(json_data)
```

## Contributing

We welcome all forms of contributions! Please see the [Contributing Guide](development/contributing.md) for how to participate in project development.

## Version

Current version: **0.3.0**

Minimum Python version: **3.12+**

## License

This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details.

## Support

If you encounter any issues or have questions:

1. Check the relevant documentation
2. Search existing [Issues](https://github.com/yourusername/information-composer/issues)
3. Create a new Issue describing your problem

---

**Information Composer** - Making information processing simpler, smarter, and more efficient!
