# Information Composer

<div align="center">

**A comprehensive toolkit for collecting, composing, and filtering information from various web resources with AI-powered markdown processing**

[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Documentation Status](https://readthedocs.org/projects/information-composer/badge/?version=latest)](https://information-composer.readthedocs.io/en/latest/?badge=latest)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

Version 0.3.0

[Installation](installation.md){ .md-button .md-button--primary }
[Quick Start](quickstart.md){ .md-button }
[API Reference](api/){ .md-button }

</div>

## Features

### Core Modules

#### PDF Processing
- **Format Validation**: Verify PDF file integrity and format
- **Batch Processing**: Process multiple PDFs efficiently
- **Error Reporting**: Detailed error information and statistics

#### Markdown Processing
- **Format Conversion**: Convert between Markdown and JSON
- **Content Extraction**: Extract headings, links, images, tables
- **LLM Filtering**: AI-powered content filtering and analysis

#### Academic Literature Processing
- **PubMed Integration**: Query and process PubMed literature data
- **DOI Management**: Download and manage academic papers by DOI
- **Crossref Integration**: Query Crossref API for bibliographic data

### AI-Powered Features
- **LLM Integration**: Support for DashScope, Ollama, and OpenAI
- **Content Analysis**: Smart analysis of document content
- **Automatic Classification**: Auto-classify and tag documents
- **Quality Assessment**: Evaluate document quality and relevance

### Web Scraping & Data Collection
- **Google Scholar**: Crawl and process Google Scholar citations
- **RSS Feeds**: Parse and manage scientific RSS feeds
- **RiceDataCN**: Extract gene data from RiceDataCN database
- **Custom Crawlers**: Extensible web scraping framework

## Architecture

```mermaid
graph TB
    A[Information Composer] --> B[PDF Processing]
    A --> C[Markdown Processing]
    A --> D[Academic Integration]
    A --> E[AI Filtering]
    A --> F[Web Scraping]
    A --> G[Data Collection]

    B --> B1[PDF Validator]
    B --> B2[Batch Processing]

    C --> C1[Format Conversion]
    C --> C2[LLM Filtering]

    D --> D1[PubMed API]
    D --> D2[DOI Downloader]
    D --> D3[Crossref API]

    E --> E1[DashScope]
    E --> E2[Ollama]
    E --> E3[OpenAI]

    F --> F1[Google Scholar]
    F --> F2[RiceDataCN]

    G --> G1[RSS Fetcher]
    G --> G2[Cache Management]
```

## Quick Start

### Installation

```bash
# Clone the repository
git clone https://github.com/yourusername/information-composer.git
cd information-composer

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
# or
.venv\Scripts\activate     # Windows

# Install the package
pip install -e .
```

### Basic Usage

```bash
# Validate PDF files
pdf-validator document.pdf

# Filter markdown with LLM
md-llm-filter -i input.md -o output.md

# Search PubMed
pubmed-cli search "machine learning" -e your@email.com

# Query Crossref
crossref-cli query --doi "10.1038/nature12373"

# Crawl Google Scholar
google-scholar-crawler -q "deep learning" -n 20

# Fetch RSS feeds
rss-fetcher -u "https://example.com/feed.xml" -o output.json
```

### Python API

```python
from information_composer import (
    PDFValidator,
    DOIDownloader,
    CrossrefClient,
    jsonify,
    markdownify,
)

# Validate PDF
validator = PDFValidator()
is_valid, error = validator.validate_single_pdf("document.pdf")

# Download by DOI
downloader = DOIDownloader()
result = downloader.download_doi("10.1038/nature12373")

# Query Crossref
client = CrossrefClient()
works = client.query_works(query="machine learning", limit=10)

# Process markdown
json_data = jsonify(markdown_content)
markdown_content = markdownify(json_data)
```

## Documentation

### User Guides
- [Installation Guide](installation.md) - Setup and configuration
- [Quick Start](quickstart.md) - Get started in 5 minutes
- [Configuration](configuration.md) - Detailed configuration options

### Feature Guides
- [PDF Validation](guides/pdf-validator.md) - PDF processing and validation
- [Markdown Processing](guides/markdown-processing.md) - Markdown tools and utilities
- [DOI Download](guides/doi-download.md) - Download academic papers
- [PubMed Integration](guides/pubmed-integration.md) - Academic literature search
- [Crossref Integration](guides/crossref-integration.md) - Crossref API queries
- [Google Scholar](guides/google-scholar-integration.md) - Web scraping
- [RSS Processing](guides/rss-integration.md) - RSS feed fetching
- [LLM Filtering](guides/llm-filtering.md) - AI-powered content filtering
- [RiceDataCN Integration](guides/ricedatacn-integration.md) - Rice gene data parsing

### API Reference
- [Core API](api/) - Core functionality
- [PDF API](api/pdf/) - PDF processing
- [Markdown API](api/markdown/) - Markdown processing
- [PubMed API](api/pubmed/) - Academic literature
- [Crossref API](api/crossref/) - Crossref integration
- [RSS API](api/rss/) - RSS feed processing
- [Sites API](api/sites/) - Web scraping
- [LLM Filter API](api/llm_filter/) - AI filtering

## Development

### Code Quality
- **Ruff**: Fast Python linter and formatter (primary tool)
- **MyPy**: Static type checking
- **Pytest**: Comprehensive testing framework
- **Coverage**: Code coverage analysis (51%+)

### Contributing
- [Contributing Guide](development/contributing.md) - How to contribute
- [Code Quality](development/code-quality.md) - Development standards
- [Changelog](development/changelog.md) - Version history

## Performance

- **Concurrent Processing**: Multi-threaded operations
- **Caching**: Intelligent caching for expensive operations
- **Memory Efficient**: Optimized for large datasets
- **Error Recovery**: Graceful handling of failures

## Use Cases

### Academic Research
- Literature collection and organization
- Document quality validation
- Content filtering and analysis
- Data standardization

### Document Management
- Batch document processing
- Format conversion
- Content extraction
- Quality assessment

### Information Processing
- Data cleaning
- Format standardization
- Content filtering
- Intelligent analysis

## Community

- **GitHub**: [information-composer](https://github.com/yourusername/information-composer)
- **Issues**: [Report bugs or request features](https://github.com/yourusername/information-composer/issues)
- **Discussions**: [Community discussions](https://github.com/yourusername/information-composer/discussions)

## License

This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details.

## Acknowledgments

- [MkDocs Material](https://squidfunk.github.io/mkdocs-material/) for documentation
- [pypdfium2](https://github.com/pypdfium2-team/pypdfium2) for PDF processing
- [DashScope](https://dashscope.aliyun.com/) for LLM integration
- [Crossref](https://www.crossref.org/) for bibliographic data
- [PubMed](https://pubmed.ncbi.nlm.nih.gov/) for academic literature

---

<div align="center">

**Information Composer** - Making information processing simpler, smarter, and more efficient!

[Get Started Now](quickstart.md){ .md-button .md-button--primary }

</div>
