# Information Composer Project Overview

## Project Introduction

Information Composer is a comprehensive toolkit for collecting, composing, and filtering information from various web resources. It integrates multiple powerful modules designed specifically for academic research, document management, and information processing.

## Core Features

### PDF Processing
- **Format Validation**: Verify PDF file format and integrity
- **Batch Processing**: Support for batch validation of multiple PDF files
- **Error Reporting**: Detailed error information and statistics
- **Recursive Search**: Support for recursive directory search

### Markdown Processing
- **Format Conversion**: Convert between Markdown and JSON
- **Content Extraction**: Extract headings, links, images, tables
- **Format Cleaning**: Clean and standardize Markdown format
- **Metadata Extraction**: Extract document metadata

### Academic Literature Processing
- **PubMed Integration**: Query and process PubMed literature data
- **DOI Management**: Download and manage DOI references
- **Crossref Integration**: Query Crossref API for bibliographic data
- **Keyword Filtering**: Filter literature based on keywords
- **Batch Processing**: Support for batch literature processing

### AI-Powered Filtering
- **LLM Integration**: Use Large Language Models for intelligent filtering
- **Content Analysis**: Smart analysis of document content
- **Automatic Classification**: Auto-classify and tag documents
- **Quality Assessment**: Evaluate document quality and relevance

### Web Scraping & Data Collection
- **Google Scholar**: Crawl and process Google Scholar citations
- **RSS Feeds**: Parse and manage scientific RSS feeds
- **RiceDataCN**: Extract gene data from RiceDataCN database
- **Custom Crawlers**: Extensible web scraping framework

## Project Architecture

```
information-composer/
├── src/information_composer/
│   ├── pdf/                    # PDF processing module
│   │   ├── validator.py        # PDF validator
│   │   └── cli/                # PDF CLI tool
│   ├── markdown/               # Markdown processing module
│   │   ├── markdown.py         # Core processing functionality
│   │   └── vendor/             # Third-party libraries
│   ├── pubmed/                 # PubMed integration module
│   │   ├── pubmed.py           # Query functionality
│   │   ├── baseline.py         # Baseline data
│   │   ├── analyzer/           # Literature analyzer
│   │   └── cli/                # PubMed CLI tool
│   ├── core/                   # Core functionality module
│   │   ├── doi_downloader.py   # DOI downloader
│   │   └── downloader.py       # Generic downloader
│   ├── crossref/               # Crossref integration module
│   │   ├── core/               # Core client and models
│   │   └── cli/                # Crossref CLI tool
│   ├── llm_filter/             # LLM filtering module
│   │   ├── core/               # Core filtering logic
│   │   ├── llm/                # LLM interface (DashScope, Ollama, OpenAI)
│   │   ├── config/             # Configuration management
│   │   └── cli/                # LLM filter CLI tool
│   ├── rss/                    # RSS feed processing module
│   │   ├── fetcher.py          # RSS feed fetcher
│   │   ├── parser.py           # RSS parser
│   │   ├── config.py           # RSS configuration
│   │   └── cli/                # RSS CLI tool
│   └── sites/                  # Website scraping module
│       ├── base.py             # Base collector class
│       ├── google_scholar/     # Google Scholar scraper
│       └── ricedatacn_gene_parser.py  # RiceDataCN parser
├── examples/                   # Usage examples
├── scripts/                    # Utility scripts
│   └── check_code.py           # Code quality check script
├── docs/                       # Project documentation
└── tests/                      # Test files
```

## Technology Stack

### Core Dependencies
- **Python 3.12+**: Primary programming language (3.12 is minimum)
- **pypdfium2**: PDF processing library
- **biopython**: PubMed API access
- **habanero**: Crossref API client
- **requests**: HTTP request library
- **beautifulsoup4**: HTML/XML parsing
- **pandas**: Data manipulation

### AI/ML Dependencies
- **dashscope**: Alibaba Cloud Large Language Model API
- **llama-index**: Large Language Model integration framework
- **langchain**: Language chain framework for PubMed Analyzer
- **ollama**: Local LLM support

### Development Tools
- **Ruff**: Fast Python linter and formatter (primary tool)
- **pytest**: Testing framework
- **mypy**: Static type checking
- **GitHub Actions**: CI/CD

### Supported Formats
- **PDF**: Document validation and processing
- **Markdown**: Document format conversion
- **JSON**: Data exchange format
- **XML**: PubMed data format
- **TXT**: Plain text processing
- **RSS**: Feed processing

## Available CLI Commands

| Command | Description |
|---------|-------------|
| `pdf-validator` | Validate PDF files |
| `md-llm-filter` | Filter markdown with LLM |
| `pubmed-cli` | Search and fetch PubMed data |
| `google-scholar-crawler` | Crawl Google Scholar citations |
| `rss-fetcher` | Fetch and process RSS feeds |
| `crossref-cli` | Query Crossref API |

## Functional Modules

### 1. PDF Validator
- File format validation
- Integrity checking
- Page statistics
- Error reporting
- Batch processing

### 2. Markdown Processor
- Format conversion
- Content extraction
- Format cleaning
- Metadata extraction
- Link validation

### 3. PubMed Integration
- Literature search
- Data parsing
- Keyword filtering
- Batch processing
- Cache management
- Literature analysis (AI-powered)

### 4. DOI Manager
- DOI download
- Metadata extraction
- Batch processing
- Format conversion
- Error handling

### 5. Crossref Integration
- Query by DOI
- Search works
- Convert PMID to DOI
- Convert DOI to PMID
- Batch processing

### 6. LLM Filter
- Intelligent filtering
- Content analysis
- Automatic classification
- Quality assessment
- Batch processing

### 7. RSS Fetcher
- Feed parsing
- Async fetching
- Cache management
- Batch processing
- Format conversion

### 8. Google Scholar Crawler
- Search configuration
- Result processing
- Export options
- Duplicate removal
- Rate limiting

## Configuration Options

### Environment Variables
```bash
# Required
export DASHSCOPE_API_KEY="your-api-key"

# Optional
export MAX_CONCURRENT_REQUESTS=5
export REQUEST_TIMEOUT=30
export ENABLE_CACHE=true
```

### Configuration File
```yaml
# config.yaml
llm:
  api_key: "your-api-key"
  model: "qwen-plus"
  max_concurrent_requests: 5

processing:
  max_file_size_mb: 10
  supported_formats: ["pdf", "md", "txt"]

pubmed:
  email: "your-email@example.com"
  batch_size: 100
```

## Quality Assurance

### Code Quality
- Ruff code checks
- Type annotations
- Docstrings
- Unit tests

### CI/CD
- GitHub Actions integration
- Multi Python version testing (3.12, 3.13)
- Automated code checks
- Automated publishing

### Test Coverage
- Unit tests
- Integration tests
- End-to-end tests
- Performance tests
- **Current coverage: 51%+**

## Performance Features

### Concurrent Processing
- Multi-threading support
- Async processing
- Batch operations
- Resource management

### Caching Mechanisms
- Intelligent caching
- Expiration management
- Storage optimization
- Performance improvement

### Error Handling
- Graceful degradation
- Retry mechanisms
- Error recovery
- Logging

## Use Cases

### Academic Research
- Literature collection and organization
- Document quality checking
- Content filtering and analysis
- Data standardization

### Document Management
- Batch document processing
- Format conversion
- Content extraction
- Quality assessment

### Information Processing
- Data cleaning
- Format standardization
- Content filtering
- Intelligent analysis

## Development Standards

### Code Standards
- Follow PEP 8
- Use type annotations
- Write docstrings
- Add unit tests

### Testing Requirements
- All new features must have tests
- Test coverage target: 70%+
- Pass all code checks
- Pass CI/CD checks

## Documentation Structure

```
docs/
├── README.md              # Documentation navigation
├── overview.md            # Project overview
├── installation.md        # Installation guide
├── quickstart.md          # Quick start guide
├── configuration.md       # Configuration options
├── guides/                # Feature guides
│   ├── pdf-validator.md
│   ├── markdown-processing.md
│   ├── pubmed-integration.md
│   ├── crossref-integration.md
│   ├── google-scholar-integration.md
│   ├── rss-integration.md
│   ├── doi-download.md
│   └── llm-filtering.md
├── api/                   # API reference
├── examples/              # Code examples
└── development/           # Development guide
```

## Support and Help

### Getting Help
1. Check the [documentation](index.md)
2. Search [Issues](https://github.com/yourusername/information-composer/issues)
3. Create a new Issue
4. Join discussions

### Common Questions
- Check the [FAQ](quickstart.md#faq)
- Search existing issues
- Check error logs
- Verify configuration settings

---

**Information Composer** - Making information processing simpler, smarter, and more efficient!
