# Sourcer - Professional Developer Sourcing Tool

**Automated sourcing pipeline for daily C++ developer recruiting workflow.**

## What's New

🎉 **Version 2.1 - Modular GitHub Scraper + Project System**

- ✅ **Modular GitHub GraphQL scraper** - 10 focused modules, <300 lines each
- ✅ **Project-based workflow** - organize searches by vacancy/project
- ✅ **Advanced tech validation** - L1 (metadata) + L2 (file tree) + L3 (content)
- ✅ **Ethnicity filtering** - ML-based CIS developer identification
- ✅ **LLM-powered matching** - AI candidate analysis and top-N selection
- ✅ **Professional architecture** - organized by function, clean separation
- ✅ **CLI interface** - easy-to-use commands for daily workflow
- ✅ **Global deduplication** - across all sources and projects

## Quick Start

```bash
# 1. Setup API keys
cp config/api_keys.env.example config/api_keys.env
# Edit config/api_keys.env and add your GitHub token

# 2. Search for candidates
python3 sourcer_cli.py search

# 3. View results
python3 sourcer_cli.py stats
python3 sourcer_cli.py export --priority A

# 4. Track outreach
python3 sourcer_cli.py status update username123 --status contacted
```

## Documentation

- **[Quick Start Guide](docs/QUICKSTART.md)** - Get started in 5 minutes
- **[Full Documentation](docs/README.md)** - Complete usage guide
- **[Project Structure](docs/PROJECT_STRUCTURE.md)** - Architecture & file organization
- **[Technical Details](docs/CLAUDE.md)** - For developers

## Features

### 🔍 Multi-Platform Sourcing
- **GitHub GraphQL scraper** (active) - Modular architecture with advanced filtering
  - L1/L2/L3 tech stack validation
  - CIS location + ethnicity filtering
  - Project-based organization
- LinkedIn integration (planned)
- Stack Overflow integration (planned)

### 🎯 Smart Prioritization
- Skill matching with custom weights
- Location scoring for relocation probability
- Activity level filtering (avoid celebrities)
- Reachability scoring (email availability)

### 📊 Campaign Management
```bash
python3 sourcer_cli.py campaign create "feb-2026-senior-cpp"
python3 sourcer_cli.py search --campaign "feb-2026-senior-cpp"
python3 sourcer_cli.py campaign list
```

### 📈 Status Tracking
```bash
python3 sourcer_cli.py status update user1 --status contacted
python3 sourcer_cli.py status list --filter replied
```

### 📑 Analytics & Reports
```bash
python3 sourcer_cli.py stats
python3 src/utils/analytics.py  # Detailed report with funnels
```

## CLI Commands

| Command | Description |
|---------|-------------|
| `search` | Search for new candidates |
| `campaign` | Manage campaigns (create, list, show) |
| `status` | Track candidate interactions |
| `export` | Export candidates by priority |
| `stats` | Show statistics |
| `cleanup` | Remove old temporary files |
| `blacklist` | Manage exclusion list |

Run `python3 sourcer_cli.py --help` for full command list.

## Project Structure

```
sourcer/
├── sourcer_cli.py              # Main CLI
├── src/
│   ├── scrapers/
│   │   └── github_graphql/     # Modular GitHub scraper (10 files)
│   ├── processors/             # Scoring, enrichment, AI, LLM analysis
│   └── utils/                  # Core utilities
├── projects/                   # Project-based organization
│   └── <project-name>/
│       ├── raw/github/         # Raw scraper results
│       ├── scored/             # Scored candidates
│       └── exports/            # Final exports
├── data/
│   └── master/                 # Global master database
├── config/                     # Configuration
├── docs/                       # Documentation
└── logs/                       # Logs

See [PROJECT_STRUCTURE.md](docs/PROJECT_STRUCTURE.md) for details.
```

## Daily Workflow

**Morning - Search:**
```bash
python3 sourcer_cli.py search --profile cpp_backend
```

**Afternoon - Review & Outreach:**
```bash
python3 sourcer_cli.py export --priority A
python3 sourcer_cli.py status update user1 --status contacted
```

**Weekly - Analytics:**
```bash
python3 src/utils/analytics.py
```

## Configuration

### Search Profiles

Edit [config/search_profiles.json](config/search_profiles.json):

- **cpp_backend** - Async I/O, gRPC, distributed systems
- **cpp_game_dev** - Game servers, multiplayer, Unreal/Unity
- **cocos2d_dev** - Cocos2d-x mobile game developers

Add your own profiles with custom keywords and weights.

### API Keys

Required:
- **GitHub Token** - https://github.com/settings/tokens
- **OpenRouter API** - https://openrouter.ai (for AI outreach)

## Requirements

```bash
pip install requests
```

That's it! No heavy dependencies.

## Data Files

| File | Purpose |
|------|---------|
| `data/master/candidates_master.csv` | Main database (all candidates) |
| `data/master/candidate_status.csv` | Status tracking |
| `data/master/processed_ids.txt` | Deduplication list |

All data stays local - full control over your candidate database.

## Troubleshooting

**Issue: No candidates found**
- Check API keys in `config/api_keys.env`
- Verify GitHub token has correct permissions

**Issue: Duplicates appearing**
- System automatically deduplicates by username
- Check `data/master/processed_ids.txt`

**Issue: Rate limits**
- GitHub: 5000 req/hour with token
- Add delays in scraper if needed

## Migration from Old Structure

Old files have been moved:
- Scripts → `src/scrapers/` and `src/processors/`
- `candidates_master.csv` → `data/master/`
- `processed_candidates.txt` → `data/master/processed_ids.txt`
- Documentation → `docs/`

Your data is safe and compatible with the new CLI.

## What's Next

Planned features:
- [ ] LinkedIn scraper integration
- [ ] Stack Overflow scraper
- [ ] Email sending integration
- [ ] Web dashboard
- [ ] Automated follow-ups
- [ ] Team collaboration features

## License

MIT - Use as you wish, respect platform ToS.

---

**Happy Sourcing! 🚀**

For questions, see [docs/README.md](docs/README.md) or [docs/QUICKSTART.md](docs/QUICKSTART.md)
