# _sources/ Directory

This directory contains the source code and compilation infrastructure for TPC benchmark
data generation tools. The compiled binaries are stored in `_binaries/` for distribution.

## Directory Structure

```
_sources/
├── README.md                    # This file
├── compilation/                 # Compilation infrastructure
│   ├── docker/                  # Docker images for cross-compilation
│   │   ├── Dockerfile.linux-arm64
│   │   ├── Dockerfile.linux-x86_64
│   │   ├── Dockerfile.windows-arm64
│   │   └── Dockerfile.windows-x86_64
│   └── scripts/
│       └── compile-all-platforms.sh
├── tpc-h/                       # TPC-H source (dbgen/qgen)
│   ├── EULA.txt                 # TPC EULA - must accept to use
│   ├── PATCHES.md               # Documentation of applied patches
│   ├── stdout-support.patch     # Patch for -z stdout output
│   ├── dbgen/                   # Data generator source
│   └── ref_data/                # Reference data files
├── tpc-ds/                      # TPC-DS source (dsdgen/dsqgen)
│   ├── EULA.txt                 # TPC EULA - must accept to use
│   ├── PATCHES.md               # Documentation of applied patches
│   ├── stdout-support.patch     # Patch for FILTER stdout output
│   ├── tools/                   # Data/query generator source
│   ├── query_templates/         # SQL query templates
│   └── answer_sets/             # Reference answer sets
└── join-order-benchmark/        # JOB benchmark queries (git submodule)
```

## Obtaining TPC Source Code

The TPC source code is distributed under the TPC End User License Agreement (EULA).
By using BenchBox with TPC benchmarks, you accept the TPC EULA terms.

### TPC-H Source

1. Visit https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp
2. Select "TPC-H" from the benchmark list
3. Download the "TPC-H Tools" package (requires free TPC account)
4. Extract to `_sources/tpc-h/`

The expected structure after extraction:
```
_sources/tpc-h/
├── EULA.txt
├── dbgen/
│   ├── bcd2.c
│   ├── bm_utils.c
│   ├── build.c
│   ├── config.h
│   ├── dists.dss
│   ├── driver.c
│   ├── dss.h
│   ├── makefile.suite
│   ├── print.c
│   ├── rnd.c
│   └── ... (other source files)
├── ref_data/
└── specification.pdf
```

### TPC-DS Source

1. Visit https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp
2. Select "TPC-DS" from the benchmark list
3. Download the "TPC-DS Tools" package (requires free TPC account)
4. Extract to `_sources/tpc-ds/`

The expected structure after extraction:
```
_sources/tpc-ds/
├── EULA.txt
├── tools/
│   ├── address.c
│   ├── build_support.c
│   ├── config.h
│   ├── dsdgen.c
│   ├── genrand.c
│   ├── makefile
│   ├── Makefile.suite
│   ├── params.h
│   ├── porting.h
│   ├── print.c
│   ├── tpcds.dst
│   └── ... (other source files)
├── query_templates/
├── query_variants/
└── specification/
```

## Applying BenchBox Patches

BenchBox includes patches that add features and fix compilation issues:

### TPC-H Patches

The `stdout-support.patch` adds a `-z` flag for streaming data to stdout:

```bash
cd _sources/tpc-h
patch -p1 < stdout-support.patch
```

**Changes applied:**
- Adds `-z` flag to stream generated data to stdout (enables piping to compression)
- Fixes format specifier bug in money printing (`%ld` → `%d`)

See `_sources/tpc-h/PATCHES.md` for detailed documentation.

### TPC-DS Patches

The `stdout-support.patch` fixes the broken `-FILTER Y` flag and adds Linux compatibility:

```bash
cd _sources/tpc-ds
patch -p1 < stdout-support.patch
```

**Changes applied:**
- Fixes `-FILTER Y` flag for stdout output (was broken in upstream)
- Adds `MAXINT` definition for Linux builds
- Adds `-fcommon` flag for GCC 10+ compatibility

See `_sources/tpc-ds/PATCHES.md` for detailed documentation.

### Verifying Patches

After applying patches, verify they were applied correctly:

```bash
# TPC-H: Check for -z flag
grep -n "zstdout" _sources/tpc-h/dbgen/dss.h
# Expected: "EXTERN int  zstdout;"

# TPC-DS: Check FILTER fix
grep -n '"FILTER"' _sources/tpc-ds/tools/params.h
# Expected: "FILTER" without leading underscore
```

## Compiling Binaries

### Prerequisites

**macOS (native compilation):**
- Xcode Command Line Tools: `xcode-select --install`
- For cross-compilation: `brew install make` (optional)

**Linux/Windows (Docker-based):**
- Docker: https://docs.docker.com/get-docker/
- Docker Buildx for multi-platform builds

### Quick Compilation (macOS ARM64 only)

For quick local development on Apple Silicon:

```bash
# TPC-H
cd _sources/tpc-h/dbgen
make -f makefile.suite CC=clang MACHINE=LINUX DATABASE=ORACLE WORKLOAD=TPCH
# Output: dbgen, qgen

# TPC-DS (requires patches applied first)
cd _sources/tpc-ds/tools
make CC=clang CFLAGS="-O2 -DMACOS -DMAXINT=INT_MAX -fcommon"
# Output: dsdgen, dsqgen, distcomp
```

### Full Multi-Platform Compilation

The `compile-all-platforms.sh` script builds binaries for all supported platforms:

```bash
cd _sources/compilation/scripts
./compile-all-platforms.sh
```

**Supported platforms:**
| Platform | Architecture | Method |
|----------|--------------|--------|
| darwin | arm64 | Native (on Apple Silicon) |
| darwin | x86_64 | Cross-compilation |
| linux | x86_64 | Docker |
| linux | arm64 | Docker |
| windows | x86_64 | Docker + MinGW |
| windows | arm64 | Docker + MinGW |

**Output location:** `_binaries/tpc-{h,ds}/{platform}-{arch}/`

### Docker Images

Build Docker images for cross-compilation:

```bash
# Linux x86_64
docker build -f _sources/compilation/docker/Dockerfile.linux-x86_64 \
    -t benchbox/tpc-linux-x86_64 _sources/compilation/docker/

# Linux ARM64
docker build -f _sources/compilation/docker/Dockerfile.linux-arm64 \
    -t benchbox/tpc-linux-arm64 _sources/compilation/docker/
```

## Binary Verification

After compilation, verify binaries work correctly:

```bash
# TPC-H: Generate small dataset
_binaries/tpc-h/darwin-arm64/dbgen -s 0.01 -f
ls -la *.tbl  # Should create customer.tbl, orders.tbl, etc.

# TPC-H: Test stdout mode
_binaries/tpc-h/darwin-arm64/dbgen -z -s 0.01 -T c | head -5

# TPC-DS: Generate small dataset (scale must be >= 1)
_binaries/tpc-ds/darwin-arm64/dsdgen -SCALE 1 -TABLE ship_mode -DIR /tmp
cat /tmp/ship_mode.dat | head -5

# TPC-DS: Test stdout mode
_binaries/tpc-ds/darwin-arm64/dsdgen -SCALE 1 -TABLE ship_mode -FILTER Y | head -5
```

## Important Constraints

### TPC-DS Scale Factor

**TPC-DS requires scale factor >= 1.** Fractional scale factors (e.g., 0.01) cause
segmentation faults in the official dsdgen binary. This is a known upstream limitation.

BenchBox handles this by generating SF1 data and sampling for smaller test datasets.

### File Sizes

Generated data sizes at various scale factors:

| Benchmark | SF 0.01 | SF 1 | SF 10 | SF 100 |
|-----------|---------|------|-------|--------|
| TPC-H | ~10 MB | ~1 GB | ~10 GB | ~100 GB |
| TPC-DS | N/A | ~1 GB | ~10 GB | ~100 GB |

### Thread Safety

Neither dbgen nor dsdgen are thread-safe for parallel generation of the same table.
Use the `-C` (chunks) and `-S` (step) flags for parallel generation:

```bash
# Generate customer table in 4 parallel chunks
for i in 1 2 3 4; do
    dbgen -s 10 -C 4 -S $i -T c &
done
wait
cat customer.tbl.* > customer.tbl
```

## Git Tracking Strategy

The `_sources/` directory is structured for selective git tracking:

**Tracked in git (included in releases):**
- `_sources/compilation/` - Docker configs and build scripts
- `_sources/tpc-*/PATCHES.md` - Patch documentation
- `_sources/tpc-*/stdout-support.patch` - Patch files
- `_sources/tpc-ds/query_templates/` - SQL templates

**Tracked but can be refreshed from upstream:**
- `_sources/tpc-h/dbgen/` - TPC-H source code
- `_sources/tpc-ds/tools/` - TPC-DS source code
- `_sources/tpc-*/EULA.txt` - License files

**NOT tracked (gitignored):**
- `_sources/tpc-*/tools/*.o` - Object files
- `_sources/tpc-*/tools/tpcds.idx*` - Generated index files
- `_sources/join-order-benchmark/` - Separate git repo

## Updating TPC Source

When a new TPC specification is released:

1. Download new source from TPC website
2. Extract over existing directory (preserving patches)
3. Re-apply patches: `patch -p1 < stdout-support.patch`
4. Verify patches applied: Check `PATCHES.md` verification commands
5. Recompile: `./compile-all-platforms.sh`
6. Run tests: `uv run pytest tests/integration/test_tpc*.py`
7. Update version references in documentation

## Troubleshooting

### "MAXINT undefined" error (TPC-DS)
Ensure patches are applied: `patch -p1 < stdout-support.patch`

### "multiple definition" errors (TPC-DS, GCC 10+)
Ensure `-fcommon` flag is in CFLAGS (handled by patch)

### Segfault with small scale factors (TPC-DS)
TPC-DS requires SF >= 1. Use SF 1 and sample for smaller datasets.

### "FILTER unknown" error (TPC-DS)
Patch not applied correctly. Re-apply: `patch -p1 < stdout-support.patch`

### Missing `dists.dss` or `tpcds.dst`
These distribution files must be in the same directory as the binary, or specified
via the `-b` flag (TPC-H) or environment (TPC-DS).

## Related Documentation

- `_binaries/README.md` - Pre-compiled binary distribution
- `AGENTS.md` - Project architecture and conventions
- `benchbox/core/tpch/` - TPC-H benchmark implementation
- `benchbox/core/tpcds/` - TPC-DS benchmark implementation
