Metadata-Version: 2.4
Name: dataform-view-migrator
Version: 0.1.5
Summary: Migrate BigQuery views into Dataform view definitions.
Author: Alan Vainsencher
License: MIT
License-File: LICENSE
Keywords: bigquery,bigquery-views,dataform
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Requires-Dist: google-cloud-bigquery>=3.25.0
Requires-Dist: polars>=1.12.0
Requires-Dist: tomli>=2.0.1; python_version < '3.11'
Requires-Dist: typer>=0.12.0
Description-Content-Type: text/markdown

# dataform-view-migrator

Export BigQuery VIEW SQL definitions into a Dataform project with a predictable layout and safe write policies.

This README focuses on using the tool. For development setup and project conventions, see DEVELOPMENT.md.

📦 PyPI: https://pypi.org/project/dataform-view-migrator/
💻 GitHub: https://github.com/elvainch/dataform-view-migrator

## Prerequisites
- Python 3.10+
- Auth via Application Default Credentials (ADC):
  - Set `GOOGLE_APPLICATION_CREDENTIALS` to a service account JSON, or
  - Run `gcloud auth application-default login`

## Install
```bash
pip install dataform-view-migrator
```

## Commands

- `dataform-view-migrator` (or `python -m dataform_view_migrator`): prints quick usage and version
- `dataform-view-migrator ping-bq [--project <id>] [--location <REGION>] [--config <toml>]`
  - Verifies auth and prints the resolved BigQuery project. If `--location` is provided (or set in TOML), it performs a regional INFORMATION_SCHEMA dry-run; otherwise it lists datasets to verify access.
- `dataform-view-migrator migrate-views [options]`
  - Discovers BigQuery views and writes files into the Dataform repo path.

Run `--help` on any command for complete options.

## Configuration

- Copy `dataform_view_migrator.example.toml` to `dataform_view_migrator.toml` and edit.
- CLI flags override TOML values when explicitly provided.

Key options (flag → TOML):
- `--source-project` → `source_project` (required via config or flag)
- `--dest <path>` → `dest` (required via config or flag)
- `--datasets a,b` → `datasets_include = ["a","b"]`
- `--exclude-datasets x,y` → `datasets_exclude = ["x","y"]`
- `--location US` → `location = "US"`
- `--ext sql|sqlx` → `ext = "sqlx"` (default `sqlx`)
- `--overwrite skip|backup|force` → `overwrite = "skip"` (default `skip`)
- `--add-dataform-header/--no-add-dataform-header` → `add_dataform_header = true|false`
- `--dry-run/--no-dry-run` → `dry_run = false`
- Optional dataset folder remapping: `dataset_folders = { src = "src_views" }`

## Output Layout

- Files are written under `dest/<dataset>/<view_name>.<ext>` by default.
- Use `dataset_folders` in TOML to map dataset names to custom subfolders.
- Extension `ext` is `sqlx` by default; set `--ext sql` for plain SQL files.

### Dataform Header (optional)
When enabled, each file is prefixed with a Dataform config block:

```
config {
  type: "view",
  schema: "<dataset>",
  name: "<view>"
  # optional fields below when configured
  # description: "...",
  # tags: ["..."]
}
```

Control via `--add-dataform-header` (or `add_dataform_header = true`).
Customize with `dataform_header.description` and `dataform_header.tags` in TOML.

## Overwrite Policy

- `skip` (default): do not modify existing files; they are reported as `skipped`.
- `backup`: if a file exists, move it to `<name>.<ext>.bak[.N]` and write the new content.
- `force`: overwrite files in place.

All modes support `--dry-run` to preview actions as `would-create`, `would-update`, or `would-skip` without writing.

## Examples

- Verify auth/project using TOML defaults:
  - `uv run dataform-view-migrator ping-bq --config dataform_view_migrator.toml`
- Migrate all views to a Dataform repo (US region, backup existing):
  - `uv run dataform-view-migrator migrate-views --source-project my-proj --dest ../dataform --location US --overwrite backup`
- Include specific datasets and emit plain SQL:
  - `uv run dataform-view-migrator migrate-views --source-project my-proj --dest ../dataform --datasets sales,finance --ext sql`
- Dry-run to review changes only:
  - `uv run dataform-view-migrator migrate-views --config dataform_view_migrator.toml --dry-run`

## Output Report

After migration, a table of per-view results is printed (dataset, view, action, path, error) followed by a compact summary grouped by action. A non-zero exit code is returned if any views fail.

## Development

See DEVELOPMENT.md for environment setup, linting/formatting, tests, and project layout.

