Metadata-Version: 2.2
Name: mozilla-bigquery-etl
Version: 2025.2.1
Summary: Tooling for building derived datasets in BigQuery
Author-email: Mozilla Corporation <fx-data-dev@mozilla.org>
Project-URL: Homepage, https://github.com/mozilla/bigquery-etl
Project-URL: Issues, https://github.com/mozilla/bigquery-etl/issues
Requires-Python: <3.12,>=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: attrs==25.1.0
Requires-Dist: authlib==1.4.1
Requires-Dist: bigeye-sdk==0.4.97
Requires-Dist: black==25.1.0
Requires-Dist: cattrs==24.1.2
Requires-Dist: click==8.1.8
Requires-Dist: cryptography==44.0.1
Requires-Dist: exceptiongroup==1.2.2
Requires-Dist: flake8<5
Requires-Dist: gcsfs==2025.2.0
Requires-Dist: gcloud==0.18.3
Requires-Dist: gitpython==3.1.44
Requires-Dist: google-auth>=2.30.0
Requires-Dist: google-cloud-bigquery==3.29.0
Requires-Dist: google-cloud-bigquery-storage[fastavro]==2.28.0
Requires-Dist: google-cloud-datacatalog-lineage==0.3.12
Requires-Dist: google-cloud-storage==3.0.0
Requires-Dist: Jinja2==3.1.5
Requires-Dist: jsonschema==4.23.0
Requires-Dist: PyJWT==2.10.1
Requires-Dist: markdown-include==0.8.1
Requires-Dist: mdx_truly_sane_lists==1.3
Requires-Dist: mkdocs==1.6.1
Requires-Dist: mkdocs-material==9.6.4
Requires-Dist: mkdocs-awesome-pages-plugin==2.9.3
Requires-Dist: mozilla-metric-config-parser==2024.11.1
Requires-Dist: mozilla-schema-generator==0.5.1
Requires-Dist: pandas==2.2.3
Requires-Dist: pathos==0.3.3
Requires-Dist: pip-tools==7.4.1
Requires-Dist: pre-commit==4.1.0
Requires-Dist: pyarrow==19.0.0
Requires-Dist: pytest-black==0.6.0
Requires-Dist: pytest-flake8==1.3.0
Requires-Dist: pytest-isort==4.0.0
Requires-Dist: pytest-mypy==0.10.3
Requires-Dist: pytest-pydocstyle==2.4.0
Requires-Dist: pytest-xdist==3.6.1
Requires-Dist: pytest==7.4.3
Requires-Dist: PyYAML==6.0.2
Requires-Dist: rich-click==1.8.5
Requires-Dist: smart_open==6.4.0
Requires-Dist: sqlglot==25.28.0
Requires-Dist: sqlparse==0.5.3
Requires-Dist: stripe==6.4.0
Requires-Dist: symbolic==12.13.4
Requires-Dist: siggen==2.2.20241029
Requires-Dist: tomli==2.2.1
Requires-Dist: types-python-dateutil==2.9.0.20241206
Requires-Dist: types-pytz==2025.1.0.20250204
Requires-Dist: types-PyYAML==6.0.12.20241230
Requires-Dist: types-requests==2.32.0.20241016
Requires-Dist: types-ujson==5.10.0.20240515
Requires-Dist: typing==3.7.4.3
Requires-Dist: ujson==5.10.0
Requires-Dist: yamllint==1.35.1
Requires-Dist: requests==2.32.3
Requires-Dist: responses==0.25.6

[![CircleCI](https://dl.circleci.com/status-badge/img/gh/mozilla/bigquery-etl/tree/main.svg?style=svg&circle-token=1df4cefd991043d7d3f13243ea80f38e7aa18341)](https://dl.circleci.com/status-badge/redirect/gh/mozilla/bigquery-etl/tree/main)
# BigQuery ETL

This repository contains Mozilla Data Team's:

- Derived ETL jobs that do not require a custom container
- User-defined functions (UDFs)
- Airflow DAGs for scheduled bigquery-etl queries
- Tools for query & UDF deployment, management and scheduling

For more information, see [https://mozilla.github.io/bigquery-etl/](https://mozilla.github.io/bigquery-etl/)

## Quick Start

### Pre-requisites
- **Pyenv** (optional) Recommended if you want to install different versions of python, see instructions [here](https://github.com/pyenv/pyenv#basic-github-checkout). After the installation of pyenv, make sure that your terminal app is [configured to run the shell as a login shell](https://github.com/pyenv/pyenv/wiki/MacOS-login-shell).
- **Homebrew** (not required, but useful for Mac) - Follow the instructions [here](https://brew.sh/) to install homebrew on your Mac.
- **Python 3.11** - (see [this guide](https://docs.python-guide.org/starting/install3/osx/) for instructions if you're on a mac and haven't installed anything other than the default system Python).

### GCP CLI tools

- **For Mozilla Employees (not in Data Engineering)** - Set up GCP command line tools, [as described on docs.telemetry.mozilla.org](https://docs.telemetry.mozilla.org/cookbooks/bigquery/access.html#using-the-bq-command-line-tool). Note that some functionality (e.g. writing UDFs or backfilling queries) may not be allowed. Run `gcloud auth login --update-adc` to authenticate against GCP.
- **For Data Engineering** - In addition to setting up the command line tools, you will want to log in to `shared-prod` if making changes to production systems. Run `gcloud auth login --update-adc --project=moz-fx-data-shared-prod` (if you have not run it previously).

### Installing bqetl

1. Clone the repository
```bash
git clone git@github.com:mozilla/bigquery-etl.git
cd bigquery-etl
```

2. Install the `bqetl` command line tool
```bash
./bqetl bootstrap
```

3. Install standard pre-commit hooks
```bash
venv/bin/pre-commit install
```

Finally, if you are using Visual Studio Code, you may also wish to use our recommended defaults:
```bash
cp .vscode/settings.json.default .vscode/settings.json
cp .vscode/launch.json.default .vscode/launch.json
```

And you should now be set up to start working in the repo! The easiest way to do this is for many tasks is to use [`bqetl`](https://mozilla.github.io/bigquery-etl/bqetl/). You may also want to read up on [common workflows](https://mozilla.github.io/bigquery-etl/cookbooks/common_workflows/).


## Releasing a new version of `bqetl`

To push a new version of `bqetl` to [PyPI](https://pypi.org/project/mozilla-bigquery-etl/), update the `version` in [`pyproject.toml`](pyproject.toml). The version numbers follow the [CalVer](https://calver.org/) scheme, with the _Micro_ version numbers starting at 1. For example, for the first package version getting published in March 2024, the version would be `2024.3.1`.
