Python packaging repo - Package auto assembler

Skip to content

# Python packaging repo

## About

A Python Packaging Repository (PPR) is a Git repository with a CI/CD pipeline designed to create and publish Python packages from code pushed to the repository. Using the `package-auto-assembler` tool, PPR can dynamically generate a packaging structure for `.py` files in a highly automated manner. This allows you to publish and maintain multiple packages from a single repository.

In its simplest form, adding a new `.py` file (or modifying an existing one) triggers the CI/CD pipeline to automatically prepare and publish release of new or existing package. Packages can be published to PyPI or private storage solutions such as Azure Artifacts Storage.

Diagram: Automated flow for packaging and publishing Python packages using PPR.

## Basic usage

### 1. Prepare Local Environment

Before developing code within a packaging repository, ensure that the `package-auto-assembler` python package is installed in your environment:

    pip install package-auto-assembler

Note: Some config files, like `.pipirc` and `.pip.conf`, might be required to configure access to private packages from Azure Artifacts storage.

### 2. Add or Edit a Package

To prepare your code for packaging:

1. Create/find a `.py` file in `module_dir` with a name of the package (use underscores (`_`) instead of hyphens (`-`) and spaces)

2. Make sure the module you're trying to create/edit follows basic requirements, described here (or use `concepts/prepackaging_todo.md` as local preflight checklist)

3. Add/edit optional files and additional documentation.

4. Define package metadata in module-level `__package_metadata__` as needed (see `components/package_metadata.md`).

Note: Relevant names of the directories, like `module_dir`, could be checked in `.paa.config` file from your instance of a packaging repository. This and the following steps assume that an instance of a packaging repository was already created or pulled. If not, to setup a new ppr take a look here.

### 3. Test-Install a Package

After adding or editing files related to your package, install it locally and ensure it works as expected.

    paa test-install your-package

Note: Use the `--skip-deps-install` flag if reinstalling dependencies is unnecessary. More flags could be seen here.

### 4. Push Changes to PPR

When code is ready for release, commit changes, including the package name and a list of changes in your commit messages. Push the changes to a new branch your packaging repository, then create a pull request to the `main` branch.

    git commit -m "[your_package] change one; comment about change two"

Note: Merge files for only one package at a time. The pipeline relies on commit history to determine which package to test and publish.

### 5. Publish a Package

If the test results are satisfactory, merge the pull request with `main`. The pipeline will then:

1. Initialize the packaging process.
2. Prepare the package.
3. Publish it to package storage.
4. Update tracking files in `.paa` and README.

Note Due to possibility of multiple packaging feeds, trigger for `azure devops + azure artifacts feed` template would not be automatic on merge, but would need to be triggered manually after merging with `main`. To do so, in your Azure DevOps project, navigate to `Pipelines` -> `your-packaging-repo-pipeline` -> `Run pipeline` and select one of the configured upload feeds in `Upload Feed` as well as indicate package to be published (ensure to use underscores (`_`) instead of hyphens (`-`)) in `Package Name` field.

### Additional Information

To see more CLI tools and options, run:

    paa --help

Or visit `package-auto-assembler` documentation. For documentation strategy across README, MkDocs, and release notes, see `concepts/documentation_flow.md`.

## Setting up new PPR

A Python Packaging Repository can be created for:

- GitHub with PyPI
- Azure DevOps with Azure Artifacts

### Prerequisites

- New Git Repository: A repository where the PPR will be set up.
- Pipeline Permissions: CI/CD pipelines must have read and write access to commit to the repository.
- Package Storage:
  - GitHub: A PyPI account.
  - Azure DevOps: An Azure Artifacts Feed.

Only two templates are provided: - `github + pypi` - `azure devops + azure artifacts feed`

### Github

1. Set Up GitHub Pages:

   - Navigate to `Settings` -> `Pages`.
   - Select `Deploy from a branch` choose the `gh-pages` branch (if it does not exist, create a new branch named `gh-pages`), and set the directory to `/root`. More details.

2. Configure GitHub Actions:

   - Navigate to `Settings` -> `Actions` -> `General`.
   - Under `Actions permissions` select `Allow all actions and reusable workflows`
   - Under `Workflow permissions` select `Read and write permissions` More details.

3. Add PyPI Credentials:

   - Go to `Settings` -> `Secrets and variables` -> `Actions`.
   - Add `TWINE_USERNAME` and `TWINE_PASSWORD` as secrets. More details.

4. Initialize the Template:

   - Use `paa` to set up the PPR:

   `paa init-ppr --github`

   Or include all optional directories:

   `paa init-ppr --github --full`

   - Edit `.paa.config` if needed
   - Run `paa init-ppr --github` or `paa init-paa` a second time to initialize directories for storing packaging files based on `.paa.config`.

5. Customize:

   - Edit `.github/docs/README_base.md` and `.github/tools/update_README.sh` to modify the repository-level README.

Once setup is complete, pushing changes to the `main` will automatically trigger the pipeline to package and publish your Python packages.

### Azure DevOps

1. Repository Permissions:

   - Navigate to `Project settings` -> `Repositories` -> `Your Repository`.
   - Set `Contribute` and `Create tag` permissions for `Your project build service` to "Allow"

2. Set Up Azure Artifacts Feed:

   - Create an artifacts feed or use an existing one. More details.

3. Add Credentials:

   - Generate a Personal Access Token (`TWINE_USERNAME` and `TWINE_PASSWORD`) with `Read & write`e permissions for `Packaging` More details.

4. Initialize the Template:

   - Use `paa` to set up the PPR:

   `paa init-ppr --azure`

   Or include all optional directories:

   `paa init-ppr --azure --full`

   - Edit `.paa.config` if needed
   - Run `paa init-ppr --azure` or `paa init-paa` a second time to initialize directories for storing packaging files based on `.paa.config`.
   - Create `.azure/feeds/YOUR_FEED_NAME.yml` files based on `.azure/feeds/example_feed.yml` and remove the example.

5. Configure the Pipeline:

   - Navigate to `Pipelines` -> `New pipeline`.
   - Choose `Azure Repos Git` -> `Your Repository`.
   - Select the `main` branch and `.azure/azure-pipelines.yml` to define the pipeline configuration for packaging and publishing.
   - Add `TWINE_USERNAME` and `TWINE_PASSWORD` under "Variables"

6. Customize:

   - Edit `.azure/docs/README_base.md` and `.azure/tools/update_README.sh` to modify the repository-level README.

Note: Pushing changes to the `main` branch does not necessarily mean that a package will be published. Since multiple feeds can be published from this repository, a manual trigger is preferred.

To trigger the workflow:

1. Navigate to `Pipelines` -> `your-packaging-repo-pipeline` -> `Run pipeline`.
2. Select one of the configured upload feeds in the `Upload Feed` dropdown.
3. Specify the package name in the `Package Name` field (use underscores (`_`) instead of hyphens (`-`)).