pyproject2conda — pyproject2conda v0.8.0 documentation

Skip to main content
`Ctrl`+`K`

# pyproject2conda

# `pyproject2conda`

A script to convert `pyproject.toml` dependencies to `environemnt.yaml` files.

## Overview

The main goal of `pyproject2conda` is to provide a means to keep all basic dependency information, for both `pip` based and `conda` based environments, in `pyproject.toml`. I often use a mix of pip and conda when developing packages, and in my everyday workflow. Some packages just aren’t available on both. If you use poetry, I’d highly recommend poetry2conda.

## Features

- Simple comment based syntax to add information to dependencies when creating `environment.yaml`

## Status

This package is actively used by the author, but is still very much a work in progress. Please feel free to create a pull request for wanted features and suggestions!

## Quick start

Use one of the following to install `pyproject2conda`:

    $ pip install pyproject2conda

or

    $ conda install -c conda-forge pyproject2conda

If using pip, to install with rich and shellingham support, either install them your self, or use:

    $ pip install pyproject2conda[all]

The conda-forge distribution of typer (which `pyproject2conda` uses) installs rich and shellingham by default.

## Example usage

### Basic usage

Consider the `toml` file test-pyproject.toml.

    [project]
    name = "hello"
    requires-python = ">=3.8,<3.11"
    dependencies = [
    "athing", # p2c: -p # a comment
    "bthing", # p2c: -s "bthing-conda"
    "cthing; python_version < '3.10'", # p2c: -c conda-forge

    ]

    # ...

Note the comment lines `# p2c:...`. These are special tokens that `pyproject2conda` will analyze. The basic options are:

    usage: -c [-h] [-c CHANNEL] [-p] [-s] [package ...]

    Parser searches for comments '# p2c: [OPTIONS] CONDA-PACKAGES

    positional arguments:
      package

    options:
      -h, --help            show this help message and exit
      -c CHANNEL, --channel CHANNEL
                            Channel to add to the pyproject requirement
      -p, --pip             If specified, install pyproject dependency with pip
      -s, --skip            If specified skip pyproject dependency on this line

So, if we run the following, we get:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml
    channels:
      - conda-forge
    dependencies:
      - bthing-conda
      - conda-forge::cthing
      - pip
      - pip:
          - athing

Note that other comments can be mixed in.

By default, the python version is not included in the resulting conda output. To include the specification from `pyproject.toml`, use `--python-include infer` option:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-include infer
    channels:
      - conda-forge
    dependencies:
      - python>=3.8,<3.11
      - bthing-conda
      - conda-forge::cthing
      - pip
      - pip:
          - athing

### Specify python version

To specify a specific value of python in the output, pass a value with:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-include \
        python=3.9
    channels:
      - conda-forge
    dependencies:
      - python=3.9
      - bthing-conda
      - conda-forge::cthing
      - pip
      - pip:
          - athing

Note that this is for including python in the resulting environment file.

You can also constrain packages by the python version using the standard `pyproject.toml` syntax `"...; python_version < 'some-version-number'"`. For is parsed for both the pip packages and conda packages:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-version 3.10
    channels:
      - conda-forge
    dependencies:
      - bthing-conda
      - pip
      - pip:
          - athing

It is common to want to specify the python version and include it in the resulting environment file. You could, for example use:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml --python-version 3.10 \
        --python-include python=3.10
    channels:
      - conda-forge
    dependencies:
      - python=3.10
      - bthing-conda
      - pip
      - pip:
          - athing

Because this is common, you can also just pass the option `-p/--python`:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml --python 3.10
    channels:
      - conda-forge
    dependencies:
      - python=3.10
      - bthing-conda
      - pip
      - pip:
          - athing

### Adding extra conda dependencies and pip requirements

You can also add additional conda and pip dependencies with the flags `-d/--deps` and `-r/--reqs`, respectively. Adding the last example:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml -d dep -r req
    channels:
      - conda-forge
    dependencies:
      - bthing-conda
      - conda-forge::cthing
      - dep
      - pip
      - pip:
          - athing
          - req

These will also obey dependencies like `dep:python_version<={version}`. Pass the flags multiple times to pass multiple dependencies.

### Command “aliases”

The name `pyproject2conda` can be a bit long to type. For this reason, the package also ships with the alias `p2c`, which has the exact same functionality. Additionally, the subcommands can be shortened to a unique match:

    $ p2c y -f tests/data/test-pyproject.toml --python 3.10
    channels:
      - conda-forge
    dependencies:
      - python=3.10
      - bthing-conda
      - pip
      - pip:
          - athing

You can also call with `python -m pyproject2conda`.

### Installing extras

Given the extra dependency:

    # ...
    [project.optional-dependencies]
    test = [
    "pandas",
    "pytest", # p2c: -c conda-forge

    ]
    dev-extras = [
    # p2c: -s "additional-thing; python_version < '3.9'" # additional pkg
    ## p2c: -s "another-thing" # skipped because of ## before p2c.
    "matplotlib", # p2c: -s conda-matplotlib

    ]
    dev = ["hello[test]", "hello[dev-extras]"]
    dist-pypi = [
    # this is intended to be parsed with --no-base option
    "setuptools",
    "build", # p2c: -p

    ]

    # ...

and running the following gives:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml -e test
    channels:
      - conda-forge
    dependencies:
      - bthing-conda
      - conda-forge::cthing
      - conda-forge::pytest
      - pandas
      - pip
      - pip:
          - athing

`pyproject2conda` also works with self referenced dependencies:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml -e dev
    channels:
      - conda-forge
    dependencies:
      - additional-thing
      - bthing-conda
      - conda-forge::cthing
      - conda-forge::pytest
      - conda-matplotlib
      - pandas
      - pip
      - pip:
          - athing

This also shows that `p2c` comments without dependencies are also parsed. To comment out such lines, make sure `p2c` is preceded by `##`.

### Header in output

By default, `pyproject2conda` includes a header in most output files to note that the files are auto generated. No header is included by default when writing to standard output. To override this behavior, pass `--header/--noheader`:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml --header
    #
    # This file is autogenerated by pyproject2conda
    # with the following command:
    #
    #     $ pyproject2conda yaml -f tests/data/test-pyproject.toml --header
    #
    # You should not manually edit this file.
    # Instead edit the corresponding pyproject.toml file.
    #
    channels:
      - conda-forge
    dependencies:
      - bthing-conda
      - conda-forge::cthing
      - pip
      - pip:
          - athing

### Usage within python

`pyproject2conda` can also be used within python:

    >>> from pyproject2conda import PyProject2Conda
    >>> p = PyProject2Conda.from_path("./tests/data/test-pyproject.toml")

    # Basic environment
    >>> print(p.to_conda_yaml(python_include="infer").strip())
    channels:
      - conda-forge
    dependencies:
      - python>=3.8,<3.11
      - bthing-conda
      - conda-forge::cthing
      - pip
      - pip:
          - athing

    # Environment with extras
    >>> print(p.to_conda_yaml(extras="test").strip())
    channels:
      - conda-forge
    dependencies:
      - bthing-conda
      - conda-forge::cthing
      - conda-forge::pytest
      - pandas
      - pip
      - pip:
          - athing

### Configuration

`pyproject2conda` can be configured with a `[tool.pyproject2conda]` section in `pyproject.toml`. To specify conda channels use:

    # ...
    [tool.pyproject2conda]
    channels = ['conda-forge']
    # these are the same as the default values of `p2c project`
    template_python = "py{py}-{env}"
    template = "{env}"
    style = "yaml"
    # options
    python = ["3.10"]
    # Note that this is relative to the location of pyproject.toml
    user_config = "config/userconfig.toml"
    default_envs = ["test", "dev", "dist-pypi"]

    [tool.pyproject2conda.envs."test-extras"]
    extras = ["test"]
    style = ["yaml", "requirements"]

    [[tool.pyproject2conda.overrides]]
    envs = ['test-extras', "dist-pypi"]
    base = false

    [[tool.pyproject2conda.overrides]]
    envs = ["test", "test-extras"]
    python = ["3.10", "3.11"]

Note that specifying channels at the command line overrides `tool.pyproject2conda.channels`.

You can also specify environments without the base dependencies (those under `project.dependencies`) by passing the `--no-base` flag. This is useful for defining environments for build, etc, that do not require the package be installed. For example:

    # ...
    dist-pypi = [
    # this is intended to be parsed with --no-base option
    "setuptools",
    "build", # p2c: -p

    ]

    # ...

These can be accessed using either of the following:

    $ pyproject2conda yaml -f tests/data/test-pyproject.toml -e dist-pypi --no-base
    channels:
      - conda-forge
    dependencies:
      - setuptools
      - pip
      - pip:
          - build

or

    >>> from pyproject2conda import PyProject2Conda
    >>> p = PyProject2Conda.from_path("./tests/data/test-pyproject.toml")

    # Basic environment
    >>> print(p.to_conda_yaml(extras='dist-pypi', include_base_dependencies=False).strip())
    channels:
      - conda-forge
    dependencies:
      - setuptools
      - pip
      - pip:
          - build

### Creating multiple environments from `pyproject.toml`

`pyproject2conda` provides a means to create all needed environment/requirement files in one go. We configure the environments using the `pyproject.toml` files in the `[tool.pyproject2conda]` section. For example, example the configuration:

    # ...
    [tool.pyproject2conda]
    channels = ['conda-forge']
    # these are the same as the default values of `p2c project`
    template_python = "py{py}-{env}"
    template = "{env}"
    style = "yaml"
    # options
    python = ["3.10"]
    # Note that this is relative to the location of pyproject.toml
    user_config = "config/userconfig.toml"
    default_envs = ["test", "dev", "dist-pypi"]

    [tool.pyproject2conda.envs."test-extras"]
    extras = ["test"]
    style = ["yaml", "requirements"]

    [[tool.pyproject2conda.overrides]]
    envs = ['test-extras', "dist-pypi"]
    base = false

    [[tool.pyproject2conda.overrides]]
    envs = ["test", "test-extras"]
    python = ["3.10", "3.11"]

run through the command `pyproject2conda project` (or `p2c project`):

    $ p2c project -f tests/data/test-pyproject.toml --dry
    # --------------------
    # Creating yaml py310-test-extras.yaml
    channels:
      - conda-forge
    dependencies:
      - python=3.10
      - conda-forge::pytest
      - pandas
    # --------------------
    # Creating yaml py311-test-extras.yaml
    channels:
      - conda-forge
    dependencies:
      - python=3.11
      - conda-forge::pytest
      - pandas
    # --------------------
    # Creating requirements test-extras.txt
    pandas
    pytest
    # --------------------
    # Creating yaml py310-test.yaml
    channels:
      - conda-forge
    dependencies:
      - python=3.10
      - bthing-conda
      - conda-forge::pytest
      - pandas
      - pip
      - pip:
          - athing
    # --------------------
    # Creating yaml py311-test.yaml
    channels:
      - conda-forge
    dependencies:
      - python=3.11
      - bthing-conda
      - conda-forge::pytest
      - pandas
      - pip
      - pip:
          - athing
    # --------------------

     ...

Note that here, we have used the `--dry` option to just print the output. In production, you’d omit this flag, and files according to `--template` and `--template-python` would be used.

The options under `[tool.pyproject2conda]` follow the command line options (replace `-` with `_`). To specify an environment, you can either use the `[tool.pyproject.envs."environment-name"]` method, or, if the environment is the same as the “extras” name, you can just specify it under `tool.pyproject2conda.default_envs`:

    [tool.pyproject2conda]
    # ...
    default_envs = ["test"]

is equivalent to

    [tool.pyproject2conda.envs.test]
    extras = ["tests"]

To specify a conda environment (`yaml`) file, pass `style = "yaml"` (the default). To specify a requirements file, pass `style = "requirements"`. You can specify both to make both.

Options in a given `tool.pyproject2conda.envs."environemnt-name"` section override those at the `tool.pyproject2conda` level. So, for example:

    # ...
    [tool.pyproject2conda.envs."test-extras"]
    extras = ["test"]
    style = ["yaml", "requirements"]

    # ...

will override use the two styles instead of the default of `yaml`.

You can also override options for multiple environments using the `[[tools.pyproject2conda.overrides]]` list. Just specify the override option(s) and the environments to apply them to. For example, above we specify that the base option is `False` for envs `test-extras` and `dist-pypi`, and that the python version should be `3.10` and `3.11` for envs `test` and `test-extras`.

So in all, options are picked up, in order, from the environment definition, then the overrides list, and finally, from the default options.

You can also define “user defined” configurations. This can be done through the option `--user-config`. This allows you to define your own environments outside of the (most likely source controlled) `pyproject.toml` file. For example, we have the option `user_config=config/userconfig.toml`.

    [tool.pyproject2conda.envs."user-dev"]
    extras = ["dev", "dist-pypi"]
    deps = ["extra-dep"]
    reqs = ["extra-req"]
    name = "hello"

Note that the full path of this file is note that the path of the `user_conifg` file is relative to them`pyproject.toml` file. So, if the `pyproject.toml` file is at `a/path/pyproject.toml`, the path of user configuration files will be `a/path/config/userconfig.toml`. We then can run the following:

    $ p2c project -f tests/data/test-pyproject.toml --dry --envs user-dev
    # --------------------
    # Creating yaml py310-user-dev.yaml
    name: hello
    channels:
      - conda-forge
    dependencies:
      - python=3.10
      - bthing-conda
      - conda-forge::pytest
      - conda-matplotlib
      - extra-dep
      - pandas
      - setuptools
      - pip
      - pip:
          - athing
          - build
          - extra-req

### CLI options

See command line interface documentation for details on the commands and options.

## Related work

The application `pyproject2conda` is used in the development of the following packages:

- `cmomy`

- `thermoextrap`

- `tmmc-lnpy`

- `module-utilities`

- `analphipy`

- `pyproject2conda` itself!

- Installation
- API Reference
- License
- Contributing
- Credits
- Changelog
- Indices and tables