PEP: 633 Title: Dependency specification in pyproject.toml using an
exploded TOML table Author: Laurie Opperman
\<<laurie_opperman@hotmail.com>\>, Arun Babu Neelicattu
\<<arun.neelicattu@gmail.com>\> Sponsor: Brett Cannon
\<<brett@python.org>\> Discussions-To:
<https://discuss.python.org/t/dependency-specification-in-pyproject-toml-using-an-exploded-toml-table/5123/>
Status: Draft Type: Standards Track Content-Type: text/x-rst Created:
2020-09-02 Post-History: 2020-09-02

Abstract
========

This PEP specifies how to write a project\'s dependencies in a
`pyproject.toml` file for packaging-related tools to consume using the
fields defined in `621`{.interpreted-text role="pep"}, as an alternative
to the `508`{.interpreted-text role="pep"}-based approach defined in
`631`{.interpreted-text role="pep"}.

Motivation
==========

There are multiple benefits to using TOML tables and other data-types to
represent requirements rather then `508`{.interpreted-text role="pep"}
strings:

-   Easy initial validation via the TOML syntax.
-   Easy secondary validation using a schema, for example a [JSON
    Schema](#json-schema).
-   Potential for users to guess the keys of given features, rather than
    memorising a syntax.
-   Users of multiple other popular languages may already be familiar
    with the TOML syntax.
-   TOML directly represents the same data structures as in JSON, and
    therefore a sub-set of Python literals, so users can understand the
    hierarchy and type of value

Rationale
=========

Most of this is taken from discussions in the [PEP 621 dependencies
topic](https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599).
This has elements from [Pipfile](https://github.com/pypa/pipfile),
[Poetry](https://python-poetry.org/docs/dependency-specification/),
[Dart\'s dependencies](https://dart.dev/tools/pub/dependencies) and
[Rust\'s
Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html).
A [comparison
document](https://github.com/uranusjr/packaging-metadata-comparisons/blob/master/topics/dependency-entries.md)
shows advantages and disadvantages between this format and
`508`{.interpreted-text role="pep"}-style specifiers.

In the specification of multiple requirements with the same distribution
name (where environment markers choose the appropriate dependency), the
chosen solution is similar to
[Poetry](https://python-poetry.org/docs/dependency-specification/)\'s,
where an array of requirements is allowed.

The direct-reference keys closely align with and utilise
pep:[610]{.title-ref} and `440`{.interpreted-text role="pep"} as to
reduce differences in the packaging ecosystem and rely on previous work
in specification.

Specification
=============

As in `621`{.interpreted-text role="pep"}, if metadata is improperly
specified then tools MUST raise an error. The metadata MUST conform to
the [TOML](https://toml.io/) specification.

To reduce confusion with this document being a specification for
specifying dependencies, the word \"requirement\" is used to mean a
`508`{.interpreted-text role="pep"} dependency specification.

The following tables are added to the added to the `project` table
specified in `621`{.interpreted-text role="pep"}.

`dependencies`
--------------

Format: table

The keys inside this table are the names of the required distribution.
The values can have one of the following types:

-   string: the requirement is defined only by a version requirement,
    with same specification as `version` in the requirement table,
    except allowing the empty string `""` to place no restriction on the
    version.
-   table: a requirement table.
-   array: an array of requirement tables. It is an error to specify an
    empty array `[]` as a value.

### Requirement table {#requirement-spec}

The keys of the requirement table are as follows (all are optional):

-   `version` (string): a `440`{.interpreted-text role="pep"} version
    specifier, which is a comma-delimited list of version specifier
    clauses. The string MUST be non-empty.
-   `extras` (array of strings): a list of `508`{.interpreted-text
    role="pep"} extras declarations for the distribution. The list MUST
    be non-empty.
-   `markers` (string): a `508`{.interpreted-text role="pep"}
    environment marker expression. The string MUST be non-empty.
-   `url` (string): the URL of the artifact to install and satisfy the
    requirement. Note that `file://` is the prefix used for packages to
    be retrieved from the local filesystem.
-   `git`, `hg`, `bzr` or `svn` (string): the URL of a VCS repository
    (as specified in `440`{.interpreted-text role="pep"}) to clone,
    whose tree will be installed to satisfy the requirement. Further VCS
    keys will be added via amendments to `610`{.interpreted-text
    role="pep"}, however tools MAY opt to support other VCS\'s using
    their command-line command prior to the acceptance of the amendment.
-   `revision` (string): the identifier for a specific revision of the
    specified VCS repository to check-out before installtion. Users MUST
    only provide this when one of `git`, `hg`, `bzr`, `svn`, or another
    VCS key is used to identify the distribution to install. Revision
    identifiers are suggested in `610`{.interpreted-text role="pep"}.

At most one of the following keys can be specified simultaneously, as
they logically conflict with each other in the requirement: `version`,
`url`, `git`, `hg`, `bzr`, `svn`, and any other VCS key.

An empty requirement table `{}` places no restriction on the
requirement, in addition to the empty string `""`.

Any keys provided which are not specified in this document MUST cause an
error in parsing.

`optional-dependencies`
-----------------------

Format: table

The keys inside this table are the names of an extra\'s required
distribution. The values can have one of the following types:

-   table: a requirement table.
-   array: an array of requirement tables.

These requirement tables have [the same specification as
above](#requirement-spec), with the addition of the following required
key:

-   `for-extra` (string): the name of the `508`{.interpreted-text
    role="pep"} extra that this requirement is required for.

Reference implementation
========================

Tools will need to convert this format to `508`{.interpreted-text
role="pep"} requirement strings. Below is an example implementation of
that conversion (assuming validation is already performed):

``` {.sourceCode .}
def convert_requirement_to_pep508(name, requirement):
    if isinstance(requirement, str):
        requirement = {"version": requirement}
    pep508 = name
    if "extras" in requirement:
        pep508 += " [" + ", ".join(requirement["extras"]) + "]"
    if "version" in requirement:
        pep508 += " " + requirement["version"]
    if "url" in requirement:
        pep508 += " @ " + requirement["url"]
    for vcs in ("git", "hg", "bzr", "svn"):
        if vcs in requirement:
            pep508 += " @ " + vcs + "+" + requirement[vcs]
            if "revision" in requirement:
                pep508 += "@" + requirement["revision"]
    extra = None
    if "for-extra" in requirement:
        extra = requirement["for-extra"]
    if "markers" in requirement:
        markers = requirement["markers"]
        if extra:
            markers = "extra = '" + extra + "' and (" + markers + ")"
        pep508 += "; " + markers
    return pep508, extra


def convert_requirements_to_pep508(dependencies):
    pep508s = []
    extras = set()
    for name, req in dependencies.items():
        if isinstance(req, list):
            for sub_req in req:
                pep508, extra = convert_requirement_to_pep508(name, sub_req)
                pep508s.append(pep508)
                if extra:
                    extras.add(extra)
        else:
            pep508, extra = convert_requirement_to_pep508(name, req)
            pep508s.append(pep508)
            if extra:
                extras.add(extra)
    return pep508s, extras


def convert_project_requirements_to_pep508(project):
    reqs, _ = convert_requirements_to_pep508(project.get("dependencies", {}))
    optional_reqs, extras = convert_requirements_to_pep508(
        project.get("optional-dependencies", {})
    )
    reqs += optional_reqs
    return reqs, extras
```

JSON schema
-----------

For initial validation, a JSON-schema can be used. Not only does this
help tools have a consistent validation, but it allows code editors to
highlight validation errors as users are building the dependencies list.

``` {.sourceCode .}
{
    "$id": "spam",
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Project metadata",
    "type": "object",
    "definitions": {
        "requirementTable": {
            "title": "Full project dependency specification",
            "type": "object",
            "properties": {
                "extras": {
                    "title": "Dependency extras",
                    "type": "array",
                    "items": {
                        "title": "Dependency extra",
                        "type": "string"
                    }
                },
                "markers": {
                    "title": "Dependency environment markers",
                    "type": "string"
                }
            },
            "propertyNames": {
                "enum": [
                    "extras",
                    "markers",
                    "version",
                    "url",
                    "git",
                    "hg",
                    "bzr",
                    "svn",
                    "for-extra"
                ]
            },
            "oneOf": [
                {
                    "title": "Version requirement",
                    "properties": {
                        "version": {
                            "title": "Version",
                            "type": "string"
                        }
                    }
                },
                {
                    "title": "URL requirement",
                    "properties": {
                        "url": {
                            "title": "URL",
                            "type": "string",
                            "format": "uri"
                        }
                    },
                    "required": [
                        "url"
                    ]
                },
                {
                    "title": "VCS requirement",
                    "properties": {
                        "revision": {
                            "title": "VCS repository revision",
                            "type": "string"
                        }
                    },
                    "oneOf": [
                        {
                            "title": "Git repository",
                            "properties": {
                                "git": {
                                    "title": "Git URL",
                                    "type": "string",
                                    "format": "uri"
                                }
                            },
                            "required": [
                                "git"
                            ]
                        },
                        {
                            "title": "Mercurial repository",
                            "properties": {
                                "hg": {
                                    "title": "Mercurial URL",
                                    "type": "string",
                                    "format": "uri"
                                }
                            },
                            "required": [
                                "hg"
                            ]
                        },
                        {
                            "title": "Bazaar repository",
                            "properties": {
                                "bzr": {
                                    "title": "Bazaar URL",
                                    "type": "string",
                                    "format": "uri"
                                }
                            },
                            "required": [
                                "bzr"
                            ]
                        },
                        {
                            "title": "Subversion repository",
                            "properties": {
                                "svn": {
                                    "title": "Subversion URL",
                                    "type": "string",
                                    "format": "uri"
                                }
                            },
                            "required": [
                                "svn"
                            ]
                        }
                    ]
                }
            ]
        },
        "requirementVersion": {
            "title": "Version project dependency specification",
            "type": "string"
        },
        "requirement": {
            "title": "Project dependency specification",
            "oneOf": [
                {
                    "$ref": "#/definitions/requirementVersion"
                },
                {
                    "$ref": "#/definitions/requirementTable"
                },
                {
                    "title": "Multiple specifications",
                    "type": "array",
                    "items": {
                        "$ref": "#/definitions/requirementTable"
                    },
                    "minLength": 1
                }
            ]
        },
        "optionalRequirementTable": {
            "title": "Project optional dependency specification table",
            "allOf": [
                {
                    "$ref": "#/definitions/requirementTable"
                },
                {
                    "properties": {
                        "for-extra": {
                            "title": "Dependency's extra",
                            "type": "string"
                        }
                    },
                    "required": [
                        "for-extra"
                    ]
                }
            ]
        },
        "optionalRequirement": {
            "title": "Project optional dependency specification",
            "oneOf": [
                {
                    "$ref": "#/definitions/optionalRequirementTable"
                },
                {
                    "title": "Multiple specifications",
                    "type": "array",
                    "items": {
                        "$ref": "#/definitions/optionalRequirementTable"
                    },
                    "minLength": 1
                }
            ]
        }
    },
    "properties": {
        "dependencies": {
            "title": "Project dependencies",
            "type": "object",
            "additionalProperties": {
                "$ref": "#/definitions/requirement"
            }
        },
        "optional-dependencies": {
            "title": "Project dependencies",
            "type": "object",
            "additionalProperties": {
                "$ref": "#/definitions/optionalRequirement"
            }
        }
    }
}
```

Examples
========

Full artificial example:

``` {.sourceCode .}
[project.dependencies]
flask = { }
django = { }
requests = { version = ">= 2.8.1, == 2.8.*", extras = ["security", "tests"], markers = "python_version < '2.7'" }
pip = { url = "https://github.com/pypa/pip/archive/1.3.1.zip" }
sphinx = { git = "ssh://git@github.com/sphinx-doc/sphinx.git" }
numpy = "~=1.18"
pytest = [
    { version = "<6", markers = "python_version < '3.5'" },
    { version = ">=6", markers = "python_version >= '3.5'" },
]

[project.optional-dependencies]
pytest-timout = { for-extra = "dev" }
pytest-mock = [
    { version = "<6", markers = "python_version < '3.5'", for-extra = "dev" },
    { version = ">=6", markers = "python_version >= '3.5'", for-extra = "dev" },
]
```

In homage to `631`{.interpreted-text role="pep"}, the following is an
equivalent dependencies specification for
[docker-compose](https://github.com/docker/compose/blob/789bfb0e8b2e61f15f423d371508b698c64b057f/setup.py#L28-L61):

``` {.sourceCode .}
[project.dependencies]
cached-property = ">= 1.2.0, < 2"
distro = ">= 1.2.0, < 2"
docker = { extras = ["ssh"], version = ">= 4.2.2, < 5" }
docopt = ">= 0.6.1, < 1"
jsonschema = ">= 2.5.1, < 4"
PyYAML = ">= 3.10, < 6"
python-dotenv = ">= 0.13.0, < 1"
requests = ">= 2.20.0, < 3"
texttable = ">= 0.9.0, < 2"
websocket-client = ">= 0.32.0, < 1"

# Conditional
"backports.shutil_get_terminal_size" = { version = "== 1.0.0", markers = "python_version < '3.3'" }
"backports.ssl_match_hostname" = { version = ">= 3.5, < 4", markers = "python_version < '3.5'" }
colorama = { version = ">= 0.4, < 1", markers = "sys_platform == 'win32'" }
enum34 = { version = ">= 1.0.4, < 2", markers = "python_version < '3.4'" }
ipaddress = { version = ">= 1.0.16, < 2", markers = "python_version < '3.3'" }
subprocess32 = { version = ">= 3.5.4, < 4", markers = "python_version < '3.2'" }

[project.optional-dependencies]
PySocks = { version = ">= 1.5.6, != 1.5.7, < 2", for-extra = "socks" }
ddt = { version = ">= 1.2.2, < 2", for-extra = "tests" }
pytest = { version = "< 6", for-extra = "tests" }
mock = { version = ">= 1.0.1, < 4", markers = "python_version < '3.4'", for-extra = "tests" }
```

Compatibility Examples
======================

The authors of this PEP recognise that various tools need to both read
from and write to this format for dependency specification. This section
aims to provide direct comparison with and examples for translating
to/from the currently used standard, `508`{.interpreted-text
role="pep"}.

::: {.note}
::: {.admonition-title}
Note
:::

For simplicity and clarity, various ways in which TOML allows you to
specify each specification is not represented. These examples use the
standard inline representation.

For example, while following are considered equivalent in TOML, we
choose the second form for the examples in this section.

``` {.sourceCode .}
aiohttp.version = "== 3.6.2"
aiohttp = { version = "== 3.6.2" }
```
:::

Version Constrained Dependencies
--------------------------------

**No Version Constraint**

``` {.sourceCode .}
aiohttp
```

``` {.sourceCode .}
aiohttp = {}
```

**Simple Version Constraint**

``` {.sourceCode .}
aiohttp >= 3.6.2, < 4.0.0
```

``` {.sourceCode .}
aiohttp = { version = ">= 3.6.2, < 4.0.0" }
```

::: {.note}
::: {.admonition-title}
Note
:::

This can, for conciseness, be also represented as a string.

``` {.sourceCode .}
aiohttp = ">= 3.6.2, < 4.0.0"
```
:::

Direct Reference Dependencies
-----------------------------

**URL Dependency**

``` {.sourceCode .}
aiohttp @ https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl
```

``` {.sourceCode .}
aiohttp = { url = "https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl" }
```

**VCS Dependency**

``` {.sourceCode .}
aiohttp @ git+ssh://git@github.com/aio-libs/aiohttp.git@master
```

``` {.sourceCode .}
aiohttp = { git = "ssh://git@github.com/aio-libs/aiohttp.git", revision = "master" }
```

Environment Markers
-------------------

``` {.sourceCode .}
aiohttp >= 3.6.1; python_version >= '3.8'
```

``` {.sourceCode .}
aiohttp = { version = ">= 3.6.1", markers = "python_version >= '3.8'" }
```

A slightly extended example of the above, where a particular version of
`aiohttp` is required based on the interpreter version.

``` {.sourceCode .}
aiohttp >= 3.6.1; python_version >= '3.8'
aiohttp >= 3.0.0, < 3.6.1; python_version < '3.8'
```

``` {.sourceCode .}
aiohttp = [
    { version = ">= 3.6.1", markers = "python_version >= '3.8'" },
    { version = ">= 3.0.0, < 3.6.1", markers = "python_version < '3.8'" }
]
```

Package Extras
--------------

**Specifying dependency for a package extra**

``` {.sourceCode .}
aiohttp >= 3.6.2; extra == 'http'
```

``` {.sourceCode .}
aiohttp = { version = ">= 3.6.2", for-extra = "http" }
```

**Using extras from a dependency**

``` {.sourceCode .}
aiohttp [speedups] >= 3.6.2
```

``` {.sourceCode .}
aiohttp = { version = ">= 3.6.2", extras = ["speedups"] }
```

Complex Examples
----------------

**Version Constraint**

``` {.sourceCode .}
aiohttp [speedups] >= 3.6.2; python_version >= '3.8' and extra == 'http'
```

``` {.sourceCode .}
aiohttp = { version = ">= 3.6.2", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }
```

**Direct Reference (VCS)**

``` {.sourceCode .}
aiohttp [speedups] @ git+ssh://git@github.com/aio-libs/aiohttp.git@master ; python_version >= '3.8' and extra == 'http'
```

``` {.sourceCode .}
aiohttp = { git = "ssh://git@github.com/aio-libs/aiohttp.git", revision = "master", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }
```

Rejected Ideas
==============

Switch to an array for `dependencies`
-------------------------------------

Use an array instead of a table in order to have each element only be a
table (with a `name` key) and no arrays of requirement tables. This was
very verbose and restrictive in the TOML format, and having multiple
requirements for a given distribution isn\'t very common.

Replace `optional-dependencies` with `extras`
---------------------------------------------

Remove the `optional-dependencies` table in favour of both including an
`optional` key in the requirement and an `extras` table which specifies
which (optional) requirements are needed for a project\'s extra. This
reduces the number of table with the same specification (to 1) and
allows for requirements to be specified once but used in multiple
extras, but distances some of the requirement\'s properties (which
extra(s) it belongs to), groups required and optional dependencies
together (possibly mixed), and there may not be a simple way to choose a
requirement when a distribution has multiple requirements. This was
rejected as `optional-dependencies` has already been used in the
`621`{.interpreted-text role="pep"} draft.

`direct` table in requirement
-----------------------------

Include the direct-reference keys in a `direct` table, have the VCS
specified as the value of a `vcs` key. This was more explicit and easier
to include in a JSON-schema validation, but was decided to be too
verbose and not as readable.

Include hash
------------

Include hash in direct-reference requirements. This was only for package
lock-files, and didn\'t really have a place in the project\'s metadata.

Dependency tables for each extra
--------------------------------

Have the `optional-dependencies` be a table of dependency tables for
each extra, with the table name being the extra\'s name. This made
`optional-dependencies` a different type (table of tables of
requirements) from `dependencies` (table of requirements), which could
be jarring for users and harder to parse.

Environment marker keys
-----------------------

Make each `508`{.interpreted-text role="pep"} environment marker as a
key (or child-table key) in the requirement. This arguably increases
readability and ease of parsing. The `markers` key would still be
allowed for more advanced specification, with which the key-specified
environment markers are `and`\'d with the result of. This was deferred
as more design needs to be undertaken.

Multiple extras which one requirement can satisfy
-------------------------------------------------

Replace the `for-extra` key with `for-extras`, with the value being an
array of extras which the requirement satisfies. This reduces some
duplication, but in this case that duplication makes explicit which
extras have which dependencies.

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
