PEP: 516 Title: Build system abstraction for pip/conda etc Version:
$Revision$ Last-Modified: $Date$ Author: Robert Collins
<rbtcollins@hp.com>, Nathaniel Smith <njs@pobox.com> BDFL-Delegate: Nick
Coghlan <ncoghlan@gmail.com> Discussions-To: distutils-sig
<distutils-sig@python.org> Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 26-Oct-2015 Resolution:
https://mail.python.org/pipermail/distutils-sig/2017-May/030517.html

Abstract
========

This PEP specifies a programmatic interface for pip \[\#pip\]\_ and
other distribution or installation tools to use when working with Python
source trees (both the developer tree - e.g. the git tree - and source
distributions).

The programmatic interface allows decoupling of pip from its current
hard dependency on setuptools \[\#setuptools\]\_ able for two key
reasons:

1.  It enables new build systems that may be much easier to use without
    requiring them to even appear to be setuptools.

2.  It facilitates setuptools itself changing its user interface without
    breaking pip, giving looser coupling.

The interface needed to permit pip to install build systems also enables
pip to install build time requirements for packages which is an
important step in getting pip to full feature parity with the
installation components of easy-install.

As PEP-426 \[\#pep426\]\_ is draft, we cannot utilise the metadata
format it defined. However PEP-427 wheels are in wide use and fairly
well specified, so we have adopted the METADATA format from that for
specifying distribution dependencies and general project metadata.
PEP-0508 \[\#pep508\] provides a self-contained language for describing
a dependency, which we encapsulate in a thin JSON schema to describe
bootstrap dependencies.

Since Python sdists specified in PEP-0314 \[\#pep314\] are also source
trees, this PEP is updating the definition of sdists.

PEP Rejection
=============

The CLI based approach proposed in this PEP has been rejected in favour
of the Python API based approach proposed in PEP 517. The specific CLI
used to communicate with build backends running as isolated subprocesses
will be considered an implementation detail of front-end developer tool
implementations.

Motivation
==========

There is significant pent-up frustration in the Python packaging
ecosystem around the current lock-in between build system and pip.
Breaking that lock-in is better for pip, for setuptools, and for other
build systems like flit \[\#flit\]\_.

Specification
=============

Overview
--------

Build tools will be located by reading a file `pypa.json` from the root
directory of the source tree. That file describes how to get the build
tool and the name of the command to run to invoke the tool.

All tools will be expected to conform to a single command line interface
modelled on pip's existing use of the setuptools setup.py interface.

pypa.json
---------

The file `pypa.json` acts as a neutral configuration file for pip and
other tools that want to build source trees to consult for
configuration. The absence of a `pypa.json` file in a Python source tree
implies a setuptools or setuptools compatible build system.

The JSON has the following schema. Extra keys are ignored, which permits
the use of `pypa.json` as a configuration file for other related tools.
If doing that the chosen keys must be namespaced under `tools`::

{"tools": {"flit": \["Flits content here"\]}}

schema The version of the schema. This PEP defines version "1". Defaults
to "1" when absent. All tools reading the file must error on an
unrecognised schema version.

bootstrap\_requires Optional list of dependency specifications
\[\#pep508\] that must be installed before running the build tool. For
instance, if using flit, then the requirements might be::

        bootstrap_requires: ["flit"]

build\_command A mandatory key, this is a list of Python format strings
\[\#strformat\]\_ describing the command to run. For instance, if using
flit then the build command might be::

        build_command: ["flit"]

    If using a command which is a runnable module fred::

        build_command: ["{PYTHON}", "-m", "fred"]

Process interface
-----------------

The command to run is defined by a simple Python format string
\[\#strformat\]\_.

This permits build systems with dedicated scripts and those that are
invoked using "python -m somemodule".

Processes will be run with the current working directory set to the root
of the source tree.

When run, processes should not read from stdin - while pip currently
runs build systems with stdin connected to its own stdin, stdout and
stderr are redirected and no communication with the user is possible.

As usual with processes, a non-zero exit status indicates an error.

Available format variables
--------------------------

PYTHON The Python interpreter in use. This is important to enable
calling things which are just Python entry points.

        {PYTHON} -m foo

Available environment variables
-------------------------------

These variables are set by the caller of the build system and will
always be available.

PATH The standard system path.

PYTHON As for format variables.

PYTHONPATH Used to control sys.path per the normal Python mechanisms.

Subcommands
-----------

There are a number of separate subcommands that build systems must
support. The examples below use a build\_command of `flit` for
illustrative purposes.

build\_requires Query build requirements. Build requirements are
returned as a UTF-8 encoded JSON document with one key `build_requires`
consisting of a list of dependency specifications \[\#pep508\]\_.
Additional keys must be ignored. The build\_requires command is the only
command run without setting up a build environment.

    Example command::

        flit build_requires

metadata Query project metadata. The metadata and only the metadata
should be output on stdout in UTF-8 encoding. pip would run metadata
just once to determine what other packages need to be downloaded and
installed. The metadata is output as a wheel METADATA file per PEP-427
\[\#pep427\]\_.

    Note that the metadata generated by the metadata command, and the metadata
    present in a generated wheel must be identical.

    Example command::

        flit metadata

wheel -d OUTPUT\_DIR Command to run to build a wheel of the project.
OUTPUT\_DIR will point to an existing directory where the wheel should
be output. Stdout and stderr have no semantic meaning. Only one file
should be output - if more are output then pip would pick an arbitrary
one to consume.

    Example command::

        flit wheel -d /tmp/pip-build_1234

develop \[--prefix PREFIX\] Command to do an in-place 'development'
installation of the project. Stdout and stderr have no semantic meaning.

    Not all build systems will be able to perform develop installs. If a build
    system cannot do develop installs, then it should error when run. Note
    that doing so will cause use operations like ``pip install -e foo`` to
    fail.

    The prefix option is used for defining an alternative prefix for the
    installation. While setuptools has ``--root`` and ``--user`` options,
    they can be done equivalently using ``--prefix``, and pip or other
    tools that accept ``--root`` or ``--user`` options should translate
    appropriately.

    The root option is used to define an alternative root within which the
    command should operate.

    For instance::

        flit develop --root /tmp/ --prefix /usr/local

    Should install scripts within `/tmp/usr/local/bin`, even if the Python
    environment in use reports that the sys.prefix is `/usr/` which would lead
    to using `/tmp/usr/bin/`. Similar logic applies for package files etc.

The build environment
---------------------

Except for the build\_requires command, all commands are run within a
build environment. No specific implementation is required, but a build
environment must achieve the following requirements.

1.  All dependencies specified by the project's build\_requires must be
    available for import from within `$PYTHON`.

2.  All command-line scripts provided by the build-required packages
    must be present in `$PATH`.

A corollary of this is that build systems cannot assume access to any
Python package that is not declared as a build\_requires or in the
Python standard library.

Hermetic builds
---------------

This specification does not prescribe whether builds should be hermetic
or not. Existing build tools like setuptools will use installed versions
of build time requirements (e.g. setuptools\_scm) and only install other
versions on version conflicts or missing dependencies. However its
likely that better consistency can be created by always isolation builds
and using only the specified dependencies.

However, there are nuanced problems there - such as how can users force
the avoidance of a bad version of a build requirement which meets some
packages dependencies. Future PEPs may tackle this problem, but it is
not currently in scope - it does not affect the metadata required to
coordinate between build systems and things that need to do builds, and
thus is not PEP material.

Upgrades
--------

'pypa.json' is versioned to permit future changes without requiring
compatibility.

The sequence for upgrading either of schemas in a new PEP will be:

1.  Issue new PEP defining an updated schema. If the schema is not
    entirely backward compatible then a new version number must be
    defined.
2.  Consumers (e.g. pip) implement support for the new schema version.
3.  Package authors opt into the new schema when they are happy to
    introduce a dependency on the version of 'pip' (and potentially
    other consumers) that introduced support for the new schema version.

The *same* process will take place for the initial deployment of this
PEP:- the propagation of the capability to use this PEP without a
`setuptools shim`\_ will be largely gated by the adoption rate of the
first version of pip that supports it.

Static metadata in sdists
-------------------------

This PEP does not tackle the current inability to trust static metadata
in sdists. That is a separate problem to identifying and consuming the
build system that is in use in a source tree, whether it came from an
sdist or not.

Handling of compiler options
----------------------------

Handling of different compiler options is out of scope for this
specification.

pip currently handles compiler options by appending user supplied
strings to the command line it runs when running setuptools. This
approach is sufficient to work with the build system interface defined
in this PEP, with the exception that globally specified options will
stop working globally as different build systems evolve. That problem
can be solved in pip (or conda or other installers) without affecting
interoperability.

In the long term, wheels should be able to express the difference
between wheels built with one compiler or options vs another, and that
is PEP material.

Examples
========

An example 'pypa.json' for using flit::

{"bootstrap\_requires": \["flit"\], "build\_command": "flit"}

When 'pip' reads this it would prepare an environment with flit in it
before trying to use flit.

Because flit doesn't have setup-requires support today,
`flit build_requires` would just output a constant string::

{"build\_requires": \[\]}

`flit metadata` would interrogate `flit.ini` and marshal the metadata
into a wheel METADATA file and output that on stdout.

`flit wheel` would need to accept a `-d` parameter that tells it where
to output the wheel (pip needs this).

Backwards Compatibility
=======================

Older pips will remain unable to handle alternative build systems. This
is no worse than the status quo - and individual build system projects
can decide whether to include a shim `setup.py` or not.

All existing build systems that can product wheels and do develop
installs should be able to run under this abstraction and will only need
a specific adapter for them constructed and published on PyPI.

In the absence of a `pypa.json` file, tools like pip should assume a
setuptools build system and use setuptools commands directly.

Network effects
---------------

Projects that adopt build systems that are not setuptools compatible -
that is that they have no setup.py, or the setup.py doesn't accept
commands that existing tools try to use - will not be installable by
those existing tools.

Where those projects are used by other projects, this effect will
cascade.

In particular, because pip does not handle setup-requires today, any
project (A) that adopts a setuptools-incompatible build system and is
consumed as a setup-requirement by a second project (B) which has not
itself transitioned to having a pypa.json will make B uninstallable by
any version of pip. This is because setup.py in B will trigger
easy-install when 'setup.py egg\_info' is run by pip, and that will try
and fail to install A.

As such we recommend that tools which are currently used as
setup-requires either ensure that they keep a `setuptools shim`\_ or
find their consumers and get them all to upgrade to the use of a
`pypa.json` in advance of moving themselves. Pragmatically that is
impossible, so the advice is to keep a setuptools shim indefinitely -
both for projects like pbr, setuptools\_scm and also projects like
numpy.

setuptools shim
---------------

It would be possible to write a generic setuptools shim that looks like
`setup.py` and under the hood uses `pypa.json` to drive the builds. This
is not needed for pip to use the system, but would allow package authors
to use the new features while still retaining compatibility with older
pip versions.

Rationale
=========

This PEP started with a long mailing list thread on distutils-sig
\[\#thread\]*. Subsequent to that an online meeting was held to debug
all the positions folk had. Minutes from that were posted to the list
\[\#minutes\]*.

This specification is a translation of the consensus reached there into
PEP form, along with some arbitrary choices on the minor remaining
questions.

The basic heuristic for the design has been to focus on introducing an
abstraction without requiring development not strictly tied to the
abstraction. Where the gap is small to improvements, or the cost of
using the existing interface is very high, then we've taken on having
the improvement as a dependency, but otherwise deferred such to future
iterations.

We chose wheel METADATA files rather than defining a new specification,
because pip can already handle wheel .dist-info directories which encode
all the necessary data in a METADATA file. PEP-426 can't be used as it's
still draft, and defining a new metadata format, while we should do
that, is a separate problem. Using a directory on disk would not add any
value to the interface (pip has to do that today due to limitations in
the setuptools CLI).

The use of 'develop' as a command is because there is no PEP specifying
the interoperability of things that do what 'setuptools develop' does -
so we'll need to define that before pip can take on the responsibility
for doing the 'develop' step. Once that's done we can issue a successor
PEP to this one.

The use of a command line API rather than a Python API is a little
contentious. Fundamentally anything can be made to work, and the pip
maintainers have spoken strongly in favour of retaining a process based
interface - something that is mature and robust in pip today.

The choice of JSON as a file format is a compromise between several
constraints. Firstly there is no stdlib YAML interpreter, nor one for
any of the other low-friction structured file formats. Secondly,
INIParser is a poor format for a number of reasons, primarily that it
has very minimal structure - but pip's maintainers are not fond of it.
JSON is in the stdlib, has sufficient structure to permit embedding
anything we want in future without requiring embedded DSL's.

Donald suggested using `setup.cfg` and the existing setuptools command
line rather than inventing something new. While that would permit
interoperability with less visible changes, it requires nearly as much
engineering on the pip side - looking for the new key in setup.cfg,
implementing the non-installed environments to run the build in. And the
desire from other build system authors not to confuse their users by
delivering something that looks like but behaves quite differently to
setuptools seems like a bigger issue than pip learning how to invoke a
custom build tool.

The metadata and wheel commands are required to have consistent metadata
to avoid a race condition that could otherwise happen where pip reads
the metadata, acts on it, and then the resulting wheel has incompatible
requirements. That race is exploited today by packages using PEP-426
environment markers, to work with older pip versions that do not support
environment markers. That exploit is not needed with this PEP, because
either the setuptools shim is in use (with older pip versions), or an
environment marker ready pip is in use. The setuptools shim can take
care of exploiting the difference older pip versions require.

We discussed having an sdist verb. The main driver for this was to make
sure that build systems were able to produce sdists that pip can build -
but this is circular: the whole point of this PEP is to let pip consume
such sdists or VCS source trees reliably and without requiring an
implementation of setuptools. Being able to create new sdists from
existing source trees isn't a thing pip does today, and while there is a
PR to do that as part of building from source, it is contentious and
lacks consensus. Rather than impose a requirement on all build systems,
we are treating it as a YAGNI, and will add such a verb in a future
version of the interface if required. The existing PEP-314 \[\#pep314\]
requirements for sdists still apply, and distutils or setuptools users
can use `setup.py sdist` to create an sdist. Other tools should create
sdists compatible with PEP-314 \[\#pep314\]. Note that pip itself does
not require PEP-314 compatibility - it does not use any of the metadata
from sdists - they are treated like source trees from disk or version
control.

References
==========

.. \[\#pip\] pip, the recommended installer for Python packages
(http://pip.readthedocs.org/en/stable/)

.. \[\#setuptools\] setuptools, the de facto Python package build system
(https://pythonhosted.org/setuptools/)

.. \[\#flit\] flit, a simple way to put packages in PyPI
(http://flit.readthedocs.org/en/latest/)

.. \[\#pypi\] PyPI, the Python Package Index (https://pypi.python.org/)

.. \[\#shellvars\] Shellvars, an implementation of shell variable rules
for Python. (https://github.com/testing-cabal/shellvars)

.. \[\#pep426\] PEP-426, Python distribution metadata.
(https://www.python.org/dev/peps/pep-0426/)

.. \[\#pep427\] PEP-427, Python distribution metadata.
(https://www.python.org/dev/peps/pep-0427/)

.. \[\#thread\] The kick-off thread.
(https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html)

.. \[\#minutes\] The minutes.
(https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html)

.. \[\#strformat\] The Python string formatting syntax.
(https://docs.python.org/3.1/library/string.html\#format-string-syntax)

.. \[\#pep314\] Metadata for Python Software Packages v1.1
(https://www.python.org/dev/peps/pep-0314/)

.. \[\#pep508\] Dependency specification language PEP.
(https://www.python.org/dev/peps/pep-0508/)

Copyright
=========

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
