PEP: 453 Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft
<donald@stufft.io>, Nick Coghlan <ncoghlan@gmail.com> BDFL-Delegate:
Martin von Löwis Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 10-Aug-2013 Post-History: 30-Aug-2013, 15-Sep-2013,
18-Sep-2013, 19-Sep-2013, 23-Sep-2013, 29-Sep-2013, 13-Oct-2013,
20-Oct-2013 Resolution:
https://mail.python.org/pipermail/python-dev/2013-October/129810.html

Abstract
========

This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`\_\_ guide
in Python 2.7, 3.3 and 3.4 be updated to officially recommend the use of
`pip` as the default installer for Python packages, and that appropriate
technical changes be made in Python 3.4 to provide `pip` by default in
support of that recommendation.

PEP Acceptance
==============

This PEP was accepted for inclusion in Python 3.4 by Martin von Löwis on
Tuesday 22nd October, 2013.

`Issue 19347 <http://bugs.python.org/issue19347>`\_\_ has been created
to track the implementation of this PEP.

Rationale
=========

There are two related, but distinct rationales for the proposal in this
PEP. The first relates to the experience of new users, while the second
relates to better enabling the evolution of the broader Python packaging
ecosystem.

Improving the new user experience
---------------------------------

Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.

Even on systems that *do* have a platform package manager, it is
unlikely to include every package that is available on the Python
Package Index, and even when a desired third-party package is available,
the correct name in the platform package manager may not be clear.

This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to
get it, and how to install it. The effect of this is that third-party
Python projects are currently required to choose from a variety of
undesirable alternatives:

-   Assume the user already has a suitable cross-platform package
    manager installed.
-   Duplicate the instructions and tell their users how to install the
    package manager.
-   Completely forgo the use of dependencies to ease installation
    concerns for their users.

All of these available options have significant drawbacks.

If a project simply assumes a user already has the tooling then
beginning users may get a confusing error message when the installation
command doesn't work. Some operating systems may ease this pain by
providing a global hook that looks for commands that don't exist and
suggest an OS package they can install to make the command work, but
that only works on systems with platform package managers that include a
package that provides the relevant cross-platform installer command
(such as many major Linux distributions). No such assistance is
available for Windows and Mac OS X users, or more conservative Linux
distributions. The challenges of dealing with this problem for beginners
(who are often also completely new to programming, the use of command
line tools and editing system environment variables) are a regular
feature of feedback the core Python developers receive from professional
educators and others introducing new users to Python.

If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how
to install their own project then whenever these instructions need
updates they need updating by every project that has duplicated them.
This is particular problematic when there are multiple competing
installation tools available, and different projects recommend different
tools.

This specific problem can be partially alleviated by strongly promoting
`pip` as the default installer and recommending that other projects
reference
`pip's own bootstrapping instructions <http://www.pip-installer.org/en/latest/installing.html>`\_\_
rather than duplicating them. However the user experience created by
this approach still isn't particularly good (although there is an effort
under way to create a combined Windows installer for `pip` and its
dependencies that should improve matters on that platform, and Mac OS X
and \*nix platforms generally have `wget` and hence the ability to
easily download and run the bootstrap scripts from the command line).

The projects that have decided to forgo dependencies altogether are
forced to either duplicate the efforts of other projects by inventing
their own solutions to problems or are required to simply include the
other projects in their own source trees. Both of these options present
their own problems either in duplicating maintenance work across the
ecosystem or potentially leaving users vulnerable to security issues
because the included code or duplicated efforts are not automatically
updated when upstream releases a new version.

By officially recommending and providing by default a specific
cross-platform package manager it will be easier for users trying to
install these third-party packages as well as easier for the people
distributing them as they should now be able to safely assume that most
users will have the appropriate installation tools available (or access
to clear instructions on how to obtain them). This is expected to become
more important in the future as the Wheel\_ package format
(deliberately) does not have a built in "installer" in the form of
`setup.py` so users wishing to install from a wheel file will want an
installer even in the simplest cases.

Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24
month feature release cycle, instead of using the general difficulty of
installing third-party packages as justification for inclusion.

Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as `zc.buildout`, `hashdist`
and `conda`. So long as `pip install <tool>` works, then a standard
Python-specific installer provides a reasonably secure, cross platform
mechanism to get access to these utilities.

Enabling the evolution of the broader Python packaging ecosystem
----------------------------------------------------------------

As no new packaging standard can achieve widespread adoption without a
transition strategy that covers the versions of Python that are in
widespread *current* use (rather than merely future versions, like most
language features), the change proposed in this PEP is considered a
necessary step in the evolution of the Python packaging ecosystem

The broader community has embraced the Python Package Index as a
mechanism for distributing and installing Python software, but the
different concerns of language evolution and secure software
distribution mean that a faster feature release cycle that encompasses
older versions is needed to properly support the latter.

In addition, the core CPython development team have the luxury of
dropping support for earlier Python versions well before the rest of the
community, as downstream commercial redistributors pick up the task of
providing support for those versions to users that still need it, while
many third party libraries maintain compatibility with those versions as
long as they remain in widespread use.

This means that the current `setup.py install` based model for package
installation poses serious difficulties for the development and adoption
of new packaging standards, as, depending on how a project writes their
`setup.py` file, the installation command (along with other operations)
may end up invoking the standard library's `distutils` package.

As an indicator of how this may cause problems for the broader
ecosystem, consider that the feature set of `distutils` in Python 2.6
was frozen in June 2008 (with the release of Python 2.6b1), while the
feature set of `distutils` in Python 2.7 was frozen in April 2010 (with
the release of Python 2.7b1).

By contrast, using a separate installer application like `pip` (which
ensures that even `setup.py` files that invoke `distutils` directly
still support the new packaging standards) makes it possible to support
new packaging standards in older versions of Python, just by upgrading
`pip` (which receives new feature releases roughly every 6 months). The
situation on older versions of Python is further improved by making it
easier for end users to install and upgrade newer build systems like
`setuptools` or improved PyPI upload utilities like `twine`.

It is not coincidental that this proposed model of using a separate
installer program with more metadata heavy and less active distribution
formats matches that used by most operating systems (including Windows
since the introduction of the installer service and the MSI file
format), as well as many other language specific installers.

For Python 2.6, this compatibility issue is largely limited to various
enterprise Linux distributions (and their downstream derivatives). These
distributions often have even slower update cycles than CPython, so they
offer full support for versions of Python that are considered "security
fix only" versions upstream (and sometimes may even be to the point
where the core development team no longer support them at all - you can
still get commercial support for Python 2.3 if you really need it!).

In practice, the fact that tools like `wget` and `curl` are readily
available on Linux systems, that most users of Python on Linux are
already familiar with the command line, and that most Linux
distributions ship with a default configuration that makes running
Python scripts easy, means that the existing `pip` bootstrapping
instructions for any \*nix system are already quite straightforward.
Even if `pip` isn't provided by the system package manager, then using
`wget` or `curl` to retrieve the bootstrap script from
www.pip-installer.org and then running it is just a couple of shell
commands that can easily be copied and pasted as necessary.

Accordingly, for any version of Python on any \*nix system, the need to
bootstrap `pip` in older versions isn't considered a major barrier to
adoption of new packaging standards, since it's just one more small
speedbump encountered by users of these long term stable releases. For
\*nix systems, this PEP's formal endorsement of `pip` as the preferred
default packaging tool is seen as more important than the underlying
technical details involved in making `pip` available by default, since
it shifts the nature of the conversation between the developers of `pip`
and downstream repackagers of both `pip` and CPython.

For Python 2.7, on the other hand, the compatibility issue for adopting
new metadata standards is far more widespread, as it affects the
python.org binary installers for Windows and Mac OS X, as well as even
relatively fast moving \*nix platforms.

Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported
upstream version, and will remain so until the release of Python 2.7.9
(currently scheduled for May 2015), at which time it is expected to
enter the usual "security fix only" mode. That means there are at least
another 19 months where Python 2.7 is a deployment target for Python
applications that enjoys full upstream support. Even after the core
development team switches 2.7 to security release only mode in 2015,
Python 2.7 will likely remain a commercially supported legacy target out
beyond 2020.

While Python 3 already presents a compelling alternative over Python 2
for *new* Python applications and deployments without an existing
investment in Python 2 and without a dependency on specific Python 2
only third party modules (a set which is getting ever smaller over
time), it is going to take longer to create compelling business cases to
update existing Python 2.7 based infrastructure to Python 3, especially
in situations where the culture of automated testing is weak (or
nonexistent), making it difficult to effectively use the available
migration utilities.

While this PEP only proposes documentation changes for Python 2.7, once
`pip` has a Windows installer available, a separate PEP will be created
and submitted proposing the creation and distribution of aggregate
installers for future CPython 2.7 maintenance releases that combine the
CPython, `pip` and Python Launcher for Windows installers into a single
download (the separate downloads would still remain available - the
aggregate installers would be provided as a convenience, and as a clear
indication of the recommended operating environment for Python in
Windows systems).

Why pip?
--------

`pip` has been chosen as the preferred default installer, as it is an
already popular tool that addresses several design and user experience
issues with its predecessor `easy_install` (these issues can't readily
be fixed in `easy_install` itself due to backwards compatibility
concerns). `pip` is also well suited to working within the bounds of a
single Python runtime installation (including associated virtual
environments), which is a desirable feature for a tool bundled with
CPython.

Other tools like `zc.buildout` and `conda` are more ambitious in their
aims (and hence substantially better than `pip` at handling external
binary dependencies), so it makes sense for the Python ecosystem to
treat them more like platform package managers to interoperate with
rather than as the default cross-platform installation tool. This
relationship is similar to that between `pip` and platform package
management systems like `apt` and `yum` (which are also designed to
handle arbitrary binary dependencies).

Proposal Overview
=================

This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`\_\_ guide
be updated to officially recommend the use of `pip` as the default
installer for Python packages, rather than the current approach of
recommending the direct invocation of the `setup.py install` command.

However, to avoid recommending a tool that CPython does not provide, it
is further proposed that the `pip`\_ package manager be made available
by default when installing CPython 3.4 or later and when creating
virtual environments using the standard library's `venv` module via the
`pyvenv` command line utility.

To support that end, this PEP proposes the inclusion of an `ensurepip`
bootstrapping module in Python 3.4, as well as automatic invocation of
that module from `pyvenv` and changes to the way Python installed
scripts are handled on Windows. Using a bootstrap module rather than
providing `pip` directly helps to clearly demarcate development
responsibilities, and to avoid inadvertently downgrading `pip` when
updating CPython.

To provide clear guidance for new users of Python that may not be
starting with the latest release, this PEP also proposes that the
"Installing Python Modules" guides in Python 2.7 and 3.3 be updated to
recommend installing and using `pip`, rather than invoking `distutils`
directly. It does *not* propose backporting any of the code changes that
are being proposed for Python 3.4.

Finally, the PEP also strongly recommends that CPython redistributors
and other Python implementations ensure that `pip` is available by
default, or at the very least, explicitly document the fact that it is
not included.

This PEP does *not* propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience of
Python users, but subject to its own development life cycle and able to
be upgraded independently of the core interpreter and standard library.

Explicit bootstrapping mechanism
================================

An additional module called `ensurepip` will be added to the standard
library whose purpose is to install pip and any of its dependencies into
the appropriate location (most commonly site-packages). It will expose a
callable named `bootstrap()` as well as offer direct execution via
`python -m ensurepip`.

The bootstrap will *not* contact PyPI, but instead rely on a private
copy of pip stored inside the standard library. Accordingly, only
options related to the installation location will be supported
(`--user`, `--root`, etc).

It is considered desirable that users be strongly encouraged to use the
latest available version of `pip`, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability
and flexibility of that ecosystem.

In order to satisfy this goal of providing the most recent version of
`pip` by default, the private copy of `pip` will be updated in CPython
maintenance releases, which should align well with the 6-month cycle
used for new `pip` releases.

Security considerations
-----------------------

The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of CPython for end users that do
not subsequently run the command `pip install --upgrade pip`.

The installers will contain all the components of a fully functioning
version of Python, including the `pip` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
`pip` and the Python package index.

Only users that choose to use `pip` to communicate with PyPI will need
to pay attention to the additional security considerations that come
with doing so.

However, the core CPython team will still assist with reviewing and
resolving at least the
`certificate update management issue <https://github.com/kennethreitz/requests/issues/1659>`\_\_
currently affecting the `requests` project (and hence `pip`), and may
also be able to offer assistance in resolving other identified security
concerns \[\#cert-verification\]\_.

Reliability considerations
--------------------------

By including the bootstrap as part of the standard library (rather than
solely as a feature of the binary installers), the correct operation of
the bootstrap command can be easily tested using the existing CPython
buildbot infrastructure rather than adding significantly to the testing
burden for the installers themselves.

Implementation strategy
-----------------------

To ensure there is no need for network access when installing Python or
creating virtual environments, the `ensurepip` module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the
target environment. It is important to stress that this private copy of
pip is *only* an implementation detail and it should *not* be relied on
or assumed to exist beyond the public capabilities exposed through the
`ensurepip` module (and indirectly through `venv`).

There is not yet a reference `ensurepip` implementation. The existing
`get-pip.py` bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage
of the improved distribution capabilities offered by the CPython
installers to include private copies of `pip` and `setuptools` as wheel
files (rather than as embedded base64 encoded data), and would not try
to contact PyPI (instead installing directly from the private wheel
files).

Rather than including separate code to handle the bootstrapping, the
`ensurepip` module will manipulate `sys.path` appropriately to allow the
wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).

It is proposed that the implementation be carried out in five separate
steps (all steps after the first two are independent of each other and
can be carried out in any order):

-   the first step would update the "Installing Python Modules"
    documentation to recommend the use of `pip` and reference the `pip`
    team's instructions for downloading and installing it. This change
    would be applied to Python 2.7, 3.3, and 3.4.
-   the `ensurepip` module and the private copies of the most recently
    released versions of pip and setuptools would be added to Python 3.4
    and the 3.4 "Installing Python Modules" documentation updated
    accordingly.
-   the CPython Windows installer would be updated to offer the new
    `pip` installation option for Python 3.4.
-   the CPython Mac OS X installer would be updated to offer the new
    `pip` installation option for Python 3.4.
-   the `venv` module and `pyvenv` command would be updated to make use
    of `ensurepip` in Python 3.4
-   the PATH handling on Windows would be updated for Python 3.4+

Integration timeline
--------------------

If this PEP is accepted, the proposed time frame for integration of
`pip` into the CPython release is as follows:

-   as soon as possible after the release of 3.4.0 alpha 4

    -   Documentation updated and `ensurepip` implemented based on a
        pre-release version of `pip` 1.5.

    -   All other proposed functional changes for Python 3.4
        implemented, including the installer updates to invoke
        `ensurepip`.

-   by November 20th (3 days prior to the scheduled date of 3.4.0
    beta 1)

    -   `ensurepip` updated to use a `pip` 1.5 release candidate.

    -   PEP 101 updated to cover ensuring the bundled version of `pip`
        is up to date.

-   by November 24th (scheduled date of 3.4.0 beta 1)

    -   As with any other new feature, all proposed functional changes
        for Python 3.4 must be implemented prior to the beta feature
        freeze.

-   by December 29th (1 week prior to the scheduled date of 3.4.0
    beta 2)

    -   `requests` certificate management issue resolved
    -   `ensurepip` updated to the final release of `pip` 1.5, or a
        subsequent maintenance release (including a suitably updated
        vendored copy of `requests`)

(See PEP 429 for the current official scheduled dates of each release.
Dates listed above are accurate as of October 20th, 2013.)

If there is no final or maintenance release of `pip` 1.5 with a suitable
updated version of `requests` available by one week before the scheduled
Python 3.4 beta 2 release, then implementation of this PEP will be
deferred to Python 3.5. Note that this scenario is considered unlikely -
the tentative date for the `pip` 1.5 release is currently December 1st.

In future CPython releases, this kind of coordinated scheduling
shouldn't be needed: the CPython release manager will be able to just
update to the latest released version of `pip`. However, in this case,
some fixes are needed in `pip` in order to allow the bundling to work
correctly, and the certificate update mechanism for `requests` needs to
be improved, so the `pip` 1.5 release cycle needs to be properly aligned
with the CPython 3.4 beta releases.

Proposed CLI
------------

The proposed CLI is based on a subset of the existing `pip install`
options::

    Usage:
      python -m ensurepip [options]

    General Options:
      -h, --help          Show help.
      -v, --verbose       Give more output. Option is additive, and can be used up to 3 times.
      -V, --version       Show the pip version that would be extracted and exit.
      -q, --quiet         Give less output.

    Installation Options:
      -U, --upgrade       Upgrade pip and dependencies, even if already installed
      --user              Install using the user scheme.
      --root <dir>        Install everything relative to this alternate root directory.

In most cases, end users won't need to use this CLI directly, as `pip`
should have been installed automatically when installing Python or when
creating a virtual environment. However, it is formally documented as a
public interface to support at least these known use cases:

-   Windows and Mac OS X installations where the "Install pip" option
    was *not* chosen during installation
-   any installation where the user previously ran "pip uninstall pip"

Users that want to retrieve the latest version from PyPI, or otherwise
need more flexibility, can then invoke the extracted `pip`
appropriately.

Proposed module API
-------------------

The proposed `ensurepip` module API consists of the following two
functions::

    def version():
        """
        Returns a string specifying the bundled version of pip.
        """

    def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
        """
        Bootstrap pip into the current Python installation (or the given root
        directory).
        """

Invocation from the CPython installers
--------------------------------------

The CPython Windows and Mac OS X installers will each gain a new option:

-   Install pip (the default Python package management utility)?

This option will be checked by default.

If the option is checked, then the installer will invoke the following
command with the just installed Python::

    python -m ensurepip --upgrade

This ensures that, by default, installing or updating CPython will
ensure that the installed version of pip is at least as recent as the
one included with that version of CPython. If a newer version of pip has
already been installed then `python -m ensurepip --upgrade` will simply
return without doing anything.

Installing from source
----------------------

Just as the prebuilt binary installers will be updated to run
`python -m ensurepip` by default, a similar change will be made to the
`make install` and `make altinstall` commands of the source
distribution. The directory settings in the `sysconfig` module should
ensure the `pip` components are automatically installed to the expected
locations.

`ensurepip` itself (including the private copy of `pip` and its
dependencies) will always be installed normally (as it is a regular part
of the standard library), but an option will be provided to skip the
invocation of `ensurepip`.

This means that even installing from source will provide `pip` by
default, but redistributors provide `pip` by other means (or not
providing it at all) will still be able to opt out of installing it
using `ensurepip`.

Changes to virtual environments
-------------------------------

Python 3.3 included a standard library approach to virtual Python
environments through the `venv` module. Since its release it has become
clear that very few users have been willing to use this feature
directly, in part due to the lack of an installer present by default
inside of the virtual environment. They have instead opted to continue
using the `virtualenv` package which *does* include pip installed by
default.

To make the `venv` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating
it. This will allow people the same convenience inside of the virtual
environment as this PEP provides outside of it as well as bringing the
`venv` module closer to feature parity with the external `virtualenv`
package, making it a more suitable replacement.

To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a `--without-pip` option will be added.

The `venv.EnvBuilder` and `venv.create` APIs will be updated to accept
one new parameter: `with_pip` (defaulting to `False`).

The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
`venv` module happens through third part tools that likely will not want
`pip` installed without explicitly requesting it), while the default for
the command line interface is chosen to try to ensure `pip` is available
in most virtual environments without additional action on the part of
the end user.

As this change will only benefit Python 3.4 and later versions, the
third-party `virtualenv` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.

Documentation
-------------

The "Installing Python Modules" section of the standard library
documentation in Python 2.7, 3.3 and 3.4 will be updated to recommend
the use of the `pip` installer, either provided by default in Python 3.4
or retrieved and installed by the user in Python 2.7 or 3.3. It will
give a brief description of the most common commands and options, but
delegate to the externally maintained `pip` documentation for the full
details.

In Python 3.4, the `pyvenv` and `venv` documentation will also be
updated to reference the revised module installation guide.

The existing content of the module installation guide will be retained
in all versions, but under a new "Invoking distutils directly"
subsection.

Bundling CA certificates with CPython
-------------------------------------

The `ensurepip` implementation will include the `pip` CA bundle along
with the rest of `pip`. This means CPython effectively includes a CA
bundle that is used solely by `pip` after it has been extracted.

This is considered preferable to relying solely on the system
certificate stores, as it ensures that `pip` will behave the same across
all supported versions of Python, even those prior to Python 3.4 that
cannot access the system certificate store on Windows.

Automatic installation of setuptools
------------------------------------

`pip` currently depends on `setuptools` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be
current when Python 3.4.0 is released).

This PEP proposes that, if pip still requires it as a dependency,
`ensurepip` will include a private copy of `setuptools` (in addition to
the private copy of `ensurepip`). `python -m ensurepip` will then
install the private copy in addition to installing `pip` itself.

However, this behavior is officially considered an implementation
detail. Other projects which explicitly require `setuptools` must still
provide an appropriate dependency declaration, rather than assuming
`setuptools` will always be installed alongside `pip`.

Once pip is able to run `pip install --upgrade pip` without needing
`setuptools` installed first, then the private copy of `setuptools` will
be removed from `ensurepip` in subsequent CPython releases.

As long as setuptools is needed, it will be a completely unmodified copy
of the latest upstream setuptools release, including the `easy_install`
script if the upstream setuptools continues to include it. The
installation of `easy_install` along with `pip` isn't considered
desirable, but installing a broken setuptools would be worse. This
problem will naturally resolve itself once the `pip` developers have
managed to eliminate their dependency on `setuptools` and the private
copy of `setuptools` can be removed entirely from CPython.

Updating the private copy of pip
--------------------------------

In order to keep up with evolutions in packaging as well as providing
users with as recent version a possible the `ensurepip` module will be
regularly updated to the latest versions of everything it bootstraps.

After each new `pip` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as
part of the implementation for this PEP, will be run to ensure the
private copies stored in the CPython source repository have been updated
to the latest versions.

Updating the ensurepip module API and CLI
-----------------------------------------

Like `venv` and `pyvenv`, the `ensurepip` module API and CLI will be
governed by the normal rules for the standard library: no new features
are permitted in maintenance releases.

However, the embedded components may be updated as noted above, so the
extracted `pip` may offer additional functionality in maintenance
releases.

Uninstallation
==============

No changes are proposed to the CPython uninstallation process by this
PEP. The bootstrapped pip will be installed the same way as any other
pip installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.

At least on Windows, that means the bootstrapped files will be left
behind after uninstallation, since those files won't be associated with
the Python MSI installer.

While the case can be made for the CPython installers clearing out these
directories automatically, changing that behaviour is considered outside
the scope of this PEP.

Script Execution on Windows
===========================

While the Windows installer was updated in Python 3.3 to optionally make
`python` available on the PATH, no such change was made to include the
script installation directory returned by
`sysconfig.get_path("scripts")`.

Accordingly, in addition to adding the option to extract and install
`pip` during installation, this PEP proposes that the Windows installer
in Python 3.4 and later be updated to also add the path returned by
`sysconfig.get_path("scripts")` to the Windows PATH when the PATH
modification option is enabled during installation

Note that this change will only be available in Python 3.4 and later.

This means that, for Python 3.3, the most reliable way to invoke pip
globally on Windows (without tinkering manually with PATH) will still
remain `py -m pip` (or `py -3 -m pip` to select the Python 3 version if
both Python 2 and 3 are installed) rather than simply calling `pip`.
This works because Python 3.3 provides the Python Launcher for Windows
(and the associated `py` command) by default.

For Python 2.7 and 3.2, the most reliable mechanism will be to install
the Python Launcher for Windows using the standalone installer and then
use `py -m pip` as noted above.

Adding the scripts directory to the system PATH will mean that `pip`
works reliably in the "only one Python installation on the system PATH"
case, with `py -m pip`, `pipX`, or `pipX.Y` needed only to select a
non-default version in the parallel installation case (and outside a
virtual environment). This change should also make the `pyvenv` command
substantially easier to invoke on Windows, along with all scripts
installed by `pip`, `easy_install` and similar tools.

While the script invocations on recent versions of Python will run
through the Python launcher for Windows, this shouldn't cause any
issues, as long as the Python files in the Scripts directory correctly
specify a Python version in their shebang line or have an adjacent
Windows executable (as `easy_install` and `pip` do).

Recommendations for Downstream Distributors
===========================================

A common source of Python installations are through downstream
distributors such as the various Linux Distributions \[\#ubuntu\]\_
\[\#debian\]\_ \[\#fedora\]*, OSX package managers \[\#homebrew\]*
\[\#macports\]\_ \[\#fink\]*, and commercial Python redistributors
\[\#ContinuumIO\]* \[\#ActiveState\]\_ \[\#Enthought\]\_. In order to
provide a consistent, user-friendly experience to all users of Python
regardless of how they obtained Python this PEP recommends and asks that
downstream distributors:

-   Ensure that whenever Python is installed `pip` is either installed
    or is otherwise made readily available to end users.

    -   For redistributors using binary installers, this may take the
        form of optionally executing the `ensurepip` bootstrap during
        installation, similar to the CPython installers.
    -   For redistributors using package management systems, it may take
        the form of separate packages with dependencies on each other so
        that installing the Python package installs the pip package and
        installing the pip package installs the Python package.
    -   Another reasonable way to implement this is to package pip
        separately but ensure that there is some sort of global hook
        that will recommend installing the separate pip package when a
        user executes `pip` without it being installed. Systems that
        choose this option should ensure that the `ensurepip` module
        still installs pip directly when invoked inside a virtual
        environment, but may modify the module in the system Python
        installation to redirect to the platform provided mechanism when
        installing `pip` globally.

-   Even if pip is made available globally by other means, do not remove
    the `ensurepip` module in Python 3.4 or later.

    -   `ensurepip` will be required for automatic installation of pip
        into virtual environments by the `venv` module.
    -   This is similar to the existing `virtualenv` package for which
        many downstream distributors have already made exception to the
        common "debundling" policy.
    -   This does mean that if `pip` needs to be updated due to a
        security issue, so does the private copy in the `ensurepip`
        bootstrap module
    -   However, altering the private copy of pip to remove the embedded
        CA certificate bundle and rely on the system CA bundle instead
        is a reasonable change.

-   Ensure that all features of this PEP continue to work with any
    modifications made to the redistributed version of Python.

    -   Checking the version of pip that will be bootstrapped using
        `python -m ensurepip --version` or `ensurepip.version()`.
    -   Installation of pip into a global or virtual python environment
        using `python -m ensurepip` or `ensurepip.bootstrap()`.
    -   `pip install --upgrade pip` in a global installation should not
        affect any already created virtual environments (but is
        permitted to affect future virtual environments, even though it
        will not do so when using the standard implementation of
        `ensurepip`).
    -   `pip install --upgrade pip` in a virtual environment should not
        affect the global installation.

-   Migrate build systems to utilize `pip`\_ and `Wheel`\_ wherever
    feasible and avoid directly invoking `setup.py`.

    -   This will help ensure a smoother and more timely migration to
        improved metadata formats as the Python packaging ecosystem
        continues to evolve.

In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream `pip`
based installation instructions into something appropriate for the
platform.

Other Python implementations are also encouraged to follow these
guidelines where applicable.

Policies & Governance
=====================

The maintainers of the bootstrapped software and the CPython core team
will work together in order to address the needs of both. The
bootstrapped software will still remain external to CPython and this PEP
does not include CPython subsuming the development responsibilities or
design decisions of the bootstrapped software. This PEP aims to decrease
the burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of `pip`, `setuptools`, PyPI, `virtualenv`
and other related projects.

Backwards Compatibility
-----------------------

The public API and CLI of the `ensurepip` module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does
not.

Most importantly, this means that the bootstrapped version of pip may
gain new features in CPython maintenance releases, and pip continues to
operate on its own 6 month release cycle rather than CPython's 18-24
month cycle.

Security Releases
-----------------

Any security update that affects the `ensurepip` module will be shared
prior to release with the Python Security Response Team
(security\@python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
`pip`.

Licensing
---------

`pip` is currently licensed as 1 Clause BSD, and it contains code taken
from other projects. Additionally this PEP will include setuptools until
such time as pip no longer requires it. The licenses for these appear in
the table below.

================= ============ Project License =================
============ requests Apache 2.0 six 1 Clause BSD html5lib 1 Clause BSD
distlib PSF colorama 3 Clause BSD Mozilla CA Bundle LGPL setuptools PSF
================= ============

All of these licenses should be compatible with the PSF license.
Additionally it is unclear if a CA Bundle is copyrightable material and
thus if it needs or can be licensed at all.

Appendix: Rejected Proposals
============================

Changing the name of the scripts directory on Windows
-----------------------------------------------------

Earlier versions of this PEP proposed changing the name of the script
installation directory on Windows from "Scripts" to "bin" in order to
improve the cross-platform consistency of the virtual environments
created by `pyvenv`.

However, Paul Moore determined that this change was likely backwards
incompatible with cross-version Windows installers created with previous
versions of Python, so the change has been removed from this PEP
\[\#windows-incompatibility\]\_.

Including ensurepip in Python 2.7, and 3.3
------------------------------------------

Earlier versions of this PEP made the case that the challenges of
getting `pip` bootstrapped for new users posed a significant enough
barrier to Python's future growth that it justified adding `ensurepip`
as a new feature in the upcoming Python 2.7 and 3.3 maintenance
releases.

While the proposal to provide `pip` with Python 3.4 was universally
popular, this part of the proposal was highly controversial and
ultimately
`rejected by MvL as BDFL-Delegate <https://mail.python.org/pipermail/python-dev/2013-September/129091.html>`\_\_.

Accordingly, the proposal to backport `ensurepip` to Python 2.7 and 3.3
has been removed from this PEP in favour of creating a Windows installer
for `pip` and a possible future PEP suggesting creation of an aggregate
installer for Python 2.7 that combines CPython 2.7, `pip` and the Python
Launcher for Windows.

Automatically contacting PyPI when bootstrapping pip
----------------------------------------------------

Earlier versions of this PEP called the bootstrapping module `getpip`
and defaulted to downloading and installing `pip` from PyPI, with the
private copy used only as a fallback option or when explicitly
requested.

This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also
significantly altered the default trust model for the binary installers
published on python.org, as end users would need to explicitly *opt-out*
of trusting the security of the PyPI ecosystem (rather than opting in to
it by explicitly invoking `pip` following installation).

As a result, the PEP was simplified to the current design, where the
bootstrapping *always* uses the private copy of `pip`. Contacting PyPI
is now always an explicit separate step, with direct access to the full
pip interface.

Removing the implicit attempt to access PyPI also made it feasible to
invoke `ensurepip` by default when installing from a custom source
build.

Implicit bootstrap
------------------

`PEP439`\_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake `pip` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This
has been rejected because it is too "magical". It hides from the end
user when exactly the pip command will be installed or that it is being
installed at all. It also does not provide any recommendations or
considerations towards downstream packagers who wish to manage the
globally installed pip through the mechanisms typical for their system.

The implicit bootstrap mechanism also ran into possible permissions
issues, if a user inadvertently attempted to bootstrap pip without write
access to the appropriate installation directories.

Including pip directly in the standard library
----------------------------------------------

Similar to this PEP is the proposal of just including pip in the
standard library. This would ensure that Python always includes pip and
fixes all of the end user facing problems with not having pip present by
default. This has been rejected because we've learned, through the
inclusion and history of `distutils` in the standard library, that
losing the ability to update the packaging tools independently can leave
the tooling in a state of constant limbo. Making it unable to ever
reasonably evolve in a time frame that actually affects users as any new
features will not be available to the general population for *years*.

Allowing the packaging tools to progress separately from the Python
release and adoption schedules allows the improvements to be used by
*all* members of the Python community and not just those able to live on
the bleeding edge of Python releases.

There have also been issues in the past with the "dual maintenance"
problem if a project continues to be maintained externally while *also*
having a fork maintained in the standard library. Since external
maintenance of `pip` will always be needed to support earlier Python
versions, the proposed bootstrapping mechanism will becoming the
explicit responsibility of the CPython core developers (assisted by the
pip developers), while pip issues reported to the CPython tracker will
be migrated to the pip issue tracker. There will no doubt still be some
user confusion over which tracker to use, but hopefully less than has
been seen historically when including complete public copies of
third-party projects in the standard library.

The approach described in this PEP also avoids some technical issues
related to handling CPython maintenance updates when pip has been
independently updated to a more recent version. The proposed pip-based
bootstrapping mechanism handles that automatically, since pip and the
system installer never get into a fight about who owns the pip
installation (it is always managed through pip, either directly, or
indirectly via the `ensurepip` bootstrap module).

Finally, the separate bootstrapping step means it is also easy to avoid
installing `pip` at all if end users so desire. This is often the case
if integrators are using system packages to handle installation of
components written in multiple languages using a common set of tools.

Defaulting to --user installation
---------------------------------

Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself) and
is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).

.. \_Wheel: http://www.python.org/dev/peps/pep-0427/ .. \_pip:
http://www.pip-installer.org .. \_setuptools:
https://pypi.python.org/pypi/setuptools .. \_PEP439:
http://www.python.org/dev/peps/pep-0439/

References
==========

.. \[1\] Discussion thread 1 (distutils-sig)
(https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html)

.. \[2\] Discussion thread 2 (distutils-sig)
(https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html)

.. \[3\] Discussion thread 3 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128723.html)

.. \[4\] Discussion thread 4 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128780.html)

.. \[5\] Discussion thread 5 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128894.html)

.. \[\#cert-verification\] pip/requests certificate management concerns
(https://mail.python.org/pipermail/python-dev/2013-October/129755.html)

.. \[\#windows-incompatibility\] Windows installer compatibility
concerns
(https://mail.python.org/pipermail/distutils-sig/2013-October/022855.html)

.. \[\#ubuntu\] `Ubuntu <http://www.ubuntu.com/>` .. \[\#debian\]
`Debian <http://www.debian.org>` .. \[\#fedora\]
`Fedora <https://fedoraproject.org/>` .. \[\#homebrew\]
`Homebrew <http://brew.sh/>` .. \[\#macports\]
`MacPorts <http://macports.org>` .. \[\#fink\]
`Fink <http://finkproject.org>` .. \[\#ContinuumIO\]
`Anaconda <https://store.continuum.io/cshop/anaconda/>` ..
\[\#ActiveState\]
`ActivePython <http://www.activestate.com/activepython>` ..
\[\#Enthought\]
`Enthought Canopy <https://www.enthought.com/products/canopy/>`

Copyright
=========

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
