PEP: 408 Title: Standard library **preview** package Version: $Revision$
Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan@gmail.com>, Eli
Bendersky <eliben@gmail.com> Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 2012-01-07 Python-Version: 3.3
Post-History: 2012-01-27 Resolution:
https://mail.python.org/pipermail/python-dev/2012-January/115962.html

Abstract
========

The process of including a new module into the Python standard library
is hindered by the API lock-in and promise of backward compatibility
implied by a module being formally part of Python. This PEP proposes a
transitional state for modules - inclusion in a special `__preview__`
package for the duration of a minor release (roughly 18 months) prior to
full acceptance into the standard library. On one hand, this state
provides the module with the benefits of being formally part of the
Python distribution. On the other hand, the core development team
explicitly states that no promises are made with regards to the module's
eventual full inclusion into the standard library, or to the stability
of its API, which may change for the next release.

PEP Rejection
=============

Based on his experience with a similar "labs" namespace in Google App
Engine, Guido has rejected this PEP \[3\]\_ in favour of the simpler
alternative of explicitly marking provisional modules as such in their
documentation.

If a module is otherwise considered suitable for standard library
inclusion, but some concerns remain regarding maintainability or certain
API details, then the module can be accepted on a provisional basis.
While it is considered an unlikely outcome, such modules *may* be
removed from the standard library without a deprecation period if the
lingering concerns prove well-founded.

As part of the same announcement, Guido explicitly accepted Matthew
Barnett's 'regex' module \[4\]\_ as a provisional addition to the
standard library for Python 3.3 (using the 'regex' name, rather than as
a drop-in replacement for the existing 're' module).

Proposal - the **preview** package
==================================

Whenever the Python core development team decides that a new module
should be included into the standard library, but isn't entirely sure
about whether the module's API is optimal, the module can be placed in a
special package named `__preview__` for a single minor release.

In the next minor release, the module may either be "graduated" into the
standard library (and occupy its natural place within its namespace,
leaving the `__preview__` package), or be rejected and removed entirely
from the Python source tree. If the module ends up graduating into the
standard library after spending a minor release in `__preview__`, its
API may be changed according to accumulated feedback. The core
development team explicitly makes no guarantees about API stability and
backward compatibility of modules in `__preview__`.

Entry into the `__preview__` package marks the start of a transition of
the module into the standard library. It means that the core development
team assumes responsibility of the module, similarly to any other module
in the standard library.

Which modules should go through `__preview__`
---------------------------------------------

We expect most modules proposed for addition into the Python standard
library to go through a minor release in `__preview__`. There may,
however, be some exceptions, such as modules that use a pre-defined API
(for example `lzma`, which generally follows the API of the existing
`bz2` module), or modules with an API that has wide acceptance in the
Python development community.

In any case, modules that are proposed to be added to the standard
library, whether via `__preview__` or directly, must fulfill the
acceptance conditions set by PEP 2.

It is important to stress that the aim of this proposal is not to make
the process of adding new modules to the standard library more
difficult. On the contrary, it tries to provide a means to add *more*
useful libraries. Modules which are obvious candidates for entry can be
added as before. Modules which due to uncertainties about the API could
be stalled for a long time now have a means to still be distributed with
Python, via an incubation period in the `__preview__` package.

Criteria for "graduation"
-------------------------

In principle, most modules in the `__preview__` package should
eventually graduate to the stable standard library. Some reasons for not
graduating are:

-   The module may prove to be unstable or fragile, without sufficient
    developer support to maintain it.
-   A much better alternative module may be found during the preview
    release

Essentially, the decision will be made by the core developers on a
per-case basis. The point to emphasize here is that a module's
appearance in the `__preview__` package in some release does not
guarantee it will continue being part of Python in the next release.

Example
-------

Suppose the `example` module is a candidate for inclusion in the
standard library, but some Python developers aren't convinced that it
presents the best API for the problem it intends to solve. The module
can then be added to the `__preview__` package in release `3.X`,
importable via::

    from __preview__ import example

Assuming the module is then promoted to the standard library proper in
release `3.X+1`, it will be moved to a permanent location in the
library::

    import example

And importing it from `__preview__` will no longer work.

Rationale
=========

Benefits for the core development team
--------------------------------------

Currently, the core developers are really reluctant to add new
interfaces to the standard library. This is because as soon as they're
published in a release, API design mistakes get locked in due to
backward compatibility concerns.

By gating all major API additions through some kind of a preview
mechanism for a full release, we get one full release cycle of community
feedback before we lock in the APIs with our standard backward
compatibility guarantee.

We can also start integrating preview modules with the rest of the
standard library early, so long as we make it clear to packagers that
the preview modules should not be considered optional. The only
difference between preview APIs and the rest of the standard library is
that preview APIs are explicitly exempted from the usual backward
compatibility guarantees.

Essentially, the `__preview__` package is intended to lower the risk of
locking in minor API design mistakes for extended periods of time.
Currently, this concern can block new additions, even when the core
development team consensus is that a particular addition is a good idea
in principle.

Benefits for end users
----------------------

For future end users, the broadest benefit lies in a better
"out-of-the-box" experience - rather than being told "oh, the standard
library tools for task X are horrible, download this 3rd party library
instead", those superior tools are more likely to be just be an import
away.

For environments where developers are required to conduct due diligence
on their upstream dependencies (severely harming the cost-effectiveness
of, or even ruling out entirely, much of the material on PyPI), the key
benefit lies in ensuring that anything in the `__preview__` package is
clearly under python-dev's aegis from at least the following
perspectives:

-   Licensing: Redistributed by the PSF under a Contributor Licensing
    Agreement.
-   Documentation: The documentation of the module is published and
    organized via the standard Python documentation tools (i.e. ReST
    source, output generated with Sphinx and published on
    http://docs.python.org).
-   Testing: The module test suites are run on the python.org buildbot
    fleet and results published via http://www.python.org/dev/buildbot.
-   Issue management: Bugs and feature requests are handled on
    http://bugs.python.org
-   Source control: The master repository for the software is published
    on http://hg.python.org.

Candidates for inclusion into **preview**
=========================================

For Python 3.3, there are a number of clear current candidates:

-   `regex` (http://pypi.python.org/pypi/regex)
-   `daemon` (PEP 3143)
-   `ipaddr` (PEP 3144)

Other possible future use cases include:

-   Improved HTTP modules (e.g. `requests`)
-   HTML 5 parsing support (e.g. `html5lib`)
-   Improved URL/URI/IRI parsing
-   A standard image API (PEP 368)
-   Encapsulation of the import state (PEP 368)
-   Standard event loop API (PEP 3153)
-   A binary version of WSGI for Python 3 (e.g. PEP 444)
-   Generic function support (e.g. `simplegeneric`)

Relationship with PEP 407
=========================

PEP 407 proposes a change to the core Python release cycle to permit
interim releases every 6 months (perhaps limited to standard library
updates). If such a change to the release cycle is made, the following
policy for the `__preview__` namespace is suggested:

-   For long term support releases, the `__preview__` namespace would
    always be empty.
-   New modules would be accepted into the `__preview__` namespace only
    in interim releases that immediately follow a long term support
    release.
-   All modules added will either be migrated to their final location in
    the standard library or dropped entirely prior to the next long term
    support release.

Rejected alternatives and variations
====================================

Using `__future__`
------------------

Python already has a "forward-looking" namespace in the form of the
`__future__` module, so it's reasonable to ask why that can't be re-used
for this new purpose.

There are two reasons why doing so not appropriate:

1.  The `__future__` module is actually linked to a separate compiler
    directives feature that can actually change the way the Python
    interpreter compiles a module. We don't want that for the preview
    package - we just want an ordinary Python package.

2.  The `__future__` module comes with an express promise that names
    will be maintained in perpetuity, long after the associated features
    have become the compiler's default behaviour. Again, this is
    precisely the opposite of what is intended for the preview package -
    it is almost certain that all names added to the preview will be
    removed at some point, most likely due to their being moved to a
    permanent home in the standard library, but also potentially due to
    their being reverted to third party package status (if community
    feedback suggests the proposed addition is irredeemably broken).

Versioning the package
----------------------

One proposed alternative \[1\]\_ was to add explicit versioning to the
`__preview__` package, i.e. `__preview34__`. We think that it's better
to simply define that a module being in `__preview__` in Python 3.X will
either graduate to the normal standard library namespace in Python 3.X+1
or will disappear from the Python source tree altogether. Versioning the
`_preview__` package complicates the process and does not align well
with the main intent of this proposal.

Using a package name without leading and trailing underscores
-------------------------------------------------------------

It was proposed \[1\]\_ to use a package name like `preview` or `exp`,
instead of `__preview__`. This was rejected in the discussion due to the
special meaning a "dunder" package name (that is, a name *with* leading
and trailing double-underscores) conveys in Python. Besides, a
non-dunder name would suggest normal standard library API stability
guarantees, which is not the intention of the `__preview__` package.

Preserving pickle compatibility
-------------------------------

A pickled class instance based on a module in `__preview__` in release
3.X won't be unpickle-able in release 3.X+1, where the module won't be
in `__preview__`. Special code may be added to make this work, but this
goes against the intent of this proposal, since it implies backward
compatibility. Therefore, this PEP does not propose to preserve pickle
compatibility.

Credits
=======

Dj Gilcrease initially proposed the idea of having a `__preview__`
package in Python \[2\]\_. Although his original proposal uses the name
`__experimental__`, we feel that `__preview__` conveys the meaning of
this package in a better way.

References
==========

.. \[1\] Discussed in this thread:
https://mail.python.org/pipermail/python-ideas/2012-January/013246.html

.. \[2\]
https://mail.python.org/pipermail/python-ideas/2011-August/011278.html

.. \[3\] Guido's decision:
https://mail.python.org/pipermail/python-dev/2012-January/115962.html

.. \[4\] Proposal for inclusion of regex:
http://bugs.python.org/issue2636

Copyright
=========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
