PEP: 606 Title: Python Compatibility Version Author: Victor Stinner
\<<vstinner@python.org>\> Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 18-Oct-2019 Python-Version: 3.9

Abstract
========

Add `sys.set_python_compat_version(version)` to enable partial
compatibility with requested Python version. Add
`sys.get_python_compat_version()`.

Modify a few functions in the standard library to implement partial
compatibility with Python 3.8.

Add `sys.set_python_min_compat_version(version)` to deny backward
compatibility with Python versions older than *version*.

Add `-X compat_version=VERSION` and `-X min_compat_version=VERSION`
command line options. Add `PYTHONCOMPATVERSION` and
`PYTHONCOMPATMINVERSION` environment variables.

Rationale
=========

The need to evolve frequently
-----------------------------

To remain relevant and useful, Python has to evolve frequently; some
enhancements require incompatible changes. Any incompatible change can
break an unknown number of Python projects. Developers can decide to not
implement a feature because of that.

Users want to get the latest Python version to obtain new features and
better performance. A few incompatible changes can prevent them from
using their applications on the latest Python version.

This PEP proposes to add a partial compatibility with old Python
versions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not that
Python 3 is backward incompatible, but how incompatible changes were
introduced.

Partial compatibility to minimize the Python maintenance burden
---------------------------------------------------------------

While technically it would be possible to provide full compatibility
with old Python versions, this PEP proposes to minimize the number of
functions handling backward compatibility to reduce the maintenance
burden of the Python project (CPython).

Each change introducing backport compatibility to a function should be
properly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped on each Python release, on a
case by case basis. Each compatibility function can be supported for a
different number of Python releases depending on its maintenance cost
and the estimated risk (number of broken projects) if it\'s removed.

The maintenance cost does not only come from the code implementing the
backward compatibility, but also comes from the additional tests.

Cases excluded from backward compatibility
------------------------------------------

The performance overhead of any compatibility code must be low when
`sys.set_python_compat_version()` is not called.

The C API is out of the scope of this PEP: `Py_LIMITED_API` macro and
the stable ABI are solving this problem differently, see the [PEP 384:
Defining a Stable ABI](https://www.python.org/dev/peps/pep-0384/).

Security fixes which break backward compatibility on purpose will not
get a compatibility layer; security matters more than compatibility. For
example, `http.client.HTTPSConnection` was modified in Python 3.4.3 to
performs all the necessary certificate and hostname checks by default.
It was a deliberate change motivated by [PEP 476: Enabling certificate
verification by default for stdlib http
clients](https://www.python.org/dev/peps/pep-0476/)
([bpo-22417](https://bugs.python.org/issue22417)).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP.
For example, Python 3.9 changed the default protocol in the `pickle`
module to Protocol 4 which was first introduced in Python 3.4. This
change is backward compatible up to Python 3.4. There is no need to use
the Protocol 3 by default when compatibility with Python 3.8 is
requested.

The new `DeprecationWarning` and `PendingDeprecatingWarning` warnings in
Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a
project runs its test suite using `-Werror` (treat any warning as an
error), these warnings must be fixed, or specific deprecation warnings
must be ignored on a case by case basis.

Upgrading a project to a newer Python
-------------------------------------

Without backward compatibility, all incompatible changes must be fixed
at once, which can be a blocker issue. It is even worse when a project
is upgraded to a newer Python which is separated by multiple releases
from the old Python.

Postponing an upgrade only makes things worse: each skipped release adds
more incompatible changes. The technical debt only steadily increases
over time.

With backward compatibility, it becomes possible to upgrade Python
increamentally in a project, without having to fix all of the issues at
once.

The \"all-or-nothing\" is a showstopper to port large Python 2 code
bases to Python 3. The list of incompatible changes between Python 2 and
Python 3 is long, and it\'s getting longer with each Python 3.x release.

Cleaning up Python and DeprecationWarning
-----------------------------------------

One of the [Zen of Python (PEP
20)](https://www.python.org/dev/peps/pep-0020/) motto is:

> There should be one\-- and preferably only one \--obvious way to do
> it.

When Python evolves, new ways inevitably emerge. `DeprecationWarning`s
are emitted to suggest using the new way, but many developers ignore
these warnings, which are silent by default (except in the `__main__`
module: see the [PEP 565](https://www.python.org/dev/peps/pep-0565/)).
Some developers simply ignore all warnings when there are too many
warnings, thus only bother with exceptions when the deprecated code is
removed.

Sometimes, supporting both ways has a minor maintenance cost, but
developers prefer to drop the old way to clean up their code. These
kinds of changes are backward incompatible.

Some developers can take the end of the Python 2 support as an
opportunity to push even more incompatible changes than usual.

Adding an opt-in backward compatibility prevents the breaking of
applications and allows developers to continue doing these cleanups.

Redistribute the maintenance burden
-----------------------------------

The backward compatibility involves authors of incompatible changes more
in the upgrade path.

Examples of backward compatibility
==================================

collections ABC aliases
-----------------------

`collections.abc` aliases to ABC classes have been removed from the
`collections` module in Python 3.9, after being deprecated since Python
3.3. For example, `collections.Mapping` no longer exists.

In Python 3.6, aliases were created in `collections/__init__.py` by
`from _collections_abc import *`.

In Python 3.7, a `__getattr__()` has been added to the `collections`
module to emit a DeprecationWarning upon first access to an attribute:

    def __getattr__(name):
        # For backwards compatibility, continue to make the collections ABCs
        # through Python 3.6 available through the collections module.
        # Note: no new collections ABCs were added in Python 3.7
        if name in _collections_abc.__all__:
            obj = getattr(_collections_abc, name)
            import warnings
            warnings.warn("Using or importing the ABCs from 'collections' instead "
                          "of from 'collections.abc' is deprecated since Python 3.3, "
                          "and in 3.9 it will be removed.",
                          DeprecationWarning, stacklevel=2)
            globals()[name] = obj
            return obj
        raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by adding
back the `__getattr__()` function, but only when backward compatibility
is requested:

    def __getattr__(name):
        if (sys.get_python_compat_version() < (3, 9)
           and name in _collections_abc.__all__):
            ...
        raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Deprecated open() \"U\" mode
----------------------------

The `"U"` mode of `open()` is deprecated since Python 3.4 and emits a
`DeprecationWarning`. [bpo-37330](https://bugs.python.org/issue37330)
proposes to drop this mode: `open(filename, "rU")` would raise an
exception.

This change falls into the \"cleanup\" category: it is not required to
implement a feature.

A backward compatibility mode would be trivial to implement and would be
welcomed by users.

Specification
=============

sys functions
-------------

Add 3 functions to the `sys` module:

-   `sys.set_python_compat_version(version)`: set the Python
    compatibility version. If it has been called previously, use the
    minimum of requested versions. Raise an exception if
    `sys.set_python_min_compat_version(min_version)` has been called and
    `version < min_version`. *version* must be greater than or equal to
    `(3, 0)`.
-   `sys.set_python_min_compat_version(min_version)`: set the
    **minimum** compatibility version. Raise an exception if
    `sys.set_python_compat_version(old_version)` has been called
    previously and `old_version < min_version`. *min\_version* must be
    greater than or equal to `(3, 0)`.
-   `sys.get_python_compat_version()`: get the Python compatibility
    version. Return a `tuple` of 3 integers.

A *version* must a tuple of 2 or 3 integers. `(major, minor)` version is
equivalent to `(major, minor, 0)`.

By default, `sys.get_python_compat_version()` returns the current Python
version.

For example, to request compatibility with Python 3.8.0:

    import collections

    sys.set_python_compat_version((3, 8))

    # collections.Mapping alias, removed from Python 3.9, is available
    # again, even if collections has been imported before calling
    # set_python_compat_version().
    parent = collections.Mapping

Obviously, calling `sys.set_python_compat_version(version)` has no
effect on code executed before the call. Use `-X compat_version=VERSION`
command line option or `PYTHONCOMPATVERSIONVERSION=VERSION` environment
variable to set the compatibility version at Python startup.

Command line
------------

Add `-X compat_version=VERSION` and `-X min_compat_version=VERSION`
command line options: call respectivelly
`sys.set_python_compat_version()` and
`sys.set_python_min_compat_version()`. `VERSION` is a version string
with 2 or 3 numbers (`major.minor.micro` or `major.minor`). For example,
`-X compat_version=3.8` calls `sys.set_python_compat_version((3, 8))`.

Add `PYTHONCOMPATVERSIONVERSION=VERSION` and
`PYTHONCOMPATMINVERSION=VERSION=VERSION` environment variables: call
respectivelly `sys.set_python_compat_version()` and
`sys.set_python_min_compat_version()`. `VERSION` is a version string
with the same format as the command line options.

Backwards Compatibility
=======================

Introducing the `sys.set_python_compat_version()` function means that an
application will behave differently depending on the compatibility
version. Moreover, since the version can be decreased multiple times,
the application can behave differently depending on the import order.

Python 3.9 with `sys.set_python_compat_version((3, 8))` is not fully
compatible with Python 3.8: the compatibility is only partial.

Security Implications
=====================

`sys.set_python_compat_version()` must not disable security fixes.

Alternatives
============

Provide a workaround for each incompatible change
-------------------------------------------------

An application can work around most incompatible changes which impacts
it.

For example, `collections` aliases can be added back using:

    import collections.abc
    collections.Mapping = collections.abc.Mapping
    collections.Sequence = collections.abc.Sequence

Handle backward compatibility in the parser
-------------------------------------------

The parser is modified to support multiple versions of the Python
language (grammar).

The current Python parser cannot be easily modified for that. AST and
grammar are hardcoded to a single Python version.

In Python 3.8, `compile()` has an undocumented `_feature_version` to not
consider `async` and `await` as keywords.

The latest major language backward incompatible change was Python 3.7
which made `async` and `await` real keywords. It seems like Twisted was
the only affected project, and Twisted had a single affected function
(it used a parameter called `async`).

Handling backward compatibility in the parser seems quite complex, not
only to modify the parser, but also for developers who have to check
which version of the Python language is used.

### from \_\_future\_\_ import python38\_syntax

Add `pythonXY_syntax` to the `__future__` module. It would enable
backward compatibility with Python X.Y syntax, but only for the current
file.

With this option, there is no need to change
`sys.implementation.cache_tag` to use a different `.pyc` filename, since
the parser will always produce the same output for the same input
(except for the optimization level).

For example:

    from __future__ import python35_syntax

    async = 1
    await = 2

### Update cache\_tag

Modify the parser to use `sys.get_python_compat_version()` to choose the
version of the Python language.

`sys.set_python_compat_version()` updates `sys.implementation.cache_tag`
to include the compatibility version without the micro version as a
suffix. For example, Python 3.9 uses `'cpython-39'` by default, but
`sys.set_python_compat_version((3, 7, 2))` sets `cache_tag` to
`'cpython-39-37'`. Changes to the Python language are now allowed in
micro releases.

One problem is that `import asyncio` is likely to fail if
`sys.set_python_compat_version((3, 6))` has been called previously. The
code of the `asyncio` module requires `async` and `await` to be real
keywords (change done in Python 3.7).

Another problem is that regular users cannot write `.pyc` files into
system directories, and so cannot create them on demand. It means that
`.pyc` optimization cannot be used in the backward compatibility mode.

One solution for that is to modify the Python installer and Python
package installers to precompile `.pyc` files not only for the current
Python version, but also for multiple older Python versions (up to
Python 3.0?).

Each `.py` file would have 3n `.pyc` files (3 optimization levels),
where `n` is the number of supported Python versions. For example, it
means 6 `.pyc` files, instead of 3, to support Python 3.8 and Python
3.9.

Temporary moratorium on incompatible changes
--------------------------------------------

In 2009, PEP 3003 \"Python Language Moratorium\" proposed a temporary
moratorium (suspension) of all changes to the Python language syntax,
semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during the PEP 572 discussions, it was also proposed to
slow down Python changes: see the python-dev thread [Slow
down\...](https://mail.python.org/archives/list/python-dev@python.org/thread/HHKRXOMRJQH75VNM3JMSQIOOU6MIUB24/#PHA35EAPNONZMTOYBINGFR6XXNMCDPFQ)

[Barry Warsaw\'s call on
this](https://mail.python.org/archives/list/python-dev@python.org/message/XR7IF2OB3S72KBP3PEQ3IKBOERE4FV2I/):

> I don't believe that the way for Python to remain relevant and useful
> for the next 10 years is to cease all language evolution. Who knows
> what the computing landscape will look like in 5 years, let alone 10?
> Something as arbitrary as a 10 year moratorium is (again, IMHO) a
> death sentence for the language.

PEP 387
-------

[PEP 387 \-- Backwards Compatibility
Policy](https://www.python.org/dev/peps/pep-0387/) proposes a process to
make incompatible changes. The main point is the 4th step of the
process:

> See if there\'s any feedback. Users not involved in the original
> discussions may comment now after seeing the warning. Perhaps
> reconsider.

PEP 497
-------

[PEP 497 \-- A standard mechanism for backward
compatibility](https://www.python.org/dev/peps/pep-0497/) proposes
different solutions to provide backward compatibility.

Except for the `__past__` mechanism idea, PEP 497 does not propose
concrete solutions:

> When an incompatible change to core language syntax or semantics is
> being made, Python-dev\'s policy is to prefer and expect that,
> wherever possible, a mechanism for backward compatibility be
> considered and provided for future Python versions after the breaking
> change is adopted by default, in addition to any mechanisms proposed
> for forward compatibility such as new future\_statements.

Examples of incompatible changes
================================

Python 3.8
----------

Examples of Python 3.8 incompatible changes:

-   (During beta phase) `PyCode_New()` required a new parameter: it
    broke all Cython extensions (all projects distributing precompiled
    Cython code). This change has been reverted during the 3.8 beta
    phase and a new `PyCode_NewWithPosOnlyArgs()` function was added
    instead.
-   `types.CodeType` requires an additional mandatory parameter. The
    `CodeType.replace()` function was added to help projects to no
    longer depend on the exact signature of the `CodeType` constructor.
-   C extensions are no longer linked to libpython.
-   `sys.abiflags` changed from `'m'` to an empty string. For example,
    `python3.8m` program is gone.
-   The C structure `PyInterpreterState` was made opaque.
    -   Blender:
        -   <https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6>
        -   <https://developer.blender.org/D6038>
-   XML attribute order:
    [bpo-34160](https://bugs.python.org/issue34160). Broken projects:
    -   [coverage](https://bugs.python.org/issue34160#msg329612)
    -   [docutils](https://sourceforge.net/p/docutils/bugs/359/)
    -   [pcs](https://bugzilla.redhat.com/show_bug.cgi?id=1705475)
    -   [python-glyphsLib](https://bugzilla.redhat.com/show_bug.cgi?id=1705391)

Backward compatibility cannot be added for all these changes. For
example, changes in the C API and in the build system are out of the
scope of this PEP.

See [What's New In Python 3.8: API and Feature
Removals](https://docs.python.org/dev/whatsnew/3.8.html#api-and-feature-removals)
for all changes.

See also the [Porting to Python
3.8](https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8)
section of What's New In Python 3.8.

Python 3.7
----------

Examples of Python 3.7 incompatible changes:

-   `async` and `await` are now reserved keywords.
-   Several undocumented internal imports were removed. One example is
    that `os.errno` is no longer available; use `import errno` directly
    instead. Note that such undocumented internal imports may be removed
    any time without notice, even in micro version releases.
-   Unknown escapes consisting of `'\'` and an ASCII letter in
    replacement templates for `re.sub()` were deprecated in Python 3.5,
    and will now cause an error.
-   The `asyncio.windows_utils.socketpair()` function has been removed:
    it was an alias to `socket.socketpair()`.
-   `asyncio` no longer exports the `selectors` and `_overlapped`
    modules as `asyncio.selectors` and `asyncio._overlapped`. Replace
    `from asyncio import selectors` with `import selectors`.
-   PEP 479 is enabled for all code in Python 3.7, meaning that
    `StopIteration` exceptions raised directly or indirectly in
    coroutines and generators are transformed into `RuntimeError`
    exceptions.
-   `socketserver.ThreadingMixIn.server_close()` now waits until all
    non-daemon threads complete. Set the new `block_on_close` class
    attribute to `False` to get the pre-3.7 behaviour.
-   The `struct.Struct.format` type is now `str` instead of `bytes`.
-   `repr` for `datetime.timedelta` has changed to include the keyword
    arguments in the output.
-   `tracemalloc.Traceback` frames are now sorted from oldest to most
    recent to be more consistent with `traceback`.

Adding backward compatibility for most of these changes would be easy.

See also the [Porting to Python
3.7](https://docs.python.org/dev/whatsnew/3.7.html#porting-to-python-3-7)
section of What's New In Python 3.7.

Micro releases
--------------

Sometimes, incompatible changes are introduced in micro releases
(`micro` in `major.minor.micro`) to fix bugs or security
vulnerabilities. Examples include:

-   Python 3.7.2, `compileall` and `py_compile` module: the
    *invalidation\_mode* parameter\'s default value is updated to
    `None`; the `SOURCE_DATE_EPOCH` environment variable no longer
    overrides the value of the *invalidation\_mode* argument, and
    determines its default value instead.
-   Python 3.7.1, `xml` modules: the SAX parser no longer processes
    general external entities by default to increase security by
    default.
-   Python 3.5.2, `os.urandom()`: on Linux, if the `getrandom()` syscall
    blocks (the urandom entropy pool is not initialized yet), fall back
    on reading `/dev/urandom`.
-   Python 3.5.1, `sys.setrecursionlimit()`: a `RecursionError`
    exception is now raised if the new limit is too low at the current
    recursion depth.
-   Python 3.4.4, `ssl.create_default_context()`: RC4 was dropped from
    the default cipher string.
-   Python 3.4.3, `http.client`: `HTTPSConnection` now performs all the
    necessary certificate and hostname checks by default.
-   Python 3.4.2, `email.message`: `EmailMessage.is_attachment()` is now
    a method instead of a property, for consistency with
    `Message.is_multipart()`.
-   Python 3.4.1, `os.makedirs(name, mode=0o777, exist_ok=False)`:
    Before Python 3.4.1, if *exist\_ok* was `True` and the directory
    existed, `makedirs()` would still raise an error if *mode* did not
    match the mode of the existing directory. Since this behavior was
    impossible to implement safely, it was removed in Python 3.4.1
    ([bpo-21082](https://bugs.python.org/issue21082)).

Examples of changes made in micro releases which are not backward
incompatible:

-   `ssl.OP_NO_TLSv1_3` constant was added to 2.7.15, 3.6.3 and 3.7.0
    for backwards compatibility with OpenSSL 1.0.2.
-   `typing.AsyncContextManager` was added to Python 3.6.2.
-   The `zipfile` module accepts a path-like object since Python 3.6.2.
-   `loop.create_future()` was added to Python 3.5.2 in the `asyncio`
    module.

No backward compatibility code is needed for these kinds of changes.

References
==========

Accepted PEPs:

-   [PEP 5 \-- Guidelines for Language
    Evolution](https://www.python.org/dev/peps/pep-0005/)
-   [PEP 236 \-- Back to the
    \_\_future\_\_](https://www.python.org/dev/peps/pep-0236/)
-   [PEP 411 \-- Provisional packages in the Python standard
    library](https://www.python.org/dev/peps/pep-0411/)
-   [PEP 3002 \-- Procedure for Backwards-Incompatible
    Changes](https://www.python.org/dev/peps/pep-3002/)

Draft PEPs:

-   [PEP 602 \-- Annual Release Cycle for
    Python](https://www.python.org/dev/peps/pep-0602/)
-   [PEP 605 \-- A rolling feature release stream for
    CPython](https://www.python.org/dev/peps/pep-0605/)
-   See also withdrawn [PEP 598 \-- Introducing incremental feature
    releases](https://www.python.org/dev/peps/pep-0598/)

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

#### 

> Local Variables: mode: indented-text indent-tabs-mode: nil
> sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
