PEP: 497 Title: A standard mechanism for backward compatibility Version:
$Revision$ Last-Modified: $Date$ Author: Ed Schofield \<ed at
pythoncharmers.com\> BDFL-Delegate: Brett Cannon (on behalf of the
steering council) Status: Rejected Type: Process Content-Type:
text/x-rst Created: 04-Aug-2015

Rejection Notice
================

The steering council decided that the `__past__` aspect of this proposal
was too complicated for the potential benefit. The other aspect of
stronger requirements for backwards-compatibility should be addressed by
PEP 387.

Scope
=====

This PEP is complementary to PEPs 5, 236, and 387, and shares similar
goals.

This PEP explains the need for an additional compatibility mechanism in
support of PEP 5, "Guidelines for Language Evolution". PEP 236, "Back to
the **future**", introduced a mechanism for forward compatibility in
support of PEP 5 but noted that a new mechanism for backward
compatibility was outside the scope of that PEP. A related PEP (in
progress) introduces such a mechanism for backward compatibility.

PEP 5, "Guidelines for Language Evolution", notes that "This PEP \[PEP
5\] does not replace or preclude other compatibility strategies such as
dynamic loading of backwards-compatible parsers."

Context
=======

From PEP 236: "From time to time, Python makes an incompatible change to
the advertised semantics of core language constructs, or changes their
accidental (implementation-dependent) behavior in some way. While this
is never done capriciously, and is always done with the aim of improving
the language over the long term, over the short term it's contentious
and disrupting. PEP 5, Guidelines for Language Evolution, suggests ways
to ease the pain, and this PEP \[236\] introduces some machinery in
support of that."

Also from PEP 236: "The purpose of future\_statement is to make life
easier for people who keep current with the latest release in a timely
fashion. We don't hate you if you don't, but your problems are much
harder to solve, and somebody with those problems will need to write a
PEP addressing them. future\_statement is aimed at a different
audience."

The current situation
=====================

When an incompatible change to core language syntax or semantics is
being made, Python currently provides the future\_statement mechanism
for providing forward compatibility until the release that enforces the
new syntax or semantics, but provides no corresponding standard
mechanism for providing backward compatibility after this release.

Problem
=======

A consequence of this asymmetry is that, with respect to a breaking
change, the older (pre-breaking) version of the Python interpreter is
more capable than the newer (breaking) version; the older interpreter
can use both code designed prior to the change and newer code, whereas
the newer interpreter is only capable of using code that has been
upgraded to support the changed feature.

As an example, consider the changes to the division operator introduced
in PEP 238 in 2001, soon after PEP 236 introduced the future\_statement
mechanism. PEP 238 outlines a suite of useful forward-compatibility
mechanisms for "true division" in the Python 2.x series but omits to
include any backward-compatibility mechanisms for after "true division"
was first enforced in Python 3.0. Python versions since 3.0 do not
provide a backward compatibility mechanism such as
`from __past__ import division` for code that expects the old "classic
division" semantics, whereas Python versions prior to 3.0 do support
both "classic division" code and also forward compatibility with code
expecting "true division". A further consequence of this is that the
"most compatible" interpreter with respect to the variety of
division-related Python code in the wild is Python 2.7, the version
before the breaking change was first enforced.

Backward compatibility as enabler for "downhill upgrades"
=========================================================

In contrast to this situation, newer versions of application software
such as office suites tend to be more capable than earlier versions with
respect to their support for loading different versions of their data
file formats. The pattern is usually that the newer application versions
can transparently load data from either their newer or their older data
formats, and that the newer version defaults to saving data in the newer
format. Newer application software versions tend to be
backward-compatible by default. Forward compatibility is relatively
rare.

This policy puts the user of the newer application software at an
advantage over the user of the older software, which is usually
incapable of loading data in the newer format. Sometimes it is possible
for a user of a newer software application version to export data in an
older version by choosing this option explicitly. In these cases, the
forward-compatibility this enables may or may not be perfect; some
features may be missing or the results may be otherwise suboptimal.
Upgrading is therefore easy, whereas downgrading is harder.

The emergent behaviour over many users from such a policy of new
attractive features plus backward compatibility features is that a
natural pressure builds up on each individual user to upgrade his or her
own application version, and, the more other users an individual
exchanges data files with, the more acute this pressure becomes.

Proposal - part 1
=================

This PEP makes two specific, related proposals. The first is that:

    PEP 5 be augmented with a 6th step in the section "Steps for
    Introducing Backwards-Incompatible Features" to indicate that, when an
    incompatible change to core language syntax or semantics is being
    made, Python-dev's policy is to prefer and expect that, wherever
    possible, a mechanism for backward compatibility be considered and
    provided for future Python versions after the breaking change is
    adopted by default, in addition to any mechanisms proposed for forward
    compatibility such as new future_statements. Furthermore, PEP 387,
    "Backwards Compatibility Policy" (if accepted) would be
    augmented with the same 6th step.

Example \~\~\~\~\~\~\~

As an example of how this PEP is to be applied, if the latest revision
of the "true division" PEP (238) were proposed today, it would be
considered incomplete. PEP 238 notes the "severe backwards compatibility
issues" raised by the proposal and describes several measures for
forward compatibility in the Abstract and API Changes sections. It also
mentions some backward compatibility ideas raised on c.l.py, including
"Use `from __past__ import division` to use classic division semantics
in a module", but it does not put forward any backward compatibility
plan as part of the proposal.

If this PEP is accepted, it would be expected that a proposal such as
PEP 238, because of its large-scale compatibility implications, would
also be accompanied by a backward compatibility plan that enables users
of future Python versions after the breaking change has come into effect
to re-enable the classic division behaviour easily in their code.

Proposal - part 2
=================

The second proposal is that:

    Python provide a standard backward compatibility mechanism in
    parallel to the ``__future__`` module mechanism for forward
    compatibility.

For reference, this document will refer to this as a "`__past__`"
mechanism hereon, although it need not have all the characteristics of
the `__future__` module and `future_statement` mechanism.

The specific form and implementation of the `__past__` mechanism is the
subject of a separate PEP (in progress). However, this PEP recommends
that this `__past__` mechanism be designed to meet similar criteria to
those outlined in PEP 296 for `__future__`. Specifically:

a.  It should enable individual modules to specify obsolete behaviours
    to re-enable from older Python versions on a module-by-module basis.

b.  It should be flexible enough for both Python 3.6+ and point releases
    of earlier versions to reintroduce backward compatibility with older
    Python syntax or semantics for user modules that invoke the
    `__past__` mechanism.

c.  It should be possible to run older code augmented to invoke
    `__past__` behaviours on older Python versions such as 2.x that have
    no knowledge of the specific `__past__` features invoked, or even
    that the `__past__` mechanism for backward-compatibility exists.

Counter-examples \~\~\~\~\~\~\~\~\~\~\~\~\~\~\~\~

Some implementations of `__past__` mechanisms that would violate these
criteria are:

a.  Import hooks. These would normally fail to work on a
    module-by-module basis; instead they apply recursively to all new
    modules imported from within a module.

b.  A new piece of syntax or new semantics for Python 3.6 that is
    incompatible with prior versions.

c.  A function added in Python 3.6 to a module in the Python standard
    library that exists under the same name in prior Python versions.

Benefits
========

The benefit to Python-dev of adopting this proposal is that future
backward-incompatible changes can be less disruptive if these changes
each have a corresponding `__past__` feature that has been implemented
and can be invoked easily by users of future Python versions. This can
help the language to evolve more quickly and more effectively to correct
for design mistakes.

The benefit to conservative users is obvious: they can add support for
the latest shiny compatibility-breaking Python version to their code
merely by adding a `__past__` incantation (perhaps a single line) to
each module, and that this can be automated. They can then upgrade their
interpreter to the latest version and gain access to the latest shiny
Python features.

The benefit to the community is that, if ten thousand users rely on
package XYZ, and package XYZ can trivially add support for the latest
Python version, those ten thousand users can also upgrade to the latest
Python version quickly, without being held back waiting for package XYZ
to do this.

Questions and answers
=====================

Q1: Does this PEP require that Python keep two possible sets of
semantics for each backward-incompatible feature forever?

A1: Definitely not. Legacy features can still be phased out when
appropriate -- that is, when the majority of the user-base has migrated
to the newer Python version. This PEP merely proposes to shift the
emphasis of the development effort directed at compatibility from 100%
forwards to at least 50% backwards. Backwards compatibility is the more
powerful of the two concepts for allowing a user-base to adopt the
latest Python interpreter version.

Notice that it has been a long time since most users have cared about
backwards compatibility for non-nested scopes, because most users have
moved comfortably past Python 2.1.

Q2: But Python-dev is already overwhelmed and doesn't have the bandwidth
to implement / maintain the additional complexity!

A2: Python-dev can ask the community of developers to step up and
maintain backward compatibility in Python for legacy language features
they care about. When the community stops caring about a particular
obsolete behaviour, Python-dev can stop caring too.

The `__past__` mechanism could possibly be designed to be extensible by
the community, e.g. as a standard but "blessed" PyPI package, to reduce
the load on the core developers.

Q3: Won't backward compatibility features lead to lots of cruft and
bloat and baggage in Python?

A3: Not necessarily. First, proposals for new compatibility-breaking
features in Python could be evaluated partly on the simplicity and
maintainability of the implementation of their associated `__past__`
feature up-front.

Second, some old features are simple to provide backward compatibility
for. Consider the "classic division" behaviour before Python 3.0. The
`python-future` project contains a compatible implementation of classic
division in the function `future.utils.old_div`:

::

    def old_div(a, b):
        """
        Equivalent to ``a / b`` on Python 2 without ``from __future__ import
        division``.
        """
        if isinstance(a, numbers.Integral) and isinstance(b, numbers.Integral):
            return a // b
        else:
            return a / b

Bundling such a function with Python 3.x versions, together with a
simple mechanism to invoke it for every appearance of `a / b` after an
appropriate `__past__` invocation, need not be onerous.

Q4: What about performance? Won't the performance of newer Python
versions suffer under the weight of legacy features?

A4: This can be evaluated on a case-by-case basis. The major potential
concern is that the performance with the new default behaviour does not
suffer unduly because of the presence of the legacy option. The
performance under the influence of the `__past__` invocation is of
secondary importance.

Copyright
=========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8
