PEP: 422 Title: Simpler customisation of class creation Version:
$Revision$ Last-Modified: $Date$ Author: Nick Coghlan
<ncoghlan@gmail.com>, Daniel Urban <urban.dani+py@gmail.com> Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
5-Jun-2012 Python-Version: 3.5 Post-History: 5-Jun-2012, 10-Feb-2013

Abstract
========

Currently, customising class creation requires the use of a custom
metaclass. This custom metaclass then persists for the entire lifecycle
of the class, creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new `namespace` parameter in the class header, and a
new `__autodecorate__` hook in the class body.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.

PEP Withdrawal
==============

This proposal has been withdrawn in favour of Martin Teichmann's
proposal in PEP 487, which achieves the same goals through a simpler,
easier to use `__init_subclass__` hook that simply isn't invoked for the
base class that defines the hook.

Background
==========

For an already created class `cls`, the term "metaclass" has a clear
meaning: it is the value of `type(cls)`.

*During* class creation, it has another meaning: it is also used to
refer to the metaclass hint that may be provided as part of the class
definition. While in many cases these two meanings end up referring to
one and the same object, there are two situations where that is not the
case:

-   If the metaclass hint refers to an instance of `type`, then it is
    considered as a candidate metaclass along with the metaclasses of
    all of the parents of the class being defined. If a more appropriate
    metaclass is found amongst the candidates, then it will be used
    instead of the one given in the metaclass hint.
-   Otherwise, an explicit metaclass hint is assumed to be a factory
    function and is called directly to create the class object. In this
    case, the final metaclass will be determined by the factory function
    definition. In the typical case (where the factory functions just
    calls `type`, or, in Python 3.3 or later, `types.new_class`) the
    actual metaclass is then determined based on the parent classes.

It is notable that only the actual metaclass is inherited - a factory
function used as a metaclass hook sees only the class currently being
defined, and is not invoked for any subclasses.

In Python 3, the metaclass hint is provided using the `metaclass=Meta`
keyword syntax in the class header. This allows the `__prepare__` method
on the metaclass to be used to create the `locals()` namespace used
during execution of the class body (for example, specifying the use of
`collections.OrderedDict` instead of a regular `dict`).

In Python 2, there was no `__prepare__` method (that API was added for
Python 3 by PEP 3115). Instead, a class body could set the
`__metaclass__` attribute, and the class creation process would extract
that value from the class namespace to use as the metaclass hint. There
is `published code`\_ that makes use of this feature.

Another new feature in Python 3 is the zero-argument form of the
`super()` builtin, introduced by PEP 3135. This feature uses an implicit
`__class__` reference to the class being defined to replace the "by
name" references required in Python 2. Just as code invoked during
execution of a Python 2 metaclass could not call methods that referenced
the class by name (as the name had not yet been bound in the containing
scope), similarly, Python 3 metaclasses cannot call methods that rely on
the implicit `__class__` reference (as it is not populated until after
the metaclass has returned control to the class creation machinery).

Finally, when a class uses a custom metaclass, it can pose additional
challenges to the use of multiple inheritance, as a new class cannot
inherit from parent classes with unrelated metaclasses. This means that
it is impossible to add a metaclass to an already published class: such
an addition is a backwards incompatible change due to the risk of
metaclass conflicts.

Proposal
========

This PEP proposes that a new mechanism to customise class creation be
added to Python 3.4 that meets the following criteria:

1.  Integrates nicely with class inheritance structures (including
    mixins and multiple inheritance)
2.  Integrates nicely with the implicit `__class__` reference and
    zero-argument `super()` syntax introduced by PEP 3135
3.  Can be added to an existing base class without a significant risk of
    introducing backwards compatibility problems
4.  Restores the ability for class namespaces to have some influence on
    the class creation process (above and beyond populating the
    namespace itself), but potentially without the full flexibility of
    the Python 2 style `__metaclass__` hook

One mechanism that can achieve this goal is to add a new implicit class
decoration hook, modelled directly on the existing explicit class
decorators, but defined in the class body or in a parent class, rather
than being part of the class definition header.

Specifically, it is proposed that class definitions be able to provide a
class initialisation hook as follows::

class Example: def **autodecorate**(cls): \# This is invoked after the
class is created, but before any \# explicit decorators are called \#
The usual super() mechanisms are used to correctly support \# multiple
inheritance. The class decorator style signature helps \# ensure that
invoking the parent class is as simple as possible. cls =
super().\_\_autodecorate\_\_() return cls

To simplify the cooperative multiple inheritance case, `object` will
gain a default implementation of the hook that returns the class
unmodified::

class object: def **autodecorate**(cls): return cls

If a metaclass wishes to block implicit class decoration for some
reason, it must arrange for `cls.__autodecorate__` to trigger
`AttributeError`.

If present on the created object, this new hook will be called by the
class creation machinery *after* the `__class__` reference has been
initialised. For `types.new_class()`, it will be called as the last step
before returning the created class object. `__autodecorate__` is
implicitly converted to a class method when the class is created (prior
to the hook being invoked).

Note, that when `__autodecorate__` is called, the name of the class is
not yet bound to the new class object. As a consequence, the two
argument form of `super()` cannot be used to call methods (e.g.,
`super(Example, cls)` wouldn't work in the example above). However, the
zero argument form of `super()` works as expected, since the `__class__`
reference is already initialised.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`*, and a
similar mechanism has long been supported by `Zope's ExtensionClass`*),
but the situation has changed sufficiently in recent years that the idea
is worth reconsidering for inclusion as a native language feature.

In addition, the introduction of the metaclass `__prepare__` method in
PEP 3115 allows a further enhancement that was not possible in Python 2:
this PEP also proposes that `type.__prepare__` be updated to accept a
factory function as a `namespace` keyword-only argument. If present, the
value provided as the `namespace` argument will be called without
arguments to create the result of `type.__prepare__` instead of using a
freshly created dictionary instance. For example, the following will use
an ordered dictionary as the class namespace::

class OrderedExample(namespace=collections.OrderedDict): def
**autodecorate**(cls): \# cls.\_\_dict\_\_ is still a read-only proxy to
the class namespace, \# but the underlying storage is an OrderedDict
instance

.. note::

    This PEP, along with the existing ability to use  __prepare__ to share a
    single namespace amongst multiple class objects, highlights a possible
    issue with the attribute lookup caching: when the underlying mapping is
    updated by other means, the attribute lookup cache is not invalidated
    correctly (this is a key part of the reason class ``__dict__`` attributes
    produce a read-only view of the underlying storage).

    Since the optimisation provided by that cache is highly desirable,
    the use of a preexisting namespace as the class namespace may need to
    be declared as officially unsupported (since the observed behaviour is
    rather strange when the caches get out of sync).

Key Benefits
============

Easier use of custom namespaces for a class
-------------------------------------------

Currently, to use a different type (such as `collections.OrderedDict`)
for a class namespace, or to use a pre-populated namespace, it is
necessary to write and use a custom metaclass. With this PEP, using a
custom namespace becomes as simple as specifying an appropriate factory
function in the class header.

Easier inheritance of definition time behaviour
-----------------------------------------------

Understanding Python's metaclasses requires a deep understanding of the
type system and the class construction process. This is legitimately
seen as challenging, due to the need to keep multiple moving parts (the
code, the metaclass hint, the actual metaclass, the class object,
instances of the class object) clearly distinct in your mind. Even when
you know the rules, it's still easy to make a mistake if you're not
being extremely careful. An earlier version of this PEP actually
included such a mistake: it stated "subclass of type" for a constraint
that is actually "instance of type".

Understanding the proposed implicit class decoration hook only requires
understanding decorators and ordinary method inheritance, which isn't
quite as daunting a task. The new hook provides a more gradual path
towards understanding all of the phases involved in the class definition
process.

Reduced chance of metaclass conflicts
-------------------------------------

One of the big issues that makes library authors reluctant to use
metaclasses (even when they would be appropriate) is the risk of
metaclass conflicts. These occur whenever two unrelated metaclasses are
used by the desired parents of a class definition. This risk also makes
it very difficult to *add* a metaclass to a class that has previously
been published without one.

By contrast, adding an `__autodecorate__` method to an existing type
poses a similar level of risk to adding an `__init__` method:
technically, there is a risk of breaking poorly implemented subclasses,
but when that occurs, it is recognised as a bug in the subclass rather
than the library author breaching backwards compatibility guarantees. In
fact, due to the constrained signature of `__autodecorate__`, the risk
in this case is actually even lower than in the case of `__init__`.

Integrates cleanly with \PEP 3135
---------------------------------

Unlike code that runs as part of the metaclass, code that runs as part
of the new hook will be able to freely invoke class methods that rely on
the implicit `__class__` reference introduced by PEP 3135, including
methods that use the zero argument form of `super()`.

Replaces many use cases for dynamic setting of `__metaclass__`
--------------------------------------------------------------

For use cases that don't involve completely replacing the defined class,
Python 2 code that dynamically set `__metaclass__` can now dynamically
set `__autodecorate__` instead. For more advanced use cases,
introduction of an explicit metaclass (possibly made available as a
required base class) will still be necessary in order to support Python
3.

Design Notes
============

Determining if the class being decorated is the base class
----------------------------------------------------------

In the body of an `__autodecorate__` method, as in any other class
method, `__class__` will be bound to the class declaring the method,
while the value passed in may be a subclass.

This makes it relatively straightforward to skip processing the base
class if necessary::

class Example: def **autodecorate**(cls): cls =
super().\_\_autodecorate\_\_() \# Don't process the base class if cls is
**class**: return \# Process subclasses here ...

Replacing a class with a different kind of object
-------------------------------------------------

As an implicit decorator, `__autodecorate__` is able to relatively
easily replace the defined class with a different kind of object.
Technically custom metaclasses and even `__new__` methods can already do
this implicitly, but the decorator model makes such code much easier to
understand and implement.

::

class BuildDict: def **autodecorate**(cls): cls =
super().\_\_autodecorate\_\_() \# Don't process the base class if cls is
**class**: return \# Convert subclasses to ordinary dictionaries return
cls.\_\_dict\_\_.copy()

It's not clear why anyone would ever do this implicitly based on
inheritance rather than just using an explicit decorator, but the
possibility seems worth noting.

Open Questions
==============

Is the `namespace` concept worth the extra complexity?
------------------------------------------------------

Unlike the new `__autodecorate__` hook the proposed `namespace` keyword
argument is not automatically inherited by subclasses. Given the way
this proposal is currently written , the only way to get a special
namespace used consistently in subclasses is still to write a custom
metaclass with a suitable `__prepare__` implementation.

Changing the custom namespace factory to also be inherited would
significantly increase the complexity of this proposal, and introduce a
number of the same potential base class conflict issues as arise with
the use of custom metaclasses.

Eric Snow has put forward a
`separate proposal <https://mail.python.org/pipermail/python-dev/2013-June/127103.html>`\_\_
to instead make the execution namespace for class bodies an ordered
dictionary by default, and capture the class attribute definition order
for future reference as an attribute (e.g. `__definition_order__`) on
the class object.

Eric's suggested approach may be a better choice for a new default
behaviour for type that combines well with the proposed
`__autodecorate__` hook, leaving the more complex configurable namespace
factory idea to a custom metaclass like the one shown below.

New Ways of Using Classes
=========================

The new `namespace` keyword in the class header enables a number of
interesting options for controlling the way a class is initialised,
including some aspects of the object models of both Javascript and Ruby.

All of the examples below are actually possible today through the use of
a custom metaclass::

    class CustomNamespace(type):
        @classmethod
        def __prepare__(meta, name, bases, *, namespace=None, **kwds):
            parent_namespace = super().__prepare__(name, bases, **kwds)
            return namespace() if namespace is not None else parent_namespace
        def __new__(meta, name, bases, ns, *, namespace=None, **kwds):
            return super().__new__(meta, name, bases, ns, **kwds)
        def __init__(cls, name, bases, ns, *, namespace=None, **kwds):
            return super().__init__(name, bases, ns, **kwds)

The advantage of implementing the new keyword directly in
`type.__prepare__` is that the *only* persistent effect is then the
change in the underlying storage of the class attributes. The metaclass
of the class remains unchanged, eliminating many of the drawbacks
typically associated with these kinds of customisations.

Order preserving classes
------------------------

::

    class OrderedClass(namespace=collections.OrderedDict):
        a = 1
        b = 2
        c = 3

Prepopulated namespaces
-----------------------

::

    seed_data = dict(a=1, b=2, c=3)
    class PrepopulatedClass(namespace=seed_data.copy):
        pass

Cloning a prototype class
-------------------------

::

    class NewClass(namespace=Prototype.__dict__.copy):
        pass

Extending a class
-----------------

.. note:: Just because the PEP makes it *possible* to do this relatively
cleanly doesn't mean anyone *should* do this!

::

    from collections import MutableMapping

    # The MutableMapping + dict combination should give something that
    # generally behaves correctly as a mapping, while still being accepted
    # as a class namespace
    class ClassNamespace(MutableMapping, dict):
        def __init__(self, cls):
            self._cls = cls
        def __len__(self):
            return len(dir(self._cls))
        def __iter__(self):
            for attr in dir(self._cls):
                yield attr
        def __contains__(self, attr):
            return hasattr(self._cls, attr)
        def __getitem__(self, attr):
            return getattr(self._cls, attr)
        def __setitem__(self, attr, value):
            setattr(self._cls, attr, value)
        def __delitem__(self, attr):
            delattr(self._cls, attr)

    def extend(cls):
        return lambda: ClassNamespace(cls)

    class Example:
        pass

    class ExtendedExample(namespace=extend(Example)):
        a = 1
        b = 2
        c = 3

    >>> Example.a, Example.b, Example.c
    (1, 2, 3)

Rejected Design Options
=======================

Calling `__autodecorate__` from `type.__init__`
-----------------------------------------------

Calling the new hook automatically from `type.__init__`, would achieve
most of the goals of this PEP. However, using that approach would mean
that `__autodecorate__` implementations would be unable to call any
methods that relied on the `__class__` reference (or used the
zero-argument form of `super()`), and could not make use of those
features themselves.

The current design instead ensures that the implicit decorator hook is
able to do anything an explicit decorator can do by running it after the
initial class creation is already complete.

Calling the automatic decoration hook `__init_class__`
------------------------------------------------------

Earlier versions of the PEP used the name `__init_class__` for the name
of the new hook. There were three significant problems with this name:

-   it was hard to remember if the correct spelling was `__init_class__`
    or `__class_init__`
-   the use of "init" in the name suggested the signature should match
    that of `type.__init__`, which is not the case
-   the use of "init" in the name suggested the method would be run as
    part of initial class object creation, which is not the case

The new name `__autodecorate__` was chosen to make it clear that the new
initialisation hook is most usefully thought of as an implicitly invoked
class decorator, rather than as being like an `__init__` method.

Requiring an explicit decorator on `__autodecorate__`
-----------------------------------------------------

Originally, this PEP required the explicit use of `@classmethod` on the
`__autodecorate__` decorator. It was made implicit since there's no
sensible interpretation for leaving it out, and that case would need to
be detected anyway in order to give a useful error message.

This decision was reinforced after noticing that the user experience of
defining `__prepare__` and forgetting the `@classmethod` method
decorator is singularly incomprehensible (particularly since PEP 3115
documents it as an ordinary method, and the current documentation
doesn't explicitly say anything one way or the other).

Making `__autodecorate__` implicitly static, like `__new__`
-----------------------------------------------------------

While it accepts the class to be instantiated as the first argument,
`__new__` is actually implicitly treated as a static method rather than
as a class method. This allows it to be readily extracted from its
defining class and called directly on a subclass, rather than being
coupled to the class object it is retrieved from.

Such behaviour initially appears to be potentially useful for the new
`__autodecorate__` hook, as it would allow `__autodecorate__` methods to
readily be used as explicit decorators on other classes.

However, that apparent support would be an illusion as it would only
work correctly if invoked on a subclass, in which case the method can
just as readily be retrieved from the subclass and called that way.
Unlike `__new__`, there's no issue with potentially changing method
signatures at different points in the inheritance chain.

Passing in the namespace directly rather than a factory function
----------------------------------------------------------------

At one point, this PEP proposed that the class namespace be passed
directly as a keyword argument, rather than passing a factory function.
However, this encourages an unsupported behaviour (that is, passing the
same namespace to multiple classes, or retaining direct write access to
a mapping used as a class namespace), so the API was switched to the
factory function version.

Reference Implementation
========================

A reference implementation for `__autodecorate__` has been posted to the
`issue tracker`\_. It uses the original `__init_class__` naming. does
not yet allow the implicit decorator to replace the class with a
different object and does not implement the suggested `namespace`
parameter for `type.__prepare__`.

TODO
====

-   address the 5 points in
    https://mail.python.org/pipermail/python-dev/2013-February/123970.html

References
==========

.. \_published code:
https://mail.python.org/pipermail/python-dev/2012-June/119878.html

.. \_more than 10 years ago:
https://mail.python.org/pipermail/python-dev/2001-November/018651.html

.. \_Zope's ExtensionClass:
http://docs.zope.org/zope\_secrets/extensionclass.html

.. \_issue tracker: http://bugs.python.org/issue17044

Copyright
=========

This document has been placed in the public domain.

.. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
