PEP: 576 Title: Rationalize Built-in function classes Author: Mark
Shannon \<<mark@hotpy.org>\> BDFL-Delegate: Petr Viktorin Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
10-May-2018 Python-Version: 3.8 Post-History: 17-May-2018 23-June-2018
08-July-2018 29-Mar-1028

Abstract
========

Expose the \"FastcallKeywords\" convention used internally by CPython to
third-party code, and make the `inspect` module use duck-typing. In
combination this will allow third-party C extensions and tools like
Cython to create objects that use the same calling conventions as
built-in and Python functions, thus gaining performance parity with
built-in functions like `len` or `print`.

A small improvement in the performance of existing code is expected.

Motivation
==========

Currently third-party module authors face a dilemma when implementing
functions in C. Either they can use one of the pre-existing built-in
function or method classes or implement their own custom class in C. The
first choice causes them to lose the ability to access the internals of
the callable object. The second choice is an additional maintenance
burden and, more importantly, has a significant negative impact on
performance.

This PEP aims to allow authors of third-party C modules, and tools like
to Cython, to utilize the faster calling convention used internally by
CPython for built-in functions and methods, and to do so without a loss
of capabilities relative to a function implemented in Python.

Introspection
-------------

The inspect module will fully support duck-typing when introspecting
callables.

The `inspect.Signature.from_callable()` function computes the signature
of a callable. If an object has a `__signature__` property, then
`inspect.Signature.from_callable()` simply returns that. To further
support duck-typing, if a callable has a `__text_signature__` then the
`__signature__` will be created from that.

This means that 3rd party builtin-functions can implement
`__text_signature__` if sufficient, and the more expensive
`__signature__` if necessary.

Efficient calls to third-party callables
----------------------------------------

Currently the majority of calls are dispatched to `function`s and
`method_descriptor`s in custom code, using the \"FastcallKeywords\"
internal calling convention. This PEP proposes that this calling
convention is implemented via a C function pointer. Third-party
callables which implement this binary interface will have the potential
to be called as fast as a built-in function.

### Continued prohibition of callable classes as base classes

Currently any attempt to use `function`, `method` or `method_descriptor`
as a base class for a new class will fail with a `TypeError`. This
behaviour is desirable as it prevents errors when a subclass overrides
the `__call__` method. If callables could be sub-classed then any call
to a `function` or a `method_descriptor` would need an additional check
that the `__call__` method had not been overridden. By exposing an
additional call mechanism, the potential for errors becomes greater. As
a consequence, any third-party class implementing the addition call
interface will not be usable as a base class.

New classes and changes to existing classes
===========================================

Python visible changes
----------------------

1.  A new built-in class, `builtin_function`, will be added.
2.  `types.BuiltinFunctionType` will refer to `builtin_function` not
    `builtin_function_or_method`.
3.  Instances of the `builtin_function` class will retain the
    `__module__` property of `builtin_function_or_method` and gain the
    `func_module` and `func_globals` properties. The `func_module`
    allows access to the module to which the function belongs. Note that
    this is different from the `__module__` property which merely
    returns the name of the module. The `func_globals` property is
    equivalent to `func_module.__dict__` and is provided to mimic the
    Python function property of the same name.
4.  When binding a `method_descriptor` instance to an instance of its
    owning class, a `bound_method` will be created instead of a
    `builtin_function_or_method`. This means that the
    `method_descriptors` now mimic the behaviour of Python functions
    more closely. In other words, `[].append` becomes a `bound_method`
    instead of a `builtin_function_or_method`.

C API changes
-------------

1.  A new function
    `PyBuiltinFunction_New(PyMethodDef *ml, PyObject *module)` is added
    to create built-in functions.
2.  `PyCFunction_NewEx()` and `PyCFunction_New()` are deprecated and
    will return a `PyBuiltinFunction` if able, otherwise a
    `builtin_function_or_method`.

Retaining backwards compatibility in the C API and ABI
======================================================

The proposed changes are fully backwards and forwards compatible at both
the API and ABI level.

Internal C changes
------------------

Two new flags will be allowed for the `typeobject.tp_flags` field. These
are `Py_TPFLAGS_EXTENDED_CALL` and `Py_TPFLAGS_FUNCTION_DESCRIPTOR`

### Py\_TPFLAGS\_EXTENDED\_CALL

For any built-in class that sets `Py_TPFLAGS_EXTENDED_CALL` The C struct
corresponding to this built-in class must begin with the struct
`PyExtendedCallable` which is defined as follows:

    typedef PyObject *(*extended_call_ptr)(PyObject *callable, PyObject** args,
                       int positional_argcount, PyTupleObject* kwnames);

    typedef struct {
        PyObject_HEAD
        extended_call_ptr ext_call;
    } PyExtendedCallable;

Any class that sets the `Py_TPFLAGS_EXTENDED_CALL` cannot be used as a
base class and a TypeError will be raised if any Python code tries to
use it a base class.

### Py\_TPFLAGS\_FUNCTION\_DESCRIPTOR

If this flag is set for a built-in class `F`, then instances of that
class are expected to behave the same as a Python function when used as
a class attribute. Specifically, this mean that the value of `c.m` where
`C.m` is an instanceof the built-in class `F` (and `c` is an instance of
`C`) must be a bound-method binding `C.m` and `c`. Without this flag, it
would be impossible for custom callables to behave like Python functions
*and* be efficient as Python or built-in functions.

### Changes to existing C structs

The `function`, `method_descriptor` and `method` classes will have their
corresponding structs changed to start with the `PyExtendedCallable`
struct.

Third-party built-in classes using the new extended call interface
------------------------------------------------------------------

To enable call performance on a par with Python functions and built-in
functions, third-party callables should set the
`Py_TPFLAGS_EXTENDED_CALL` bit of `tp_flags` and ensure that the
corresponding C struct starts with the `PyExtendedCallable`. Any
built-in class that has the `Py_TPFLAGS_EXTENDED_CALL` bit set must also
implement the `tp_call` function and make sure its behaviour is
consistent with the `ext_call` function.

### Performance implications of these changes

Adding a function pointer to each callable, rather than each class of
callable, enables the choice of dispatching function (the code to
shuffle arguments about and do error checking) to be made when the
callable object is created rather than when it is called. This should
reduce the number of instructions executed between the call-site in the
interpreter and the execution of the callee.

Alternative Suggestions
=======================

PEP 580 \<<https://www.python.org/dev/peps/pep-0580/>\> is an
alternative approach to solving the same problem as this PEP.

Reference implementation
========================

A draft implementation can be found at
<https://github.com/markshannon/cpython/tree/pep-576-minimal>

Copyright
=========

This document has been placed in the public domain.
