PEP: 384 Title: Defining a Stable ABI Version: $Revision$ Last-Modified:
$Date$ Author: Martin v. Löwis <martin@v.loewis.de> Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 17-May-2009
Python-Version: 3.2 Post-History:

Abstract
========

Currently, each feature release introduces a new name for the Python DLL
on Windows, and may cause incompatibilities for extension modules on
Unix. This PEP proposes to define a stable set of API functions which
are guaranteed to be available for the lifetime of Python 3, and which
will also remain binary-compatible across versions. Extension modules
and applications embedding Python can work with different feature
releases as long as they restrict themselves to this stable ABI.

Rationale
=========

The primary source of ABI incompatibility are changes to the lay-out of
in-memory structures. For example, the way in which string interning
works, or the data type used to represent the size of an object, have
changed during the life of Python 2.x. As a consequence, extension
modules making direct access to fields of strings, lists, or tuples,
would break if their code is loaded into a newer version of the
interpreter without recompilation: offsets of other fields may have
changed, making the extension modules access the wrong data.

In some cases, the incompatibilities only affect internal objects of the
interpreter, such as frame or code objects. For example, the way line
numbers are represented has changed in the 2.x lifetime, as has the way
in which local variables are stored (due to the introduction of
closures). Even though most applications probably never used these
objects, changing them had required to change the PYTHON\_API\_VERSION.

On Linux, changes to the ABI are often not much of a problem: the system
will provide a default Python installation, and many extension modules
are already provided pre-compiled for that version. If additional
modules are needed, or additional Python versions, users can typically
compile them themselves on the system, resulting in modules that use the
right ABI.

On Windows, multiple simultaneous installations of different Python
versions are common, and extension modules are compiled by their
authors, not by end users. To reduce the risk of ABI incompatibilities,
Python currently introduces a new DLL name pythonXY.dll for each feature
release, whether or not ABI incompatibilities actually exist.

With this PEP, it will be possible to reduce the dependency of binary
extension modules on a specific Python feature release, and applications
embedding Python can be made work with different releases.

Specification
=============

The ABI specification falls into two parts: an API specification,
specifying what function (groups) are available for use with the ABI,
and a linkage specification specifying what libraries to link with. The
actual ABI (layout of structures in memory, function calling
conventions) is not specified, but implied by the compiler. As a
recommendation, a specific ABI is recommended for selected platforms.

During evolution of Python, new ABI functions will be added.
Applications using them will then have a requirement on a minimum
version of Python; this PEP provides no mechanism for such applications
to fall back when the Python library is too old.

Terminology
-----------

Applications and extension modules that want to use this ABI are
collectively referred to as "applications" from here on.

Header Files and Preprocessor Definitions
-----------------------------------------

Applications shall only include the header file Python.h (before
including any system headers), or, optionally, include pyconfig.h, and
then Python.h.

During the compilation of applications, the preprocessor macro
Py\_LIMITED\_API must be defined. Doing so will hide all definitions
that are not part of the ABI.

Structures
----------

Only the following structures and structure fields are accessible to
applications:

-   PyObject (ob\_refcnt, ob\_type)
-   PyVarObject (ob\_base, ob\_size)
-   PyMethodDef (ml\_name, ml\_meth, ml\_flags, ml\_doc)
-   PyMemberDef (name, type, offset, flags, doc)
-   PyGetSetDef (name, get, set, doc, closure)
-   PyModuleDefBase (ob\_base, m\_init, m\_index, m\_copy)
-   PyModuleDef (m\_base, m\_name, m\_doc, m\_size, m\_methods,
    m\_traverse, m\_clear, m\_free)
-   PyStructSequence\_Field (name, doc)
-   PyStructSequence\_Desc (name, doc, fields, sequence)
-   PyType\_Slot (see below)
-   PyType\_Spec (see below)

The accessor macros to these fields (Py\_REFCNT, Py\_TYPE, Py\_SIZE) are
also available to applications.

The following types are available, but opaque (i.e. incomplete):

-   PyThreadState
-   PyInterpreterState
-   struct \_frame
-   struct symtable
-   struct \_node
-   PyWeakReference
-   PyLongObject
-   PyTypeObject

Type Objects
------------

The structure of type objects is not available to applications;
declaration of "static" type objects is not possible anymore (for
applications using this ABI). Instead, type objects get created
dynamically. To allow an easy creation of types (in particular, to be
able to fill out function pointers easily), the following structures and
functions are available::

typedef struct{ int slot; /\* slot id, see below */ void *pfunc; /\*
function pointer \*/ } PyType\_Slot;

typedef struct{ const char\* name; int basicsize; int itemsize; unsigned
int flags; PyType\_Slot *slots; /* terminated by slot==0. \*/ }
PyType\_Spec;

PyObject\* PyType\_FromSpec(PyType\_Spec\*);

To specify a slot, a unique slot id must be provided. New Python
versions may introduce new slot ids, but slot ids will never be
recycled. Slots may get deprecated, but continue to be supported
throughout Python 3.x.

The slot ids are named like the field names of the structures that hold
the pointers in Python 3.1, with an added `Py_` prefix (i.e.
Py\_tp\_dealloc instead of just tp\_dealloc):

-   tp\_dealloc, tp\_getattr, tp\_setattr, tp\_repr, tp\_hash, tp\_call,
    tp\_str, tp\_getattro, tp\_setattro, tp\_doc, tp\_traverse,
    tp\_clear, tp\_richcompare, tp\_iter, tp\_iternext, tp\_methods,
    tp\_base, tp\_descr\_get, tp\_descr\_set, tp\_init, tp\_alloc,
    tp\_new, tp\_is\_gc, tp\_bases, tp\_del
-   nb\_add nb\_subtract nb\_multiply nb\_remainder nb\_divmod nb\_power
    nb\_negative nb\_positive nb\_absolute nb\_bool nb\_invert
    nb\_lshift nb\_rshift nb\_and nb\_xor nb\_or nb\_int nb\_float
    nb\_inplace\_add nb\_inplace\_subtract nb\_inplace\_multiply
    nb\_inplace\_remainder nb\_inplace\_power nb\_inplace\_lshift
    nb\_inplace\_rshift nb\_inplace\_and nb\_inplace\_xor
    nb\_inplace\_or nb\_floor\_divide nb\_true\_divide
    nb\_inplace\_floor\_divide nb\_inplace\_true\_divide nb\_index
-   sq\_length sq\_concat sq\_repeat sq\_item sq\_ass\_item sq\_contains
    sq\_inplace\_concat sq\_inplace\_repeat
-   mp\_length mp\_subscript mp\_ass\_subscript

The following fields cannot be set during type definition: - tp\_dict
tp\_mro tp\_cache tp\_subclasses tp\_weaklist tp\_print -
tp\_weaklistoffset tp\_dictoffset

typedefs
--------

In addition to the typedefs for structs listed above, the following
typedefs are available. Their inclusion in the ABI means that the
underlying type must not change on a platform (even though it may differ
across platforms).

-   Py\_uintptr\_t Py\_intptr\_t Py\_ssize\_t
-   unaryfunc binaryfunc ternaryfunc inquiry lenfunc ssizeargfunc
    ssizessizeargfunc ssizeobjargproc ssizessizeobjargproc objobjargproc
    objobjproc visitproc traverseproc destructor getattrfunc
    getattrofunc setattrfunc setattrofunc reprfunc hashfunc richcmpfunc
    getiterfunc iternextfunc descrgetfunc descrsetfunc initproc newfunc
    allocfunc
-   PyCFunction PyCFunctionWithKeywords PyNoArgsFunction
    PyCapsule\_Destructor
-   getter setter
-   PyOS\_sighandler\_t
-   PyGILState\_STATE
-   Py\_UCS4

Most notably, Py\_UNICODE is not available as a typedef, since the same
Python version may use different definitions of it on the same platform
(depending on whether it uses narrow or wide code units). Applications
that need to access the contents of a Unicode string can convert it to
wchar\_t.

Functions and function-like Macros
----------------------------------

By default, all functions are available, unless they are excluded below.
Whether a function is documented or not does not matter.

Function-like macros (in particular, field access macros) remain
available to applications, but get replaced by function calls (unless
their definition only refers to features of the ABI, such as the various
\_Check macros)

ABI function declarations will not change their parameters or return
types. If a change to the signature becomes necessary, a new function
will be introduced. If the new function is source-compatible (e.g. if
just the return type changes), an alias macro may get added to redirect
calls to the new function when the applications is recompiled.

If continued provision of the old function is not possible, it may get
deprecated, then removed, causing applications that use that function to
break.

Excluded Functions
------------------

All functions starting with \_Py are not available to applications.
Also, all functions that expect parameter types that are unavailable to
applications are excluded from the ABI, such as PyAST\_FromNode (which
expects a `node*`).

Functions declared in the following header files are not part of the
ABI:

-   bytes\_methods.h
-   cellobject.h
-   classobject.h
-   code.h
-   compile.h
-   datetime.h
-   dtoa.h
-   frameobject.h
-   funcobject.h
-   genobject.h
-   longintrepr.h
-   parsetok.h
-   pyarena.h
-   pyatomic.h
-   pyctype.h
-   pydebug.h
-   pytime.h
-   symtable.h
-   token.h
-   ucnhash.h

In addition, functions expecting `FILE*` are not part of the ABI, to
avoid depending on a specific version of the Microsoft C runtime DLL on
Windows.

Module and type initializer and finalizer functions are not available
(PyByteArray\_Init, PyOS\_FiniInterrupts and all functions ending in
\_Fini or \_ClearFreeList).

Several functions dealing with interpreter implementation details are
not available:

-   PyInterpreterState\_Head, PyInterpreterState\_Next,
    PyInterpreterState\_ThreadHead, PyThreadState\_Next
-   Py\_SubversionRevision, Py\_SubversionShortBranch

PyStructSequence\_InitType is not available, as it requires the caller
to provide a static type object.

Py\_FatalError will be moved from pydebug.h into some other header file
(e.g. pyerrors.h).

The exact list of functions being available is given in the Windows
module definition file for python3.dll \[1\]\_.

Global Variables
----------------

Global variables representing types and exceptions are available to
applications. In addition, selected global variables referenced in
macros (such as Py\_True and Py\_False) are available.

A complete list of global variable definitions is given in the
python3.def file \[1\]\_; those declared DATA denote variables.

Other Macros
------------

All macros defining symbolic constants are available to applications;
the numeric values will not change.

In addition, the following macros are available:

-   Py\_BEGIN\_ALLOW\_THREADS, Py\_BLOCK\_THREADS, Py\_UNBLOCK\_THREADS,
    Py\_END\_ALLOW\_THREADS

The Buffer Interface
--------------------

The buffer interface (type Py\_buffer, type slots bf\_getbuffer and
bf\_releasebuffer, etc) has been omitted from the ABI, since the
stability of the Py\_buffer structure is not clear at this time.
Inclusion in the ABI can be considered in future releases.

Signature Changes
-----------------

A number of functions currently expect a specific struct, even though
callers typically have PyObject\* available. These have been changed to
expect PyObject\* as the parameter; this will cause warnings in
applications that currently explicitly cast to the parameter type. These
functions are PySlice\_GetIndices, PySlice\_GetIndicesEx,
PyUnicode\_AsWideChar, and PyEval\_EvalCode.

Linkage
-------

On Windows, applications shall link with python3.dll; an import library
python3.lib will be available. This DLL will redirect all of its API
functions through /export linker options to the full interpreter DLL,
i.e. python3y.dll.

On Unix systems, the ABI is typically provided by the python executable
itself. PyModule\_Create is changed to pass `3` as the API version if
the extension module was compiled with Py\_LIMITED\_API; the version
check for the API version will accept either 3 or the current
PYTHON\_API\_VERSION as conforming. If Python is compiled as a shared
library, it is installed as both libpython3.so, and libpython3.y.so;
applications conforming to this PEP should then link to the former
(extension modules can continue to link with no libpython shared object,
but rather rely on runtime linking). The ABI version is symbolically
available as `PYTHON_ABI_VERSION`.

Also on Unix, the PEP 3149 tag abi<PYTHON_ABI_VERSION> is accepted in
file names of extension modules. No checking is performed that files
named in this way are actually restricted to the limited API, and no
support for building such files will be added to distutils due to the
distutils code freeze.

Implementation Strategy
=======================

This PEP will be implemented in a branch \[2\]*, allowing users to check
whether their modules conform to the ABI. To avoid users having to
rewrite their type definitions, a script to convert C source code
containing type definitions will be provided \[3\]*.

References
==========

.. \[1\] "python3 module definition file":
http://svn.python.org/projects/python/branches/pep-0384/PC/python3.def
.. \[2\] "PEP 384 branch":
http://svn.python.org/projects/python/branches/pep-0384/ .. \[3\] "ABI
type conversion script":
http://svn.python.org/projects/python/branches/pep-0384/Tools/scripts/abitype.py

Copyright
=========

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
