PEP: 3121 Title: Extension Module Initialization and Finalization
Version: $Revision$ Last-Modified: $Date$ Author: Martin von Löwis
<martin@v.loewis.de> Status: Accepted Type: Standards Track
Content-Type: text/x-rst Created: 27-Apr-2007 Python-Version: 3.0
Post-History:

Abstract
========

Extension module initialization currently has a few deficiencies. There
is no cleanup for modules, the entry point name might give naming
conflicts, the entry functions don't follow the usual calling
convention, and multiple interpreters are not supported well. This PEP
addresses these issues.

Problems
========

Module Finalization
-------------------

Currently, extension modules are initialized usually once and then
"live" forever. The only exception is when Py\_Finalize() is called:
then the initialization routine is invoked a second time. This is bad
from a resource management point of view: memory and other resources
might get allocated each time initialization is called, but there is no
way to reclaim them. As a result, there is currently no way to
completely release all resources Python has allocated.

Entry point name conflicts
--------------------------

The entry point is currently called init<module>. This might conflict
with other symbols also called init<something>. In particular,
initsocket is known to have conflicted in the past (this specific
problem got resolved as a side effect of renaming the module to
\_socket).

Entry point signature
---------------------

The entry point is currently a procedure (returning void). This deviates
from the usual calling conventions; callers can find out whether there
was an error during initialization only by checking PyErr\_Occurred. The
entry point should return a PyObject\*, which will be the module
created, or NULL in case of an exception.

Multiple Interpreters
---------------------

Currently, extension modules share their state across all interpreters.
This allows for undesirable information leakage across interpreters: one
script could permanently corrupt objects in an extension module,
possibly breaking all scripts in other interpreters.

Specification
=============

The module initialization routines change their signature to::

PyObject \*PyInit\_<modulename>()

The initialization routine will be invoked once per interpreter, when
the module is imported. It should return a new module object each time.

In order to store per-module state in C variables, each module object
will contain a block of memory that is interpreted only by the module.
The amount of memory used for the module is specified at the point of
creation of the module.

In addition to the initialization function, a module may implement a
number of additional callback functions, which are invoked when the
module's tp\_traverse, tp\_clear, and tp\_free functions are invoked,
and when the module is reloaded.

The entire module definition is combined in a struct PyModuleDef::

struct PyModuleDef{ PyModuleDef\_Base m\_base; /\* To be filled out by
the interpreter */ Py\_ssize\_t m\_size; /* Size of per-module data */
PyMethodDef *m\_methods; inquiry m\_reload; traverseproc m\_traverse;
inquiry m\_clear; freefunc m\_free; };

Creation of a module is changed to expect an optional PyModuleDef\*. The
module state will be null-initialized.

Each module method will be passed the module object as the first
parameter. To access the module data, a function::

void\* PyModule\_GetState(PyObject\*);

will be provided. In addition, to lookup a module more efficiently than
going through sys.modules, a function::

PyObject\* PyState\_FindModule(struct PyModuleDef\*);

will be provided. This lookup function will use an index located in the
m\_base field, to find the module by index, not by name.

As all Python objects should be controlled through the Python memory
management, usage of "static" type objects is discouraged, unless the
type object itself has no memory-managed state. To simplify definition
of heap types, a new method::

PyTypeObject\* PyType\_Copy(PyTypeObject\*);

is added.

Example
=======

xxmodule.c would be changed to remove the initxx function, and add the
following code instead::

struct xxstate{ PyObject *ErrorObject; PyObject *Xxo\_Type; };

\#define xxstate(o) ((struct xxstate\*)PyModule\_GetState(o))

static int xx\_traverse(PyObject *m, visitproc v, void *arg) {
Py\_VISIT(xxstate(m)-\>ErrorObject); Py\_VISIT(xxstate(m)-\>Xxo\_Type);
return 0; }

static int xx\_clear(PyObject \*m) {
Py\_CLEAR(xxstate(m)-\>ErrorObject); Py\_CLEAR(xxstate(m)-\>Xxo\_Type);
return 0; }

static struct PyModuleDef xxmodule = { {}, /\* m\_base */ sizeof(struct
xxstate), &xx\_methods, 0, /* m\_reload */ xx\_traverse, xx\_clear, 0,
/* m\_free - not needed, since all is done in m\_clear \*/ }

PyObject* PyInit\_xx() { PyObject *res = PyModule\_New("xx", &xxmodule);
if (!res) return NULL; xxstate(res)-\>ErrorObject =
PyErr\_NewException("xx.error", NULL, NULL); if
(!xxstate(res)-\>ErrorObject) { Py\_DECREF(res); return NULL; }
xxstate(res)-\>XxoType = PyType\_Copy(&Xxo\_Type); if
(!xxstate(res)-\>Xxo\_Type) { Py\_DECREF(res); return NULL; } return
res; }

Discussion
==========

Tim Peters reports in \[1\]\_ that PythonLabs considered such a feature
at one point, and lists the following additional hooks which aren't
currently supported in this PEP:

-   when the module object is deleted from sys.modules

-   when Py\_Finalize is called

-   when Python exits

-   when the Python DLL is unloaded (Windows only)

References
==========

.. \[1\] Tim Peters, reporting earlier conversation about such a feature
https://mail.python.org/pipermail/python-3000/2006-April/000726.html

Copyright
=========

This document has been placed in the public domain.

 .. Local Variables: mode: indented-text indent-tabs-mode: nil
sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
