PEP: 562 Title: Module \_\_getattr\_\_ and \_\_dir\_\_ Author: Ivan
Levkivskyi \<<levkivskyi@gmail.com>\> Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7
Post-History: 09-Sep-2017 Resolution:
<https://mail.python.org/pipermail/python-dev/2017-December/151033.html>

Abstract
========

It is proposed to support `__getattr__` and `__dir__` function defined
on modules to provide basic customization of module attribute access.

Rationale
=========

It is sometimes convenient to customize or otherwise have control over
access to module attributes. A typical example is managing deprecation
warnings. Typical workarounds are assigning `__class__` of a module
object to a custom subclass of `types.ModuleType` or replacing the
`sys.modules` item with a custom wrapper instance. It would be
convenient to simplify this procedure by recognizing `__getattr__`
defined directly in a module that would act like a normal `__getattr__`
method, except that it will be defined on module *instances*. For
example:

    # lib.py

    from warnings import warn

    deprecated_names = ["old_function", ...]

    def _deprecated_old_function(arg, other):
        ...

    def __getattr__(name):
        if name in deprecated_names:
            warn(f"{name} is deprecated", DeprecationWarning)
            return globals()[f"_deprecated_{name}"]
        raise AttributeError(f"module {__name__} has no attribute {name}")

    # main.py

    from lib import old_function  # Works, but emits the warning

Another widespread use case for `__getattr__` would be lazy submodule
imports. Consider a simple example:

    # lib/__init__.py

    import importlib

    __all__ = ['submod', ...]

    def __getattr__(name):
        if name in __all__:
            return importlib.import_module("." + name, __name__)
        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

    # lib/submod.py

    print("Submodule loaded")
    class HeavyClass:
        ...

    # main.py

    import lib
    lib.submod.HeavyClass  # prints "Submodule loaded"

There is a related proposal PEP 549 that proposes to support instance
properties for a similar functionality. The difference is this PEP
proposes a faster and simpler mechanism, but provides more basic
customization. An additional motivation for this proposal is that PEP
484 already defines the use of module `__getattr__` for this purpose in
Python stub files, see[^1].

In addition, to allow modifying result of a `dir()` call on a module to
show deprecated and other dynamically generated attributes, it is
proposed to support module level `__dir__` function. For example:

    # lib.py

    deprecated_names = ["old_function", ...]
    __all__ = ["new_function_one", "new_function_two", ...]

    def new_function_one(arg, other):
       ...
    def new_function_two(arg, other):
        ...

    def __dir__():
        return sorted(__all__ + deprecated_names)

    # main.py

    import lib

    dir(lib)  # prints ["new_function_one", "new_function_two", "old_function", ...]

Specification
=============

The `__getattr__` function at the module level should accept one
argument which is the name of an attribute and return the computed value
or raise an `AttributeError`:

    def __getattr__(name: str) -> Any: ...

If an attribute is not found on a module object through the normal
lookup (i.e. `object.__getattribute__`), then `__getattr__` is searched
in the module `__dict__` before raising an `AttributeError`. If found,
it is called with the attribute name and the result is returned. Looking
up a name as a module global will bypass module `__getattr__`. This is
intentional, otherwise calling `__getattr__` for builtins will
significantly harm performance.

The `__dir__` function should accept no arguments, and return a list of
strings that represents the names accessible on module:

    def __dir__() -> List[str]: ...

If present, this function overrides the standard `dir()` search on a
module.

The reference implementation for this PEP can be found in[^2].

Backwards compatibility and impact on performance
=================================================

This PEP may break code that uses module level (global) names
`__getattr__` and `__dir__`. (But the language reference explicitly
reserves *all* undocumented dunder names, and allows \"breakage without
warning\"; see[^3].) The performance implications of this PEP are
minimal, since `__getattr__` is called only for missing attributes.

Some tools that perform module attributes discovery might not expect
`__getattr__`. This problem is not new however, since it is already
possible to replace a module with a module subclass with overridden
`__getattr__` and `__dir__`, but with this PEP such problems can occur
more often.

Discussion
==========

Note that the use of module `__getattr__` requires care to keep the
referred objects pickleable. For example, the `__name__` attribute of a
function should correspond to the name with which it is accessible via
`__getattr__`:

    def keep_pickleable(func):
        func.__name__ = func.__name__.replace('_deprecated_', '')
        func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
        return func

    @keep_pickleable
    def _deprecated_old_function(arg, other):
        ...

One should be also careful to avoid recursion as one would do with a
class level `__getattr__`.

To use a module global with triggering `__getattr__` (for example if one
wants to use a lazy loaded submodule) one can access it as:

    sys.modules[__name__].some_global

or as:

    from . import some_global

Note that the latter sets the module attribute, thus `__getattr__` will
be called only once.

References
==========

Copyright
=========

This document has been placed in the public domain.


-

> Local Variables: mode: indented-text indent-tabs-mode: nil
> sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[^1]: PEP 484 section about `__getattr__` in stub files
    (<https://www.python.org/dev/peps/pep-0484/#stub-files>)

[^2]: The reference implementation
    (<https://github.com/ilevkivskyi/cpython/pull/3/files>)

[^3]: Reserved classes of identifiers
    (<https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers>)
