PEP: 561 Title: Distributing and Packaging Type Information Author:
Ethan Smith \<<ethan@ethanhs.me>\> Status: Accepted Type: Standards
Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7
Post-History: 10-Sep-2017, 12-Sep-2017, 06-Oct-2017, 26-Oct-2017,
12-Apr-2018

Abstract
========

PEP 484 introduced type hinting to Python, with goals of making typing
gradual and easy to adopt. Currently, typing information must be
distributed manually. This PEP provides a standardized means to leverage
existing tooling to package and distribute type information with minimal
work and an ordering for type checkers to resolve modules and collect
this information for type checking.

Rationale
=========

Currently, package authors wish to distribute code that has inline type
information. Additionally, maintainers would like to distribute stub
files to keep Python 2 compatibility while using newer annotation
syntax. However, there is no standard method to distribute packages with
type information. Also, if one wished to ship stub files privately the
only method available would be via setting `MYPYPATH` or the equivalent
to manually point to stubs. If the package can be released publicly, it
can be added to typeshed[^1]. However, this does not scale and becomes a
burden on the maintainers of typeshed. In addition, it ties bug fixes in
stubs to releases of the tool using typeshed.

PEP 484 has a brief section on distributing typing information. In this
section[^2] the PEP recommends using `shared/typehints/pythonX.Y/` for
shipping stub files. However, manually adding a path to stub files for
each third party library does not scale. The simplest approach people
have taken is to add `site-packages` to their `MYPYPATH`, but this
causes type checkers to fail on packages that are highly dynamic (e.g.
sqlalchemy and Django).

Definition of Terms
===================

The definition of \"MAY\", \"MUST\", and \"SHOULD\", and \"SHOULD NOT\"
are to be interpreted as described in RFC 2119.

\"inline\" - the types are part of the runtime code using PEP 526 and
PEP 3107 syntax (the filename ends in `.py`).

\"stubs\" - files containing only type information, empty of runtime
code (the filename ends in `.pyi`).

\"Distributions\" are the packaged files which are used to publish and
distribute a release.[^3]

\"Module\" a file containing Python runtime code or stubbed type
information.

\"Package\" a directory or directories that namespace Python modules.
(Note the distinction between packages and distributions. While most
distributions are named after the one package they install, some
distributions install multiple packages.)

Specification
=============

There are several motivations and methods of supporting typing in a
package. This PEP recognizes three types of packages that users of
typing wish to create:

1.  The package maintainer would like to add type information inline.
2.  The package maintainer would like to add type information via stubs.
3.  A third party or package maintainer would like to share stub files
    for a package, but the maintainer does not want to include them in
    the source of the package.

This PEP aims to support all three scenarios and make them simple to add
to packaging and deployment.

The two major parts of this specification are the packaging
specifications and the resolution order for resolving module type
information. The type checking spec is meant to replace the
`shared/typehints/pythonX.Y/` spec of PEP 484[^4].

New third party stub libraries SHOULD distribute stubs via the third
party packaging methods proposed in this PEP in place of being added to
typeshed. Typeshed will remain in use, but if maintainers are found,
third party stubs in typeshed MAY be split into their own package.

Packaging Type Information
--------------------------

In order to make packaging and distributing type information as simple
and easy as possible, packaging and distribution is done through
existing frameworks.

Package maintainers who wish to support type checking of their code MUST
add a marker file named `py.typed` to their package supporting typing.
This marker applies recursively: if a top-level package includes it, all
its sub-packages MUST support type checking as well. To have this file
installed with the package, maintainers can use existing packaging
options such as `package_data` in distutils, shown below.

Distutils option example:

    setup(
        ...,
        package_data = {
            'foopkg': ['py.typed'],
        },
        ...,
        )

For namespace packages (see PEP 420), the `py.typed` file should be in
the submodules of the namespace, to avoid conflicts and for clarity.

This PEP does not support distributing typing information as part of
module-only distributions. The code should be refactored into a
package-based distribution and indicate that the package supports typing
as described above.

### Stub-only Packages

For package maintainers wishing to ship stub files containing all of
their type information, it is preferred that the `*.pyi` stubs are
alongside the corresponding `*.py` files. However, the stubs can also be
put in a separate package and distributed separately. Third parties can
also find this method useful if they wish to distribute stub files. The
name of the stub package MUST follow the scheme `foopkg-stubs` for type
stubs for the package named `foopkg`. Note that for stub-only packages
adding a `py.typed` marker is not needed since the name `*-stubs` is
enough to indicate it is a source of typing information.

Third parties seeking to distribute stub files are encouraged to contact
the maintainer of the package about distribution alongside the package.
If the maintainer does not wish to maintain or package stub files or
type information inline, then a third party stub-only package can be
created.

In addition, stub-only distributions SHOULD indicate which version(s) of
the runtime package are supported by indicating the runtime
distribution\'s version(s) through normal dependency data. For example,
the stub package `flyingcircus-stubs` can indicate the versions of the
runtime `flyingcircus` distribution it supports through
`install_requires` in distutils-based tools, or the equivalent in other
packaging tools. Note that in pip 9.0, if you update
`flyingcircus-stubs`, it will update `flyingcircus`. In pip 9.0, you can
use the `--upgrade-strategy=only-if-needed` flag. In pip 10.0 this is
the default behavior.

Type Checker Module Resolution Order
------------------------------------

The following is the order in which type checkers supporting this PEP
SHOULD resolve modules containing type information:

1.  Stubs or Python source manually put in the beginning of the path.
    Type checkers SHOULD provide this to allow the user complete control
    of which stubs to use, and to patch broken stubs/inline types from
    packages. In mypy the `$MYPYPATH` environment variable can be used
    for this.
2.  User code - the files the type checker is running on.
3.  Stub packages - these packages SHOULD supersede any installed inline
    package. They can be found at `foopkg-stubs` for package `foopkg`.
4.  Inline packages - if there is nothing overriding the installed
    package, *and* it opts into type checking, inline types SHOULD be
    used.
5.  Typeshed (if used) - Provides the stdlib types and several third
    party libraries.

Type checkers that check a different Python version than the version
they run on MUST find the type information in the
`site-packages`/`dist-packages` of that Python version. This can be
queried e.g.
`pythonX.Y -c 'import site; print(site.getsitepackages())'`. It is also
recommended that the type checker allow for the user to point to a
particular Python binary, in case it is not in the path.

Partial Stub Packages
---------------------

Many stub packages will only have part of the type interface for
libraries completed, especially initially. For the benefit of type
checking and code editors, packages can be \"partial\". This means
modules not found in the stub package SHOULD be searched for in parts
four and five of the module resolution order above, namely inline
packages and typeshed.

Type checkers should merge the stub package and runtime package or
typeshed directories. This can be thought of as the functional
equivalent of copying the stub package into the same directory as the
corresponding runtime package or typeshed folder and type checking the
combined directory structure. Thus type checkers MUST maintain the
normal resolution order of checking `*.pyi` before `*.py` files.

If a stub package is partial it MUST include `partial\n` in a top level
`py.typed` file.

Implementation
==============

The proposed scheme of indicating support for typing is completely
backwards compatible, and requires no modification to package tooling. A
sample package with inline types is available
[\[typed\_package\]](#typed_package){.citation}, as well as a
[\[stub\_package\]](#stub_package){.citation}. A sample package checker
[\[pkg\_checker\]](#pkg_checker){.citation} which reads the metadata of
installed packages and reports on their status as either not typed,
inline typed, or a stub package.

The mypy type checker has an implementation of PEP 561 searching which
can be read about in the mypy docs[^5].

[\[numpy-stubs\]](#numpy-stubs){.citation} is an example of a real
stub-only package for the numpy distribution.

Acknowledgements
================

This PEP would not have been possible without the ideas, feedback, and
support of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F
Moisset, Andrey Vlasovskikh, Nathaniel Smith, and Guido van Rossum.

Version History
===============

-   2018-07-09

    > -   Add links to sample stub-only packages

-   2018-06-19

    > -   Partial stub packages can look at typeshed as well as runtime
    >     packages

-   2018-05-15

    > -   Add partial stub package spec.

-   2018-04-09

    > -   Add reference to mypy implementation
    > -   Clarify stub package priority.

-   2018-02-02

    > -   Change stub-only package suffix to be -stubs not \_stubs.
    > -   Note that py.typed is not needed for stub-only packages.
    > -   Add note about pip and upgrading stub packages.

-   2017-11-12

    > -   Rewritten to use existing tooling only
    > -   No need to indicate kind of type information in metadata
    > -   Name of marker file changed from `.typeinfo` to `py.typed`

-   2017-11-10

    > -   Specification re-written to use package metadata instead of
    >     distribution metadata.
    > -   Removed stub-only packages and merged into third party
    >     packages spec.
    > -   Removed suggestion for typecheckers to consider checking
    >     runtime versions
    > -   Implementations updated to reflect PEP changes.

-   2017-10-26

    > -   Added implementation references.
    > -   Added acknowledgements and version history.

-   2017-10-06

    > -   Rewritten to use .distinfo/METADATA over a distutils specific
    >     command.
    > -   Clarify versioning of third party stub packages.

-   2017-09-11

    > -   Added information about current solutions and typeshed.
    > -   Clarify rationale.

References
==========

Copyright
=========

This document has been placed in the public domain.

#### 

> Local Variables: mode: indented-text indent-tabs-mode: nil
> sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

::: {#citations}

[numpy-stubs]{#numpy-stubs .citation-label}

:   Stubs for numpy (<https://github.com/numpy/numpy-stubs>)

[pkg\_checker]{#pkg_checker .citation-label}

:   Sample package checker (<https://github.com/ethanhs/check_typedpkg>)

[stub\_package]{#stub_package .citation-label}

:   A stub-only package (<https://github.com/ethanhs/stub-package>)

[typed\_package]{#typed_package .citation-label}

:   Sample typed package
    (<https://github.com/ethanhs/sample-typed-package>)
:::

[^1]: Typeshed (<https://github.com/python/typeshed>)

[^2]: PEP 484, Storing and Distributing Stub Files
    (<https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files>)

[^3]: PEP 426 definitions (<https://www.python.org/dev/peps/pep-0426/>)

[^4]: PEP 484, Storing and Distributing Stub Files
    (<https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files>)

[^5]: Example implementation in a type checker
    (<https://mypy.readthedocs.io/en/latest/installed_packages.html>)
