PySpark Stubs
=============

|Build Status| |PyPI version| |Conda Forge version|

A collection of the Apache Spark `stub
files <https://www.python.org/dev/peps/pep-0484/#stub-files>`__. These
files were generated by
`stubgen <https://github.com/python/mypy/blob/master/mypy/stubgen.py>`__
and manually edited to include accurate type hints.

Tests and configuration files have been originally contributed to the
`Typeshed project <https://github.com/python/typeshed/>`__. Please refer
to its `contributors
list <https://github.com/python/typeshed/graphs/contributors>`__ and
`license <https://github.com/python/typeshed/blob/master/LICENSE>`__ for
details.

Motivation
----------

-  Static error detection (see
   `SPARK-20631 <https://issues.apache.org/jira/browse/SPARK-20631>`__)

   |SPARK-20631|

-  Improved completion for chained method calls.

   |Syntax completion|

Installation and usage
----------------------

Please note that the guidelines for distribution of type information is
still work in progress (`PEP 561 - Distributing and Packaging Type
Information <https://www.python.org/dev/peps/pep-0561/>`__). Currently
installation script overlays existing Spark installations (``pyi`` stub
files are copied next to their ``py`` counterparts in the PySpark
installation directory). If this approach is not acceptable you can add stub
files to the search path manually.

According to `PEP
484 <https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files>`__:

    Third-party stub packages can use any location for stub storage.
    Type checkers should search for them using PYTHONPATH.

Moreover:

    Third-party stub packages can use any location for stub storage.
    Type checkers should search for them using PYTHONPATH. A default
    fallback directory that is always checked is
    shared/typehints/python3.5/ (or 3.6, etc.)

Please check usage before proceeding.

The package is available on PYPI:

.. code:: bash

    pip install pyspark-stubs

and conda-forge:

.. code:: bash

    conda install -c conda-forge pyspark-stubs

Depending on your environment you might also need a type checker, like `Mypy <https://github.com/python/mypy>`__
or `Pytype <https://github.com/google/pytype/>`__.

- `Atom <https://atom.io/>`__ - Requires `atom-mypy <https://atom.io/packages/atom-mypy>`__ or equivalent.
- `Jupyter Notebooks <https://jupyter.org/>`__ - `It is possible <https://web.archive.org/web/20190126155957/http://journalpanic.com/post/spice-up-thy-jupyter-notebooks-with-mypy/>`__ to use magics to type check directly in the notebook.
- `PyCharm <https://www.jetbrains.com/pycharm/>`__ - Works out-of-the-box, with excellent code completion, though as of today (PyCharm 2018.2.4) built-in type checker is somewhat limited compared to MyPy and `mypy-PyCharm-plugin <https://github.com/dropbox/mypy-PyCharm-plugin>`__ can be an interesting alternative.
- `PyDev <http://www.pydev.org/index.html>`__ - Works out-of-the-box via built-in MyPy code analyzer (7.0.3+).
- `VIM <https://www.vim.org/>`__ / `Neovim <https://neovim.io/>`__ - Using `vim-mypy <https://github.com/Integralist/vim-mypy>`__, `syntastic <https://github.com/vim-syntastic/syntastic>`__ or `Neomake <https://github.com/neomake/neomake>`__.
- `Visual Studio Code <https://code.visualstudio.com/>`__ - With `Mypy linter <https://code.visualstudio.com/docs/python/linting#_specific-linters>`__.
- Environment independent - Just use your favorite checker directly, optionally combined with tool like `entr <http://eradman.com/entrproject/>`__.

This package is tested against MyPy development branch and in rare cases (primarily important upstrean bugfixes), is not compatible with the preceding MyPy release.

PySpark Version Compatibility
-----------------------------

Package versions follow PySpark versions with exception to maintenance releases - i.e. `pyspark-stubs==2.3.0` should be compatible with `pyspark>=2.3.0,<2.4.0`.
Maintenance releases (`post1`, `post2`, ..., `postN`) are reserved for internal annotations updates.

API Coverage:
-------------

As of release 2.4.0 most of the public API is covered. For details please check `API coverage document <https://github.com/zero323/pyspark-stubs/blob/master/doc/api-coverage.rst>`__.

See also
--------

- `SPARK-17333 <https://issues.apache.org/jira/browse/SPARK-17333>`__ - *Make pyspark interface friendly with static analysis*.
- `PySpark typing hints <http://apache-spark-developers-list.1001551.n3.nabble.com/PYTHON-PySpark-typing-hints-td21560.html>`__ and `Revisiting PySpark type annotations <http://apache-spark-developers-list.1001551.n3.nabble.com/Re-PySpark-Revisiting-PySpark-type-annotations-td26232.html>`__ on `Apache Spark Developers List <http://apache-spark-developers-list.1001551.n3.nabble.com/>`__.


Disclaimer
----------


Apache Spark, Spark, PySpark, Apache, and the Spark logo are `trademarks <https://www.apache.org/foundation/marks/>`__ of `The
Apache Software Foundation <http://www.apache.org/>`__. This project is not owned, endorsed, or
sponsored by The Apache Software Foundation.

.. |Build Status| image:: https://travis-ci.org/zero323/pyspark-stubs.svg?branch=master
   :target: https://travis-ci.org/zero323/pyspark-stubs
.. |PyPI version| image:: https://img.shields.io/pypi/v/pyspark-stubs.svg
   :target: https://pypi.org/project/pyspark-stubs/
.. |Conda Forge version| image:: https://img.shields.io/conda/vn/conda-forge/pyspark-stubs.svg
   :target: https://anaconda.org/conda-forge/pyspark-stubs
.. |SPARK-20631| image:: https://i.imgur.com/GfDCGjv.gif
     :alt: SPARK-20631
.. |Syntax completion| image:: https://i.imgur.com/qvkLTAp.gif
     :alt: Syntax completion
