|jazzband| |pypi| |pyversions| |pre-commit| |buildstatus-gha| |codecov|

==================================
pip-tools = pip-compile + pip-sync
==================================

A set of command line tools to help you keep your ``pip``-based packages fresh,
even when you've pinned them.  You do pin them, right? (In building your Python application and its dependencies for production, you want to make sure that your builds are predictable and deterministic.)

.. image:: https://github.com/jazzband/pip-tools/raw/main/img/pip-tools-overview.svg
   :alt: pip-tools overview for phase II

.. |buildstatus-gha| image:: https://github.com/jazzband/pip-tools/workflows/CI/badge.svg
   :alt: GitHub Actions build status
   :target: https://github.com/jazzband/pip-tools/actions?query=workflow%3ACI
.. |codecov| image:: https://codecov.io/gh/jazzband/pip-tools/branch/main/graph/badge.svg
   :alt: Coverage
   :target: https://codecov.io/gh/jazzband/pip-tools
.. |jazzband| image:: https://jazzband.co/static/img/badge.svg
   :alt: Jazzband
   :target: https://jazzband.co/
.. |pre-commit| image:: https://results.pre-commit.ci/badge/github/jazzband/pip-tools/main.svg
   :alt: pre-commit.ci status
   :target: https://results.pre-commit.ci/latest/github/jazzband/pip-tools/main
.. |pypi| image:: https://img.shields.io/pypi/v/pip-tools.svg
   :alt: PyPI version
   :target: https://pypi.org/project/pip-tools/
.. |pyversions| image:: https://img.shields.io/pypi/pyversions/pip-tools.svg
   :alt: Supported Python versions
   :target: https://pypi.org/project/pip-tools/
.. _You do pin them, right?: https://nvie.com/posts/pin-your-packages/

Installation
============

Similar to ``pip``, ``pip-tools`` must be installed in each of your project's
`virtual environments`_:

.. code-block:: bash

    $ source /path/to/venv/bin/activate
    (venv) $ python -m pip install pip-tools

**Note**: all of the remaining example commands assume you've activated your
project's virtual environment.

.. _virtual environments: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments

Example usage for ``pip-compile``
=================================

The ``pip-compile`` command lets you compile a ``requirements.txt`` file from
your dependencies, specified in either ``pyproject.toml``, ``setup.cfg``,
``setup.py``, or ``requirements.in``.

Run it with ``pip-compile`` or ``python -m piptools compile``. If you use
multiple Python versions, you can also run ``py -X.Y -m piptools compile`` on
Windows and ``pythonX.Y -m piptools compile`` on other systems.

``pip-compile`` should be run from the same virtual environment as your
project so conditional dependencies that require a specific Python version,
or other environment markers, resolve relative to your project's
environment.

**Note**: If ``pip-compile`` finds an existing ``requirements.txt`` file that
fulfils the dependencies then no changes will be made, even if updates are
available. To compile from scratch, first delete the existing
``requirements.txt`` file, or see `Updating requirements`_ for alternative
approaches.

Requirements from ``pyproject.toml``
------------------------------------

The ``pyproject.toml`` file is the
`latest standard <https://peps.python.org/pep-0621/>`_ for configuring
packages and applications, and is recommended for new projects. ``pip-compile``
supports both installing your ``project.dependencies`` as well as your
``project.optional-dependencies``. Thanks to the fact that this is an
official standard, you can use ``pip-compile`` to pin the dependencies
in projects that use modern standards-adhering packaging tools like
`Hatch <https://hatch.pypa.io/>`_ or `flit <https://flit.pypa.io/>`_.

Suppose you have a Django application that is packaged using ``Hatch``, and you
want to pin it for production. You also want to pin your development tools
in a separate pin file. You declare ``django`` as a dependency and create an
optional dependency ``dev`` that includes ``pytest``:

.. code-block:: toml

    [build-system]
    requires = ["hatchling"]
    build-backend = "hatchling.build"

    [project]
    name = "my-cool-django-app"
    version = "42"
    dependencies = ["django"]

    [project.optional-dependencies]
    dev = ["pytest"]

You can produce your pin files as easily as:

.. code-block:: console

    $ pip-compile -o requirements.txt pyproject.toml
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    pip-compile --output-file=requirements.txt pyproject.toml
    #
    asgiref==3.6.0
        # via django
    django==4.1.7
        # via my-cool-django-app (pyproject.toml)
    sqlparse==0.4.3
        # via django

    $ pip-compile --extra dev -o dev-requirements.txt pyproject.toml
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    pip-compile --extra=dev --output-file=dev-requirements.txt pyproject.toml
    #
    asgiref==3.6.0
        # via django
    attrs==22.2.0
        # via pytest
    django==4.1.7
        # via my-cool-django-app (pyproject.toml)
    exceptiongroup==1.1.1
        # via pytest
    iniconfig==2.0.0
        # via pytest
    packaging==23.0
        # via pytest
    pluggy==1.0.0
        # via pytest
    pytest==7.2.2
        # via my-cool-django-app (pyproject.toml)
    sqlparse==0.4.3
        # via django
    tomli==2.0.1
        # via pytest

This is great for both pinning your applications, but also to keep the CI
of your open-source Python package stable.

Requirements from ``setup.py`` and ``setup.cfg``
------------------------------------------------

``pip-compile`` has also full support for ``setup.py``- and
``setup.cfg``-based projects that use ``setuptools``.

Just define your dependencies and extras as usual and run
``pip-compile`` as above.

Requirements from ``requirements.in``
-------------------------------------

You can also use plain text files for your requirements (e.g. if you don't
want your application to be a package). To use a ``requirements.in`` file to
declare the Django dependency:

.. code-block:: ini

    # requirements.in
    django

Now, run ``pip-compile requirements.in``:

.. code-block:: bash

    $ pip-compile requirements.in
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    pip-compile requirements.in
    #
    asgiref==3.6.0
        # via django
    django==4.1.7
        # via -r requirements.in
    sqlparse==0.4.3
        # via django

And it will produce your ``requirements.txt``, with all the Django dependencies
(and all underlying dependencies) pinned.

.. _Updating requirements:

Updating requirements
---------------------

``pip-compile`` generates a ``requirements.txt`` file using the latest versions
that fulfil the dependencies you specify in the supported files.

If ``pip-compile`` finds an existing ``requirements.txt`` file that fulfils the
dependencies then no changes will be made, even if updates are available.

To force ``pip-compile`` to update all packages in an existing
``requirements.txt``, run ``pip-compile --upgrade``.

To update a specific package to the latest or a specific version use the
``--upgrade-package`` or ``-P`` flag:

.. code-block:: bash

    # only update the django package
    $ pip-compile --upgrade-package django

    # update both the django and requests packages
    $ pip-compile --upgrade-package django --upgrade-package requests

    # update the django package to the latest, and requests to v2.0.0
    $ pip-compile --upgrade-package django --upgrade-package requests==2.0.0

You can combine ``--upgrade`` and ``--upgrade-package`` in one command, to
provide constraints on the allowed upgrades. For example to upgrade all
packages whilst constraining requests to the latest version less than 3.0:

.. code-block:: bash

    $ pip-compile --upgrade --upgrade-package 'requests<3.0'

Using hashes
------------

If you would like to use *Hash-Checking Mode* available in ``pip`` since
version 8.0, ``pip-compile`` offers ``--generate-hashes`` flag:

.. code-block:: bash

    $ pip-compile --generate-hashes requirements.in
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    pip-compile --generate-hashes requirements.in
    #
    asgiref==3.6.0 \
        --hash=sha256:71e68008da809b957b7ee4b43dbccff33d1b23519fb8344e33f049897077afac \
        --hash=sha256:9567dfe7bd8d3c8c892227827c41cce860b368104c3431da67a0c5a65a949506
        # via django
    django==4.1.7 \
        --hash=sha256:44f714b81c5f190d9d2ddad01a532fe502fa01c4cb8faf1d081f4264ed15dcd8 \
        --hash=sha256:f2f431e75adc40039ace496ad3b9f17227022e8b11566f4b363da44c7e44761e
        # via -r requirements.in
    sqlparse==0.4.3 \
        --hash=sha256:0323c0ec29cd52bceabc1b4d9d579e311f3e4961b98d174201d5622a23b85e34 \
        --hash=sha256:69ca804846bb114d2ec380e4360a8a340db83f0ccf3afceeb1404df028f57268
        # via django

Output File
-----------

To output the pinned requirements in a filename other than
``requirements.txt``, use ``--output-file``. This might be useful for compiling
multiple files, for example with different constraints on django to test a
library with both versions using `tox <https://tox.readthedocs.io/en/latest/>`__:

.. code-block:: bash

    $ pip-compile --upgrade-package 'django<1.0' --output-file requirements-django0x.txt
    $ pip-compile --upgrade-package 'django<2.0' --output-file requirements-django1x.txt

Or to output to standard output, use ``--output-file=-``:

.. code-block:: bash

    $ pip-compile --output-file=- > requirements.txt
    $ pip-compile - --output-file=- < requirements.in > requirements.txt

Forwarding options to ``pip``
-----------------------------

Any valid ``pip`` flags or arguments may be passed on with ``pip-compile``'s
``--pip-args`` option, e.g.

.. code-block:: bash

    $ pip-compile requirements.in --pip-args "--retries 10 --timeout 30"

Configuration
-------------

You might be wrapping the ``pip-compile`` command in another script. To avoid
confusing consumers of your custom script you can override the update command
generated at the top of requirements files by setting the
``CUSTOM_COMPILE_COMMAND`` environment variable.

.. code-block:: bash

    $ CUSTOM_COMPILE_COMMAND="./pipcompilewrapper" pip-compile requirements.in
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    ./pipcompilewrapper
    #
    asgiref==3.6.0
        # via django
    django==4.1.7
        # via -r requirements.in
    sqlparse==0.4.3
        # via django

Workflow for layered requirements
---------------------------------

If you have different environments that you need to install different but
compatible packages for, then you can create layered requirements files and use
one layer to constrain the other.

For example, if you have a Django project where you want the newest ``2.1``
release in production and when developing you want to use the Django debug
toolbar, then you can create two ``*.in`` files, one for each layer:

.. code-block:: ini

    # requirements.in
    django<2.2

At the top of the development requirements ``dev-requirements.in`` you use ``-c
requirements.txt`` to constrain the dev requirements to packages already
selected for production in ``requirements.txt``.

.. code-block:: ini

    # dev-requirements.in
    -c requirements.txt
    django-debug-toolbar<2.2

First, compile ``requirements.txt`` as usual:

.. code-block:: bash

    $ pip-compile
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    pip-compile
    #
    django==2.1.15
        # via -r requirements.in
    pytz==2023.3
        # via django


Now compile the dev requirements and the ``requirements.txt`` file is used as
a constraint:

.. code-block:: bash

    $ pip-compile dev-requirements.in
    #
    # This file is autogenerated by pip-compile with Python 3.10
    # by the following command:
    #
    #    pip-compile dev-requirements.in
    #
    django==2.1.15
        # via
        #   -c requirements.txt
        #   django-debug-toolbar
    django-debug-toolbar==2.1
        # via -r dev-requirements.in
    pytz==2023.3
        # via
        #   -c requirements.txt
        #   django
    sqlparse==0.4.3
        # via django-debug-toolbar

As you can see above, even though a ``2.2`` release of Django is available, the
dev requirements only include a ``2.1`` version of Django because they were
constrained. Now both compiled requirements files can be installed safely in
the dev environment.

To install requirements in production stage use:

.. code-block:: bash

    $ pip-sync

You can install requirements in development stage by:

.. code-block:: bash

    $ pip-sync requirements.txt dev-requirements.txt


Version control integration
---------------------------

You might use ``pip-compile`` as a hook for the `pre-commit <https://github.com/pre-commit/pre-commit>`_.
See `pre-commit docs <https://pre-commit.com/>`_ for instructions.
Sample ``.pre-commit-config.yaml``:

.. code-block:: yaml

    repos:
      - repo: https://github.com/jazzband/pip-tools
        rev: 6.13.0
        hooks:
          - id: pip-compile

You might want to customize ``pip-compile`` args by configuring ``args`` and/or ``files``, for example:

.. code-block:: yaml

    repos:
      - repo: https://github.com/jazzband/pip-tools
        rev: 6.13.0
        hooks:
          - id: pip-compile
            files: ^requirements/production\.(in|txt)$
            args: [--index-url=https://example.com, requirements/production.in]

If you have multiple requirement files make sure you create a hook for each file.

.. code-block:: yaml

    repos:
      - repo: https://github.com/jazzband/pip-tools
        rev: 6.13.0
        hooks:
          - id: pip-compile
            name: pip-compile setup.py
            files: ^(setup\.py|requirements\.txt)$
          - id: pip-compile
            name: pip-compile requirements-dev.in
            args: [requirements-dev.in]
            files: ^requirements-dev\.(in|txt)$
          - id: pip-compile
            name: pip-compile requirements-lint.in
            args: [requirements-lint.in]
            files: ^requirements-lint\.(in|txt)$
          - id: pip-compile
            name: pip-compile requirements.txt
            args: [requirements.txt]
            files: ^requirements\.(in|txt)$


Example usage for ``pip-sync``
==============================

Now that you have a ``requirements.txt``, you can use ``pip-sync`` to update
your virtual environment to reflect exactly what's in there. This will
install/upgrade/uninstall everything necessary to match the
``requirements.txt`` contents.

Run it with ``pip-sync`` or ``python -m piptools sync``. If you use multiple
Python versions, you can also run ``py -X.Y -m piptools sync`` on Windows and
``pythonX.Y -m piptools sync`` on other systems.

``pip-sync`` must be installed into and run from the same virtual
environment as your project to identify which packages to install
or upgrade.

**Be careful**: ``pip-sync`` is meant to be used only with a
``requirements.txt`` generated by ``pip-compile``.

.. code-block:: bash

    $ pip-sync
    Uninstalling flake8-2.4.1:
      Successfully uninstalled flake8-2.4.1
    Collecting click==4.1
      Downloading click-4.1-py2.py3-none-any.whl (62kB)
        100% |................................| 65kB 1.8MB/s
      Found existing installation: click 4.0
        Uninstalling click-4.0:
          Successfully uninstalled click-4.0
    Successfully installed click-4.1

To sync multiple ``*.txt`` dependency lists, just pass them in via command
line arguments, e.g.

.. code-block:: bash

    $ pip-sync dev-requirements.txt requirements.txt

Passing in empty arguments would cause it to default to ``requirements.txt``.

Any valid ``pip install`` flags or arguments may be passed with ``pip-sync``'s
``--pip-args`` option, e.g.

.. code-block:: bash

    $ pip-sync requirements.txt --pip-args "--no-cache-dir --no-deps"

**Note**: ``pip-sync`` will not upgrade or uninstall packaging tools like
``setuptools``, ``pip``, or ``pip-tools`` itself. Use ``python -m pip install --upgrade``
to upgrade those packages.

Should I commit ``requirements.in`` and ``requirements.txt`` to source control?
===============================================================================

Generally, yes. If you want a reproducible environment installation available from your source control,
then yes, you should commit both ``requirements.in`` and ``requirements.txt`` to source control.

Note that if you are deploying on multiple Python environments (read the section below),
then you must commit a separate output file for each Python environment.
We suggest to use the ``{env}-requirements.txt`` format
(ex: ``win32-py3.7-requirements.txt``, ``macos-py3.10-requirements.txt``, etc.).


Cross-environment usage of ``requirements.in``/``requirements.txt`` and ``pip-compile``
=======================================================================================

The dependencies of a package can change depending on the Python environment in which it
is installed.  Here, we define a Python environment as the combination of Operating
System, Python version (3.7, 3.8, etc.), and Python implementation (CPython, PyPy,
etc.). For an exact definition, refer to the possible combinations of `PEP 508
environment markers`_.

As the resulting ``requirements.txt`` can differ for each environment, users must
execute ``pip-compile`` **on each Python environment separately** to generate a
``requirements.txt`` valid for each said environment.  The same ``requirements.in`` can
be used as the source file for all environments, using `PEP 508 environment markers`_ as
needed, the same way it would be done for regular ``pip`` cross-environment usage.

If the generated ``requirements.txt`` remains exactly the same for all Python
environments, then it can be used across Python environments safely. **But** users
should be careful as any package update can introduce environment-dependent
dependencies, making any newly generated ``requirements.txt`` environment-dependent too.
As a general rule, it's advised that users should still always execute ``pip-compile``
on each targeted Python environment to avoid issues.

.. _PEP 508 environment markers: https://www.python.org/dev/peps/pep-0508/#environment-markers

Other useful tools
==================

- `pipdeptree`_ to print the dependency tree of the installed packages.
- ``requirements.in``/``requirements.txt`` syntax highlighting:

  * `requirements.txt.vim`_ for Vim.
  * `Python extension for VS Code`_ for VS Code.
  * `pip-requirements.el`_ for Emacs.

.. _pipdeptree: https://github.com/naiquevin/pipdeptree
.. _requirements.txt.vim: https://github.com/raimon49/requirements.txt.vim
.. _Python extension for VS Code: https://marketplace.visualstudio.com/items?itemName=ms-python.python
.. _pip-requirements.el: https://github.com/Wilfred/pip-requirements.el


Deprecations
============

This section lists ``pip-tools`` features that are currently deprecated.

- In future versions, the ``--allow-unsafe`` behavior will be enabled by
  default. Use ``--no-allow-unsafe`` to keep the old behavior. It is
  recommended to pass the ``--allow-unsafe`` now to adapt to the upcoming
  change.
- Legacy resolver is deprecated and will be removed in future versions.
  Use ``--resolver=backtracking`` instead.

A Note on Resolvers
===================

You can choose from either the legacy or the backtracking resolver.
The backtracking resolver is recommended, and will become the default
with the 7.0 release.

Use it now with the ``--resolver=backtracking`` option to ``pip-compile``.

The legacy resolver will occasionally fail to resolve dependencies. The
backtracking resolver is more robust, but can take longer to run in
general.

You can continue using the legacy resolver with ``--resolver=legacy``.

Versions and compatibility
==========================

The table below summarizes the latest ``pip-tools`` versions with the required
``pip`` and Python versions. Generally, ``pip-tools`` supports the same Python
versions as the required ``pip`` versions.

+----------------+----------------+----------------+
| pip-tools      | pip            | Python         |
+================+================+================+
| 4.5.*          | 8.1.3 - 20.0.2 | 2.7, 3.5 - 3.8 |
+----------------+----------------+----------------+
| 5.0.0 - 5.3.0  | 20.0 - 20.1.1  | 2.7, 3.5 - 3.8 |
+----------------+----------------+----------------+
| 5.4.0          | 20.1 - 20.3.*  | 2.7, 3.5 - 3.8 |
+----------------+----------------+----------------+
| 5.5.0          | 20.1 - 20.3.*  | 2.7, 3.5 - 3.9 |
+----------------+----------------+----------------+
| 6.0.0 - 6.3.1  | 20.3 - 21.2.*  | 3.6 - 3.9      |
+----------------+----------------+----------------+
| 6.4.0          | 21.2 - 21.3.*  | 3.6 - 3.10     |
+----------------+----------------+----------------+
| 6.5.0 - 6.10.0 | 21.2 - 22.3.*  | 3.7 - 3.11     |
+----------------+----------------+----------------+
| 6.11.0+        | 22.2+          | 3.7 - 3.11     |
+----------------+----------------+----------------+
