Kedro plugins — Kedro 0.17.6 documentation

# Kedro plugins

Kedro plugins allow you to create new features for Kedro and inject additional commands into the CLI. Plugins are developed as separate Python packages that exist outside of any Kedro project.

## Overview

Kedro uses `setuptools`, which is a collection of enhancements to the Python `distutils` to allow developers to build and distribute Python packages. Kedro uses various entry points in `pkg_resources` to provide plugin functionality.

## Example of a simple plugin

Here is a simple example of a plugin that prints the pipeline as JSON:

`kedrojson/plugin.py`

    import click
    from kedro.framework.session import KedroSession

    @click.group(name="JSON")
    def commands():
        """Kedro plugin for printing the pipeline in JSON format"""
        pass

    @commands.command()
    @click.pass_obj
    def to_json(metadata):
        """Display the pipeline in JSON format"""
        session = KedroSession.create(metadata.package_name)
        context = session.load_context()
        print(context.pipeline.to_json())

The plugin provides the following `entry_points` config in `setup.py`:

    setup(
        entry_points={"kedro.project_commands": ["kedrojson = kedrojson.plugin:commands"]}
    )

Once the plugin is installed, you can run it as follows:

    kedro to_json

## Working with `click`

Commands must be provided as `click` `Groups`

The `click Group` will be merged into the main CLI Group. In the process, the options on the group are lost, as is any processing that was done as part of its callback function.

## Project context

When they run, plugins may request information about the current project by creating a session and loading its context:

    from pathlib import Path

    from kedro.framework.startup import _get_project_metadata
    from kedro.framework.session import KedroSession

    project_path = Path.cwd()
    metadata = _get_project_metadata(project_path)
    session = KedroSession.create(metadata.package_name, project_path)
    context = session.load_context()

## Initialisation

If the plugin initialisation needs to occur prior to Kedro starting, it can declare the `entry_point` key `kedro.init`. This entry point must refer to a function that currently has no arguments, but for future proofing you should declare it with `**kwargs`.

## `global` and `project` commands

Plugins may also add commands to the Kedro CLI, which supports two types of commands:

- global - available both inside and outside a Kedro project. Global commands use the `entry_point` key `kedro.global_commands`.

- project - available only when a Kedro project is detected in the current directory. Project commands use the `entry_point` key `kedro.project_commands`.

## Suggested command convention

We use the following command convention: `kedro <plugin-name> <command>`, with `kedro <plugin-name>` acting as a top-level command group. This is our suggested way of structuring your plugin bit it is not necessary for your plugin to work.

## Hooks

You can develop hook implementations and have them automatically registered to the project context when the plugin is installed. To enable this for your custom plugin, simply add the following entry in your `setup.py`:

    setup(entry_points={"kedro.hooks": ["plugin_name = plugin_name.plugin:hooks"]})

where `plugin.py` is the module where you declare hook implementations:

    import logging

    from kedro.framework.hooks import hook_impl

    class MyHooks:
        @hook_impl
        def after_catalog_created(self, catalog):  # pylint: disable=unused-argument
            logging.info("Reached after_catalog_created hook")

    hooks = MyHooks()

Note

`hooks` should be an instance of the class defining the Hooks.

## CLI Hooks

You can also develop hook implementations to extend Kedro’s CLI behaviour in your plugin. To find available CLI hooks, please visit kedro.framework.cli.hooks. To register CLI hooks developed in your plugin with Kedro, add the following entry in your project’s `setup.py`:

    setup(entry_points={"kedro.cli_hooks": ["plugin_name = plugin_name.plugin:cli_hooks"]})

where `plugin.py` is the module where you declare hook implementations:

    import logging

    from kedro.framework.cli.hooks import cli_hook_impl

    class MyCLIHooks:
        @cli_hook_impl
        def before_command_run(self, project_metadata, command_args):
            logging.info(
                "Command %s will be run for project %s", command_args, project_metadata
            )

    cli_hooks = MyCLIHooks()

## Contributing process

When you are ready to submit your code:

1. Create a separate repository using our naming convention for `plugin`s (`kedro-<plugin-name>`)

2. Choose a command approach: `global` and / or `project` commands:

   - All `global` commands should be provided as a single `click` group

   - All `project` commands should be provided as another `click` group

   - The `click` groups are declared through the `pkg_resources` entry_point system

3. Include a `README.md` describing your plugin’s functionality and all dependencies that should be included

4. Use GitHub tagging to tag your plugin as a `kedro-plugin` so that we can find it

## Supported Kedro plugins

- Kedro-Docker, a tool for packaging and shipping Kedro projects within containers

- Kedro-Airflow, a tool for converting your Kedro project into an Airflow project

- Kedro-Viz, a tool for visualising your Kedro pipelines

## Community-developed plugins

See the full list of plugins using the GitHub tag kedro-plugin.

Note

Your plugin needs to have an Apache 2.0 compatible license to be considered for this list.

- Kedro-Pandas-Profiling, by Justin Malloy, uses Pandas Profiling to profile datasets in the Kedro catalog

- find-kedro, by Waylon Walker, automatically constructs pipelines using `pytest`-style pattern matching

- kedro-static-viz, by Waylon Walker, generates a static Kedro-Viz site (HTML, CSS, JS)

- steel-toes, by Waylon Walker, prevents stepping on toes by automatically branching data paths

- kedro-wings, by Tam-Sanh Nguyen, simplifies and speeds up pipeline creation by auto-generating catalog datasets

- kedro-great, by Tam-Sanh Nguyen, integrates Kedro with Great Expectations, enabling catalog-based expectation generation and data validation on pipeline run

- Kedro-Accelerator, by Deepyaman Datta, speeds up pipelines by parallelizing I/O in the background

- kedro-dataframe-dropin, by Zain Patel, lets you swap out pandas datasets for modin or RAPIDs equivalents for specialised use to speed up your workflows (e.g on GPUs)

- kedro-kubeflow, by Mateusz Pytel and Mariusz Strzelecki, lets you run and schedule pipelines on Kubernetes clusters using Kubeflow Pipelines

- kedro-mlflow, by Yolan Honoré-Rougé, and Takieddine Kadiri facilitates Mlflow integration inside Kedro projects while enforcing Kedro’s principles. Its main features are modular configuration, automatic parameters tracking, datasets versioning, Kedro pipelines packaging and serving and automatic synchronization between training and inference pipelines for high reproducibility of machine learning experiments and ease of deployment. A tutorial is provided in the kedro-mlflow-tutorial repo. You can find more information in the documentation.

- kedro-neptune, by Jakub Czakon and Rafał Jankowski, lets you have all the benefits of a nicely organized Kedro pipeline with Neptune: a powerful user interface built for ML metadata management. It lets you browse and filter pipeline executions, compare nodes and pipelines on metrics and parameters, and visualize pipeline metadata like learning curves, node outputs, and charts. For more information, tutorials and videos, go to the documentation.

- kedro-dolt, Max Hoffman and Oscar Batori, a plugin that allows you to expand the data versioning abilities of data scientists and engineers.

- kedro-airflow-k8s, GetInData, a plugin that enable running a Kedro pipeline with Airflow on Kubernetes Cluster.