huey — huey 2.4.5 documentation

# huey

a lightweight alternative.

huey is:

- a task queue (2019-04-01: version 2.0 released)

- written in python (2.7+, 3.4+)

- clean and simple API

- redis, sqlite, file-system, or in-memory storage

- example code.

huey supports:

- multi-process, multi-thread or greenlet task execution models

- schedule tasks to execute at a given time, or after a given delay

- schedule recurring tasks, like a crontab

- automatically retry tasks that fail

- task prioritization

- task result storage

- task expiration

- task locking

- task pipelines and chains

## At a glance

`task()` and `periodic_task()` decorators turn functions into tasks executed by the consumer:

    from huey import RedisHuey, crontab

    huey = RedisHuey('my-app', host='redis.myapp.com')

    @huey.task()
    def add_numbers(a, b):
        return a + b

    @huey.task(retries=2, retry_delay=60)
    def flaky_task(url):
        # This task might fail, in which case it will be retried up to 2 times
        # with a delay of 60s between retries.
        return this_might_fail(url)

    @huey.periodic_task(crontab(minute='0', hour='3'))
    def nightly_backup():
        sync_all_data()

Calling a `task`-decorated function will enqueue the function call for execution by the consumer. A special result handle is returned immediately, which can be used to fetch the result once the task is finished:

    >>> from demo import add_numbers
    >>> res = add_numbers(1, 2)
    >>> res
    <Result: task 6b6f36fc-da0d-4069-b46c-c0d4ccff1df6>

    >>> res()
    3

Tasks can be scheduled to run in the future:

    >>> res = add_numbers.schedule((2, 3), delay=10)  # Will be run in ~10s.
    >>> res(blocking=True)  # Will block until task finishes, in ~10s.
    5

For much more, check out the Guide or take a look at the example code.

### Running the consumer

Run the consumer with four worker processes:

    $ huey_consumer.py my_app.huey -k process -w 4

To run the consumer with a single worker thread (default):

    $ huey_consumer.py my_app.huey

If your work-loads are mostly IO-bound, you can run the consumer with threads or greenlets instead. Because greenlets are so lightweight, you can run quite a few of them efficiently:

    $ huey_consumer.py my_app.huey -k greenlet -w 32

For more information, see the Consuming Tasks document.

## Table of contents

- Installing
  - Using git
- Guide
  - Scheduling tasks
  - Periodic tasks
  - Retrying tasks that fail
  - Task priority
  - Canceling or pausing tasks
  - Canceling or pausing periodic tasks
  - Task expiration
  - Task pipelines
  - Locking tasks
  - Signals
  - Immediate mode
  - Logging
  - Storage Options
  - Tips and tricks
  - Reading more
- Consuming Tasks
  - Options for the consumer
  - Worker types
  - Consumer shutdown
  - Consumer restart
  - Consumer Internals
  - Signals
- Understanding how tasks are imported
  - Suggested organization of code
- Managing shared resources
  - Startup hooks
  - Pre and post execute hooks
- Signals
  - Examples
- Huey’s API
  - Huey types
  - Huey object
  - Result
  - Serializer
  - Exceptions
  - Storage
- Huey Extensions
  - Mini-Huey
  - Django
- Troubleshooting and Common Pitfalls
- Changes in 2.0
  - Summary
  - Details

Huey is named in honor of my cat

# Indices and tables

- Index

- Module Index

- Search Page