Metadata-Version: 2.1
Name: scrapyduler
Version: 1.0.0
Summary: Scrapyd launcher module that schedules scrapy spiders by time
Author-email: Dmitriy Chuenkov <slymit@gmail.com>
License: BSD-3-Clause
Keywords: scrapy,scrapyd,scheduler,launcher
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Environment :: Console
Classifier: Environment :: No Input/Output (Daemon)
Classifier: Topic :: Internet :: WWW/HTTP
Requires-Python: >=3.7
Description-Content-Type: text/x-rst

scrapyduler
===========

Scrapyd launcher module that schedules scrapy spiders by time.

Install
-------

.. code-block:: shell

    $ pip install scrapyduler

Config
------

To start using this library you just need to override
the ``launcher`` option in your ``scrapyd.conf`` file:

.. code-block:: text

    [scrapyd]
    launcher = scrapyduler.launcher.Launcher

and then add the schedulers configuration, e.g.:

.. code-block:: text

    [scheduler.1]
    cron        = * * * * *
    project     = quotesbot
    spider      = toscrape-xpath
    settings    = HTTPPROXY_ENABLED=True
    args        = key1=value1 key2=value2 start_url=http://quotes.toscrape.com/

    [scheduler.2]
    interval    = weeks=0 days=0 hours=0 minutes=0 seconds=30
    project     = quotesbot
    spider      = toscrape-css
    settings    = HTTPPROXY_ENABLED=True
    args        = key1=value1 key2=value2 start_url=http://quotes.toscrape.com/

In the examples above, we set up two schedulers.
The first scheduler uses cron syntax to run spiders.
The second scheduler triggers on specified intervals,
starting on ``start_date`` if specified, ``datetime.now()`` + interval otherwise.
See https://github.com/agronholm/apscheduler for more information.
