Metadata-Version: 2.4
Name: spydey
Version: 0.6.0
Summary: A web spider, useful for link checking, with several recursion strategies.
Author-email: Paul Winkler <slinkp@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/slinkp/spydey
Project-URL: Repository, https://github.com/slinkp/spydey
Project-URL: Changelog, https://github.com/slinkp/spydey/blob/main/CHANGES.txt
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Site Management :: Link Checking
Requires-Python: >=3.9
Description-Content-Type: text/x-rst
Requires-Dist: setuptools>=79.0.0
Requires-Dist: lxml>=5
Requires-Dist: httplib2>=0.20.1
Provides-Extra: build
Requires-Dist: build>=1.2; extra == "build"
Requires-Dist: docutils; extra == "build"
Provides-Extra: cli
Requires-Dist: fabulous>=0.4.0; extra == "cli"

Spydey
=======

A simple web spider with several recursion strategies.
Home page is at http://github.com/slinkp/spydey.

It doesn't do much except follow links and report status.  I mostly
use it for quick and dirty smoke testing and link checking.

The only unusual feature is the ``--traversal=pattern`` option, which
does recursive traversal in an unusual order: It tries to recognize
patterns in URLs, and will follow URLs of novel patterns before those
with patterns it has seen before.  When there are no novel patterns to
follow, it follows random links to URLs of known patterns. If you use
this for smoke-testing a typical modern web app that maps URL
patterns to views/controllers, this will very quickly hit all your
views/controllers at least once... usually.  But it's not very
interesting when pointed at a website that has arbitrarily deep trees
(static files, VCS repositories, and the like).

Also, it's designed so that adding a new recursion strategy is
trivial. Spydey was originally written for the purpose of
experimenting with different recursive crawling strategies. Read the
source.

Oh, and if you install Fabulous, console output is in color.

For lazy, zero-configuration smoke testing, I typically run it like::

  spydey -r --stop-on-error --max-requests=200 --traversal=pattern --profile --log-referrer URL

There are a number of other command-line options, many stolen from
wget. Use ``--help`` to see what they are.


Installation
==============

Any of the standard python install tools should work;
eg ``pip install spydey`` will install the latest from pypi.

Development
============

To build a new distribution:

- Bump the version number in ``spydey/__init__.py``
- Add an entry to the top of ``CHANGES.txt``
- Run this command::

    python3 -m build --sdist .
