spidy Web Crawler
=================

Spidy (/spˈɪdi/) is the simple, easy to use command line web crawler.
Given a list of web links, it uses Python
```requests`` <http://docs.python-requests.org>`__ to query the
webpages, and ```lxml`` <http://lxml.de/index.html>`__ to extract all
links from the page. Pretty simple!

|spidy Logo|

|Version: 1.5.1| |Release: 1.4.0| |License: GPL v3| |Python 3.3+| |All
Platforms!| |Open Source Love| |Lines of Code: 1437| |Lines of Docs:
564| |Last Commit| |Travis CI Status| |Contributors| |Forks| |Stars|

Created by `rivermont <https://github.com/rivermont>`__ (/rɪvɜːrmɒnt/)
and `FalconWarriorr <https://github.com/Casillas->`__ (/fælcʌnraɪjɔːr/),
and developed with help from `these awesome
people <https://github.com/rivermont/spidy#contributors>`__. Looking for
technical documentation? Check out
`DOCS.md <https://github.com/rivermont/spidy/blob/master/docs/DOCS.md>`__\ 
Looking to contribute to this project? Have a look at
```CONTRIBUTING.md`` <https://github.com/rivermont/spidy/blob/master/docs/CONTRIBUTING.md>`__,
then check out the docs.

--------------

Contributors
============

-  The logo was designed by `Cutwell <https://github.com/Cutwell>`__

-  `3onyc <https://github.com/3onyc>`__ - PEP8 Compliance.
-  `DeKaN <https://github.com/DeKaN>`__ - Getting PyPI packaging to
   work.
-  `esouthren <https://github.com/esouthren>`__ - Unit testing.
-  `j-setiawan <https://github.com/j-setiawan>`__ - Paths that work on
   all OS's.
-  `michellemorales <https://github.com/michellemorales>`__ - Confirmed
   OS/X support.
-  `quatroka <https://github.com/quatroka>`__ - Fixed testing bugs.
-  `stevelle <https://github.com/stevelle>`__ - Respect robots.txt.
-  `thatguywiththatname <https://github.com/thatguywiththatname>`__ -
   README link corrections

License
=======

We used the `Gnu General Public
License <https://www.gnu.org/licenses/gpl-3.0.en.html>`__ (see
`LICENSE <https://github.com/rivermont/spidy/blob/master/LICENSE>`__) as
it was the license that best suited our needs. Honestly, if you link to
this repo and credit ``rivermont`` and ``FalconWarriorr``, and you
aren't selling spidy in any way, then we would love for you to
distribute it. Thanks!

--------------

.. |spidy Logo| image:: https://raw.githubusercontent.com/rivermont/spidy/master/media/spidy_logo.png
   :target: https://github.com/rivermont/spidy#contributors
.. |Version: 1.5.1| image:: https://img.shields.io/badge/version-1.5.1-brightgreen.svg
.. |Release: 1.4.0| image:: https://img.shields.io/github/release/rivermont/spidy.svg
   :target: https://github.com/rivermont/spidy/releases
.. |License: GPL v3| image:: https://img.shields.io/badge/license-GPLv3.0-blue.svg
   :target: http://www.gnu.org/licenses/gpl-3.0
.. |Python 3.3+| image:: https://img.shields.io/badge/python-3.3+-brightgreen.svg
   :target: https://docs.python.org/3/
.. |All Platforms!| image:: https://img.shields.io/badge/Windows,%20OS/X,%20Linux-%20%20-brightgreen.svg
.. |Open Source Love| image:: https://badges.frapsoft.com/os/v1/open-source.png?v=103
.. |Lines of Code: 1437| image:: https://img.shields.io/badge/lines%20of%20code-1437-brightgreen.svg
.. |Lines of Docs: 564| image:: https://img.shields.io/badge/lines%20of%20docs-564-orange.svg
.. |Last Commit| image:: https://img.shields.io/github/last-commit/rivermont/spidy.svg
   :target: https://github.com/rivermont/spidy/graphs/punch-card
.. |Travis CI Status| image:: https://img.shields.io/travis/rivermont/spidy/master.svg
   :target: https://travis-ci.org/rivermont/spidy
.. |Contributors| image:: https://img.shields.io/github/contributors/rivermont/spidy.svg
   :target: https://github.com/rivermont/spidy/graphs/contributors
.. |Forks| image:: https://img.shields.io/github/forks/rivermont/spidy.svg?style=social&label=Forks
   :target: https://github.com/rivermont/spidy/network
.. |Stars| image:: https://img.shields.io/github/stars/rivermont/spidy.svg?style=social&label=Stars
   :target: https://github.com/rivermont/spidy/stargazers
