Metadata-Version: 2.1
Name: reppy2
Version: 0.3.6
Summary: Replacement robots.txt Parser in pure Python
Home-page: http://github.com/seomoz/reppy
Author: Dan Lecocq
Author-email: dan@moz.com
License: MIT License
Keywords: utilities
Platform: UNKNOWN
Classifier: License :: OSI Approved :: MIT License
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP
Provides-Extra: testing
Provides-Extra: docs
License-File: LICENSE

Replaces the built-in robotsparser with a
RFC-conformant implementation that supports modern robots.txt constructs like
Sitemaps, Allow, and Crawl-delay. Main features:

- Memoization of fetched robots.txt
- Expiration taken from the `Expires` header
- Batch queries
- Configurable user agent for fetching robots.txt
- Automatic refetching basing on expiration

This is a patched fork of the last pure Python version that
works on Python 2 and 3.

