Metadata-Version: 1.1
Name: spidey.py
Version: 0.4.3
Summary: A terribly coded web spider, but useful for recursive API downloads.
Home-page: https://github.com/khilnani/spidey.py
Author: khilnani
Author-email: nik@khilnani.org
License: MIT
Description: # spidey.py
        
        > Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.
        
        ## Installation
        
        > Pypi Location: https://pypi.python.org/pypi/spidey
        
        - Using Pypi - `pip install spidey`
        
        ## Usage
        
        > Run `spidey` for Detailed help.
        
        - `spidey --dir NEW_DIR --filter DOMAIN --url URL [--base BASE_URL]`
        - `spidey --dir NEW_DIR --filter DOMAIN --url URL --max MAX_DOWNLOADS`
        - Example - `spidey --dir test --filter 'www.google.com' --url 'https://www.google.com/' --max 20`
        
        ### More Examples
        
        ```
        spidey \
        	-d test \
        	-f 'www.google.com' \
        	-u 'https://www.google.com/' \
            -b 'https://www.google.com/' \
        	-hh '{"Accept" : "application/json"}' \
        	-n 2 \
            -m 10 \
            -s 5
        ```
        ```
        spidey \
        	--dir test \
        	--filter 'www.google.com' \
        	--url 'https://www.google.com/'' \ \
            --base 'https://www.google.com/
        	--headers '{"Accept" : "application/json"}' \
        	--depth 2 \
            --max 10 \
            --sleep 5
        
        ```
        
Keywords: web,api,downloader,spider,crawler
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Unix
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
