# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['buildaspider']

package_data = \
{'': ['*']}

install_requires = \
['bs4>=0.0.1,<0.0.2', 'prettytable>=0.7.2,<0.8.0', 'requests>=2.24.0,<3.0.0']

setup_kwargs = {
    'name': 'buildaspider',
    'version': '0.9.3',
    'description': 'Build yourself a small site crawler.',
    'long_description': '============\nbuildaspider\n============\n\n\nA simple, configurable web crawler written in Python.\n\n\nDesigned to be a jumping off point for:\n\n+ understanding and implementing your own crawler\n+ parsing markup with `bs4 <https://www.crummy.com/software/BeautifulSoup/bs4/doc/BeautifulSoup>`_\n+ working with `requests <https://requests.readthedocs.io/en/master/>`_\n\n\nWhile its aims are more educational than industrial, it may still be suitable for crawling sites of moderate size (<1000 unique pages). \n\n\nWritten such that it can either be used as-is for small sites, or extended for any number of crawling applications.\n\n\n**buildaspider** is intended as a platform to learn to build tools for your own quality assurance purposes.\n\n============\nInstallation\n============\n\n**Option 1:**\n\n\n.. code-block::\n\n    pip install buildaspider\n\n\n**Option 2:**\n\n\n.. code-block:: \n\n    git clone git@github.com:joedougherty/buildaspider.git\n    cd buildaspider/\n    python3 setup.py install\n\n\n\n===================\nExample Config File\n===================\n\n\nA config file is **required**. In addition to the sample given below, you can find an example file in **examples/cgf.ini**.\n\n\n.. code-block::\n\n    [buildaspider]\n\n    ; login = true \n    ; In order to programatically login, uncomment the line above and ensure login = true\n    ;\n    ; You will also need to ensure that:\n    ;   + the username line is uncommented and set correctly\n    ;   + the password line is uncommented and set correctly\n    ;   + the login_url line is uncommented and set correctly\n\n    ; username = <USERNAME>\n    ; password = <PASSWORD>\n    ; login_url = http://example.com/login\n\n    ; Absolute path to directory containing per-run logs\n    ; log_dir = /path/to/logs\n\n    ; Literal URLs to visit -- there must be at least one!\n    seed_urls = \n        http://httpbin.org/\n\n    ; List of regex patterns to include\n    include_patterns =\t\n        httpbin.org\n\n    ; List of regex patterns to exclude\n    exclude_patterns =\n        ^#$\n        ^javascript\n\n    max_num_retries = 5\n\n\n\n===========\nBasic Usage\n===========\n\n\nOnce the config file is created and ready to go, it is time to create a **Spider** instance.\n\n\n.. code-block:: python\n\n    from buildaspider import Spider\n\n\n    myspider = Spider(\n        \'/path/to/cfg.ini\',\n        # These are the default settings\n        max_workers=8,\n        time_format="%Y-%m-%d_%H:%M",\n    )\n\n    myspider.weave()\n\n\nThis will start the web crawling process, beginning with the URLs specified in **seed_urls** in the config file.\n\n\n=======\nLogging\n=======\n\n\nBy default, each run generates four logs:\n\n\n+ status log\n+ broken links log\n+ checked links log\n+ exception links log \n\n\nThe implementation lives in the  **setup_logging** method of the **Spider** base class:\n\n\n.. code-block:: python\n\n\n    def setup_logging(self):\n        now = datetime.now().strftime(self.time_format)\n\n        logging.basicConfig(\n            filename=os.path.join(self.cfg.log_dir, f"spider_{now}.log"),\n            level=logging.INFO,\n            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",\n        )\n\n        self.status_logger = logging.getLogger(__name__)\n\n        self.broken_links_logpath = os.path.join(\n            self.cfg.log_dir, f"broken_links_{now}.log"\n        )\n        self.checked_links_logpath = os.path.join(\n            self.cfg.log_dir, f"checked_links_{now}.log"\n        )\n        self.exception_links_logpath = os.path.join(\n            self.cfg.log_dir, f"exception_links_{now}.log"\n        )\n\n\n\nThere are three rudimentary methods provided that write to each of the above logs:\n\n\n+ **log_checked_link**\n+ **log_broken_link**\n+ **log_exception_link**\n\n\nFor example:\n\n\n.. code-block:: python\n\n    def log_checked_link(self, link):\n        append_line_to_log(self.checked_links_logpath, f\'{link}\')\n\n\nThis can be overridden to extend logging capabilities. \n\n\nThese methods can also can be overriden to trigger custom behavior when:\n\n+ a link is checked\n+ a broken link is found\n+ a link that threw an exception is found\n\n\n==================\nBeyond Basic Usage\n==================\n\nAdding the Ability to Login\n---------------------------\n\nYou can extend the functionality of **buildaspider** by inheriting from the **Spider** class and overriding methods. \n\n\nThis is how you implement the ability for your spider to programmatically login.\n\n\nHere\'s the documentation from the base **Spider** class:\n\n\n.. code-block:: python\n\n    \n    def login(self):\n        # If your session doesn\'t require logging in, you can leave this method unimplemented.\n        #\n        # Otherwise, this method needs to return an instance of `requests.Session`.\n        #\n        # A new session can be obtained by calling `mint_new_session()`.\n        #\n        raise NotImplementedError("You\'ll need to implement the login method.")\n\n\n\nHere\'s an example of a fleshed-out login method to **POST** credentials (as obtained from the config file) to the login_url. (For more details on logging in with **requests** see: `<https://pybit.es/requests-session.html>`_.)\n\n\n\n.. code-block:: python\n\n    from buildaspider import Spider, mint_new_session, FailedLoginError\n\n\n    class MySpider(Spider):\n        def login(self):\n            new_session = mint_new_session()\n\n            login_payload = {\n                \'username\': self.cfg.username,\n                \'password\': self.cfg.password,\n            }\n\n            response = new_session.post(self.cfg.login_url, data=login_payload)\n            \n            if response.status_code != 200:\n                raise FailedLoginError("Login Failed :(")\n\n            return response\n        \n\n\n    myspider = MySpider(\'/path/to/cfg.ini\')\n\n    myspider.weave()\n\n\n\nProviding Custom Functionality by Attaching to Event Hooks\n----------------------------------------------------------\n\nThere are a few events that occur during the crawling process that you may want to attach some additional functionality to.\n\nThere are pre-visit and post-visit methods you can override/extend.\n\n\n\n+---------------------------------------------------+---------------------------+\n| Event                                             | Method                    |\n+===================================================+===========================+\n| link visit is about to begin                      | **.pre_visit_hook()**     |\n+---------------------------------------------------+---------------------------+\n| link visit is about to end                        | **.post_visit_hook()**    | \n+---------------------------------------------------+---------------------------+\n| a link has been marked as checked                 | **.log_checked_link()**   | \n+---------------------------------------------------+---------------------------+\n| a link has been marked as broken                  | **.log_broken_link()**    | \n+---------------------------------------------------+---------------------------+\n| a link has been marked as causing an exception    | **.log_exception_link()** | \n+---------------------------------------------------+---------------------------+\n| crawling is complete                              | **.cleanup()**            | \n+---------------------------------------------------+---------------------------+\n\n\n\n**Spider.pre_visit_hook()** provides the ability to run code when **.visit()** is called. Code specified in **.pre_visit_hook()** will execute prior to library-provided functionality in **.visit()**. \n\n**Spider.post_visit_hook()** provides the ability to run code right before **.visit()** finishes.\n\n\nThe overridden methods **.pre_visit_hook()** and **.post_visit_hook()** ought to pass in **link** in order to keep the current link in scope and available as a variable with that name. \n\n\nYou may choose to store visited links in some custom container:\n\n\n.. code-block:: python\n\n\n    custom_visited_links = list()\n    \n    def pre_visit_hook(self, link):\n        # The `link` being referenced here\n        # is the link about to be visited\n        custom_visited_links.append(link)\n\n\n\n**NOTE:** this provides direct access to the current **Link** object in scope. \n\nA safe strategy is to make a copy of the current **Link** using **deepcopy**.\n\n\n.. code-block:: python\n\n\n\n    from copy import deepcopy\n    \n\n    custom_visited_links = list()\n\n\n    def pre_visit_hook(self, link):\n        current_link_copy = deepcopy(link) \n        custom_visited_links.append(current_link_copy)\n\n\n\nExtending/Overriding Pre-Defined Events \n---------------------------------------\n\n\nBy default, broken links are logged to the location specified by **self.broken_links_logpath**.\n\nWe can see this in the **Spider** class:\n\n\n.. code-block:: python\n\n    def log_broken_link(self, link):\n        append_line_to_log(self.broken_links_logpath, f\'{link} :: {link.http_code}\')\n\n\n\nWhat if you want to *extend* (not merely override) the functionality of **.log_broken_link()**?\n\n\n\n.. code-block:: python\n\n    def log_broken_link(self, link):\n        super().log_broken_link(link)  \n        # You\'ve now retained the original functionality \n        # by running the method as defined on the parent instance\n\n        # Perhaps now you want to: \n        #   + cache this value?\n        #   + run some action(s) as a result of this event firing?\n        #   + ???\n\n\n\n======================\nRunning the Test Suite\n======================\n\n\n**NOTE:** You will need to ensure that the **log_dir** config file field is set correctly before you run the test suite. \n\n\n.. code-block::\n\n\n    cd tests/\n    pytest\n\n\n\n\n====================\nAdditional Resources\n====================\n\n\n**Official Retry Documentation**\n\nhttps://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retry\n\n\n**Advanced usage of Python requests - timeouts, retries, hooks**\n\nhttps://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/#retry-on-failure\n\n\n**Python stdlib Logging: basicConfig**\n\nhttps://docs.python.org/3.8/library/logging.html#logging.basicConfig\n\n\n**BFS / FIFO Queue**\n\nhttps://en.wikipedia.org/wiki/Breadth-first_search#Pseudocode\n\n\n**Python: A quick introduction to the concurrent.futures module**\n\nhttp://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html\n\n\n**Using Python Requests on a Page Behind a Login**\n\nhttps://pybit.es/requests-session.html\n\n\n**The Offical collections.deque Documentation**\n\nhttps://docs.python.org/3.8/library/collections.html#collections.deque\n',
    'author': 'Joe Dougherty',
    'author_email': 'joseph.dougherty@gmail.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': None,
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)
