# -*- coding: utf-8 -*-
from setuptools import setup

package_dir = \
{'': 'src'}

packages = \
['thatscrapper', 'thatscrapper.common']

package_data = \
{'': ['*']}

install_requires = \
['lxml>=4.9.1,<5.0.0',
 'pandas>=1.5.0,<2.0.0',
 'polling2>=0.5.0,<0.6.0',
 'selenium>=4.5.0,<5.0.0']

setup_kwargs = {
    'name': 'thatscrapper',
    'version': '0.1.0',
    'description': 'A selenium framework for automated actions on browsers',
    'long_description': '# thatscrapper\n\nScrap more, write less.\n\n`thatscrapper` is a selenium adapter.\n\n>Selenium automates browsers. That\'s it! What you do with that power is entirely up to you.\n\n<div style="text-align: right;">\n<p>from <a href="https://www.selenium.dev/">Selenium official webpage</a></p>\n</div>\n\n>Selenium is a simple API to write functional/acceptance tests using Selenium WebDriver. Through Selenium Python API you can access all functionalities of Selenium WebDriver in an intuitive way.\n\n<div style="text-align: right;">\n<p>from <a href="https://selenium-python.readthedocs.io/index.html">Selenium with python</a></p>\n</div>\n\nSince there are so many websites full of javascript code, scrapping pages using static methods, as those found on Beautifulsoup, gets harder or impossible. With Selenium WebDriver you can get around with it.\n\nHowever it is very common to make sure the desired element is expected with certain conditions. For that you have to add \'waits\' contexts, like:\n\n```python\nfrom selenium import webdriver\nfrom selenium.webdriver.common.by import By\nfrom selenium.webdriver.support.ui import WebDriverWait\nfrom selenium.webdriver.support import expected_conditions as EC\n\ndriver = webdriver.Firefox()\ndriver.get("http://somedomain/url_that_delays_loading")\ntry:\n    # the waiting context with the selection of element by id\n    element = WebDriverWait(driver, 10).until(\n        EC.presence_of_element_located((By.ID, "myDynamicElement"))\n    )\nfinally:\n    driver.quit()\n```\n\nFor a single purpose web scraping single script that means some couple of lines like that. However, for a larger testing or scraping project, that can become anoying. With using `thatscrapper` above script becomes:\n\n```python\nimport thatscrapper as ts\n\ncrawler = ts.Crawler().goto("http://somedomain/url_that_delays_loading")\nelement = crawler.element_id("myDynamicElement")\ncrawler.quit()\n```\n\nIt is very important to quit the webdriver to avoid memory leakage or overcrowding. Always call `quit()` method when job is done . However, `Crawler` comes with a decorator that make sure the webdriver quits in case of any exception.\n\n\n## Installation\n\n```bash\n$ pip install thatscrapper\n```\n\n## Usage\n\n`thatscrapper` can be used to perform basic actions on webpages, such as clicking buttons, dropdown menu, press keyboard keys, send text or filling forms. It is also suitable to extract data.\n\nInstances of `thatscrapper.Crawler` are used to navigate pages, perform actions and select elements.\n\n\n### Run the webdriver:\n\n\n```python\nimport time\nimport thatscrapper\n\ncrawler = thatscrapper.Crawler()\n# open page\ncrawler.goto("https://phptravels.com/demo/")\n# wait long enough so you can check the result\ntime.sleep(5)\n# always quit the driver\ncrawler.quit()\n```\n\nAlternatively, you can crawl pages withou opening browser graphics:\n\n```python\ncrawler = thatscrapper.Crawler(headless=True)\n\n```\n\n### Choosing the webdriver\n\nBy default `thatscrapper` make use of FireFox webdriver ([geckodriver](https://github.com/mozilla/geckodriver/releases)), however other drivers can be selected. But make sure you have the one of your choosing, and its [path is added to your enviroment variables](https://selenium-python.readthedocs.io/installation.html#drivers). For Linux users, download the werdriver and put it in /usr/bin or /usr/local/bin (windows user, [check this out](https://stackoverflow.com/a/56926716/13599189) in order to see how to do that in your system).\n\nHere\'s a list of suported browser drivers:\n\n| **Browser**  | **Download link** |\n| ----------   | ----------------- |\n| **Chrome**   | https://sites.google.com/chromium.org/driver/ |\n| **Edge**     | https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ |\n| **FireFox**  | https://github.com/mozilla/geckodriver/releases |\n| **Safari**   | https://webkit.org/blog/6900/webdriver-support-in-safari-10/ |\n\nTo use other driver, pass it\'s browser name to ```Crawler``` class:\n\n```python\nimport time\nimport thatscrapper as ts\n\ncrawler = ts.Crawler(browser=\'chrome\')\n\n# extract data from\ncrawler.goto("https://www.techlistic.com/p/demo-selenium-practice.html")\n# wait long enough so you can check the result\ntime.sleep(5)\n# always quit the driver\ncrawler.quit()\n```\n\n### Elements their children\n\nElements can be selected with one of four methods:\n\n- `element(value, by)`: selects an element based on given attribute `by`, with value `value`. A list of attributes is given by `thatscrapper.ATTR_SELECTOR.keys()`.\n- `elements(value, by)`: selects all elements based on given attribute `by`, with value `value`.\n- `child_of(element, value, by)`: selects an element child of `element` (WebElement) based on given attribute `by`, with value `value`.\n- `children_of`: selects all elements child of `element` based on given attribute `by`, with value `value`.\n\nConsider the following section of a page:\n\n```html\n<div class="form">\n    <input type="text" name="first_name" class="first_name input mb1" placeholder="First Name">\n    <input type="text" name="last_name" class="last_name input mb1" placeholder="Last Name">\n    <input type="text" name="business_name" class="business_name input mb1" placeholder="Business Name">\n    ...\n</div>\n```\n\nIn order to make sure you select the input tags from that div with class="from", and not another input that the page may contain, first you can select that div, and then select its children:\n\n```python\nform_element = crawler.element("form", "class name")\nfields = crawler.children_of(form_element, "input", "tag name")\n``` \n\n### Sending keys\n\nSending inputs or keys to and element is performed one of two methods:\n\n- `send(key, value, by)`: send `key` to an element based on given attribute `by`, with value `value`.\n- `send(key, element)`: send `key` to previously selected `element`.\n\nConsider the same abov section of a page, and the selected elements `field`. Sending a string to element `<input type="text" name="first_name" class="first_name input mb1" placeholder="First Name">` can be:\n\n```python\ncrawler.send_to_element(fields[0], "Vagner Bessa")\n```\n\nSending keyboard keys works the same way. Check `thatscrapper.Key` or `selenium.webdriver.common.keys.Keys` for valid keys.\n\n## Contributing\n\nInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.\n\n## TODOs\n\n- [ ] Test all methods (current coverage: 61%).  \n- [ ] Implement data extraction modules and classes.  \n- [ ] Link or adapt to a database handler.\n- [ ] Implement an API boilerplate builder. That API is to serve data extracted from extractor modules and classes.  \n\n## License\n\n`thatscrapper` was created by Vagner Bessa. It is licensed under the terms of the MIT license.\n\n## Credits\n\n`thatscrapper` was created with [`cookiecutter`](https://cookiecutter.readthedocs.io/en/latest/) and the `py-pkgs-cookiecutter` [template](https://github.com/py-pkgs/py-pkgs-cookiecutter).\n',
    'author': 'Vagner Bessa',
    'author_email': 'None',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'None',
    'package_dir': package_dir,
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.9,<4.0',
}


setup(**setup_kwargs)
