# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['dude']

package_data = \
{'': ['*']}

install_requires = \
['PyYAML>=6.0,<7.0', 'playwright>=1.19.0,<2.0.0']

extras_require = \
{'bs4': ['beautifulsoup4>=4.10.0,<5.0.0', 'httpx>=0.22.0,<0.23.0']}

entry_points = \
{'console_scripts': ['dude = dude:cli']}

setup_kwargs = {
    'name': 'pydude',
    'version': '0.2.0',
    'description': 'dude uncomplicated data extraction',
    'long_description': '<table>\n    <tr>\n        <td>License</td>\n        <td><img src=\'https://img.shields.io/pypi/l/pydude.svg?style=for-the-badge\' alt="License"></td>\n        <td>Version</td>\n        <td><img src=\'https://img.shields.io/pypi/v/pydude.svg?logo=pypi&style=for-the-badge\' alt="Version"></td>\n    </tr>\n    <tr>\n        <td>Github Actions</td>\n        <td><img src=\'https://img.shields.io/github/workflow/status/roniemartinez/dude/Python?label=actions&logo=github%20actions&style=for-the-badge\' alt="Github Actions"></td>\n        <td>Coverage</td>\n        <td><img src=\'https://img.shields.io/codecov/c/github/roniemartinez/dude/branch?label=codecov&logo=codecov&style=for-the-badge\' alt="CodeCov"></td>\n    </tr>\n    <tr>\n        <td>Supported versions</td>\n        <td><img src=\'https://img.shields.io/pypi/pyversions/pydude.svg?logo=python&style=for-the-badge\' alt="Python Versions"></td>\n        <td>Wheel</td>\n        <td><img src=\'https://img.shields.io/pypi/wheel/pydude.svg?style=for-the-badge\' alt="Wheel"></td>\n    </tr>\n    <tr>\n        <td>Status</td>\n        <td><img src=\'https://img.shields.io/pypi/status/pydude.svg?style=for-the-badge\' alt="Status"></td>\n        <td>Downloads</td>\n        <td><img src=\'https://img.shields.io/pypi/dm/pydude.svg?style=for-the-badge\' alt="Downloads"></td>\n    </tr>\n</table>\n\n# dude uncomplicated data extraction\n\nDude is a very simple framework to write a web scraper using Python decorators.\nThe design, inspired by [Flask](https://github.com/pallets/flask), was to easily build a web scraper in just a few lines of code.\nDude has an easy to learn syntax.\n\n> 🚨 Dude is currently in Pre-Alpha. Please expect breaking changes.\n\n## Minimal web scraper\n\nThe simplest web scraper will look like this:\n\n```python\nfrom dude import select\n\n\n@select(selector="a")\ndef get_link(element):\n    return {"url": element.get_attribute("href")}\n```\n\nThe example above will get all the [hyperlink](https://en.wikipedia.org/wiki/Hyperlink#HTML) elements in a page and calls the handler function `get_link()` for each element.\nTo start scraping, just simply run in your terminal:\n\n```bash\ndude scrape --url "<url>" path/to/file.py\n```\n\nAnother option is to run from python code by calling `dude.run()` like below and running `python path/to/file.py`:\n\n```python\nfrom dude import select\n\n\n@select(selector="a")\ndef get_link(element):\n    return {"url": element.get_attribute("href")}\n\n\nif __name__ == "__main__":\n    import dude\n\n    dude.run(urls=["https://dude.ron.sh/"])\n```\n\n## Features\n\n- Simple Flask-inspired design - build a scraper with decorators.\n- Uses Playwright API - run your scraper in Chrome, Firefox and Webkit and leverage Playwright\'s powerful selector engine supporting CSS, XPath, text, regex, etc.\n- Data grouping - group related scraping data.\n- URL pattern matching - run functions on specific URLs.\n- Priority - reorder functions based on priority.\n- Setup function - enable setup steps (clicking dialogs or login).\n- Navigate function - enable navigation steps to move to other pages.\n- Custom storage - option to save data to other formats or database.\n- Async support - write async handlers.\n- BeautifulSoup4 - option to use BeautifulSoup4 instead of Playwright.\n\n## Support\n\nThis project is at a very early stage. This dude needs some love! ❤️\n\nContribute to this project by feature requests, idea discussions, reporting bugs, opening pull requests, or through Github Sponsors. Your help is highly appreciated.\n\n[![Github Sponsors](https://img.shields.io/github/sponsors/roniemartinez?label=github%20sponsors&logo=github%20sponsors&style=for-the-badge)](https://github.com/sponsors/roniemartinez)\n\n## How to use\n\n### Requirements\n\n- ✅ Any dude should know how to work with selectors (CSS or XPath).\n- ✅ This library was built on top of [Playwright](https://github.com/microsoft/playwright-python). Any dude should be at least familiar with the basics of Playwright - they also extended the selectors to support text, regular expressions, etc. See [Selectors | Playwright Python](https://playwright.dev/python/docs/selectors).\n- ✅ Python decorators... you\'ll live, dude!\n\n### Installation\n\nTo install, simply run:\n\n```bash\npip install pydude\nplaywright install\n```\n\nThe second command will install playwright binaries for Chrome, Firefox and Webkit. See https://playwright.dev/python/docs/intro#pip\n\n### Basic Usage\n\nTo use `dude`, start by importing the library.\n\n```python\nfrom dude import select\n```\n\nA basic handler function consists of the structure below. A handler function should accept 1 argument (element) and should be decorated with `@select()`. The handler should return a dictionary.\n\n```python\n@select(selector="<put-your-selector-here>")\ndef handler(element):\n    ...\n    # This dictionary can contain multiple items\n    return {"<key>": "<value-extracted-from-element>"}\n```\n\nThe example handler below extracts the text content of any element that matches the selector `css=.title`.\n\n```python\n@select(selector="css=.title")\ndef result_title(element):\n    """\n    Result title.\n    """\n    return {"title": element.text_content()}\n```\n\nTo run your handler functions, simply call `dude.run(urls=["<url-you-want-to-scrape>"])`.\n\n```python\nimport dude\n\ndude.run(urls=["https://dude.ron.sh/"])\n```\n\nIt is possible to attach a single handler to multiple selectors.\n\n```python\n@select(selector="<a-selector>")\n@select(selector="<another-selector>")\ndef handler(element):\n    return {"<key>": "<value-extracted-from-element>"}\n```\n\nCheck out the example in [examples/flat.py](examples/flat.py) and run it on your terminal using the command `python examples/flat.py`.\n\n### Advanced Usage\n\n#### Setup\n\nSome websites might require you to click on dialog buttons. You can pass `setup=True` parameter to declare the setup actions.\n\n```python\n@select(selector="text=I agree", setup=True)\ndef agree(element, page):\n    """\n    Clicks "I agree" in order to use the website.\n    """\n    with page.expect_navigation():\n        element.click()\n```\n\n#### Navigate\n\nTo navigate to another page, you can pass `navigate=True` parameter to declare the navigate actions.\n\n```python\n@select(selector="text=Next", navigate=True)\ndef next_page(element, page):\n    """\n    Clicks the Next button/link to navigate to the next page.\n    """\n    with page.expect_navigation():\n        element.click()\n```\n\n#### Grouping Results\n\nWhen scraping a page containing a list of information, for example, a Search Engine Results Page (SERP) can have URLs, titles and descriptions,\nit is important to know how data can be grouped. By default, all scraped results are grouped by `:root` which is the root document, creating a flat list.\nTo specify grouping, pass `group=<selector-for-grouping>` to `@select()` decorator.\n\nIn the example below, the results are grouped by an element with class `custom-group`. The matched selectors should be children of this element.\n\n```python\n@select(selector="css=.title", group="css=.custom-group")\ndef result_title(element):\n    return {"title": element.text_content()}\n```\n\nA more extensive example can be found at [examples/grouping.py](examples/grouping.py).\n\nThe `group` parameter has the advantage of making sure that items are in their correct group. \nTake for example the HTML below, notice that in the second `div`, there is no description.\n\n```html\n    <div class="custom-group">\n        <p class="title">Title 1</p>\n        <p class="description">Description 1</p>\n    </div>\n    <div class="custom-group">\n        <p class="title">Title 2</p>\n    </div>\n    <div class="custom-group">\n        <p class="title">Title 3</p>\n        <p class="description">Description 3</p>\n    </div>\n```\n\nWhen the group is not specified, it will result in "Description 3" being grouped with "Title 2".\n\n```json5\n[\n  {\n    "_page_number": 1,\n    // ...\n    "description": "Description 1",\n    "title": "Title 1"\n  },\n  {\n    "_page_number": 1,\n    // ...\n    "description": "Description 3",\n    "title": "Title 2"\n  },\n  {\n    "_page_number": 1,\n    // ...\n    "title": "Title 3"\n  }\n]\n```\n\nBy specifying the group in `@select(..., group="css=.custom-group")`, we will be able to get a better result.\n\n```json5\n[\n  {\n    "_page_number": 1,\n    // ...\n    "description": "Description 1",\n    "title": "Title 1"\n  },\n  {\n    "_page_number": 1,\n    // ...\n    "title": "Title 2"\n  },\n  {\n    "_page_number": 1,\n    // ...\n    "description": "Description 3",\n    "title": "Title 3"\n  }\n]\n```\n\n##### The `group` parameter simplifies how you write your code\n\n> ℹ️ The examples below are both acceptable way to write a scraper. You have the option to choose how you write the code.\n\nA common way developers write scraper can be illustrated using this example below (see [examples/single_handler.py](examples/single_handler.py) for the complete script).\nWhile this works, it can be hard to maintain.\n\n```python\n@select(selector="css=.custom-group")\ndef result_handler(element):\n    """\n    Perform all the heavy-lifting in a single handler.\n    """\n    data = {}\n\n    url = element.query_selector("a.url")\n    if url:\n        data["url"] = url.get_attribute("href")\n\n    title = element.query_selector(".title")\n    if title:\n        data["title"] = title.text_content()\n\n    description = element.query_selector(".description")\n    if description:\n        data["description"] = description.text_content()\n\n    return data\n```\n\nThis can be rewritten in a much simpler way like below (see [examples/grouping.py](examples/grouping.py) for the complete script).\nIt will require you to write 3 simple functions but is much easier to read as you don\'t have to deal with querying the child elements.\n\n```python\n@select(selector="css=a.url", group="css=.custom-group")\ndef result_url(element):\n    return {"url": element.get_attribute("href")}\n\n\n@select(selector="css=.title", group="css=.custom-group")\ndef result_title(element):\n    return {"title": element.text_content()}\n\n\n@select(selector="css=.description", group="css=.custom-group")\ndef result_description(element):\n    return {"description": element.text_content()}\n```\n\n#### URL Pattern Matching\n\nIn order to use a handler function to just specific websites, a `url` pattern parameter can be passed to `@select()`.\nThe `url` pattern parameter should be a valid regular expression. \nThe example below will only run if the URL of the current page matches `.*\\.com`.\n\n```python\n@select(selector="css=.title", url=r".*\\.com")\ndef result_title(element):\n    return {"title": element.text_content()}\n```\n\nA more extensive example can be found at [examples/url_pattern.py](examples/url_pattern.py).\n\n#### Prioritization\n\nHandlers are sorted based on the following sequence:\n\n1. URL Pattern\n2. Group\n3. Selector\n4. Priority\n\nIf all handlers have the same priority value, they will be executed based on which handler was inserted into the rule list first.\nThis arrangement depends on how handlers are defined inside python files and which python files was imported first.\nIf no priority was provided to `@select()` decorator, the value defaults to 100.\n\nThe example below makes sure that `result_description()` will be called first before `result_title()`.\n\n```python\n@select(selector="css=.title", priority=1)\ndef result_title(element):\n    return {"title": element.text_content()}\n\n\n@select(selector="css=.description", priority=0)\ndef result_description(element):\n    return {"description": element.text_content()}\n```\n\nThe priority value is most useful on Setup and Navigate handlers. As an example below, the selector `css=#pnnext` will be queried first before looking for `text=Next`.\nTake note that if `css=#pnnext` exists, then `text=Next` will not be queried anymore.\n\n```python\n@select(selector="text=Next", navigate=True)\n@select(selector="css=#pnnext", navigate=True, priority=0)\ndef next_page(element, page):\n    with page.expect_navigation():\n        element.click()\n```\n\nA more extensive example can be found at [examples/priority.py](examples/priority.py).\n\n#### Custom Storage\n\nDude currently support `json`, `yaml/yml` and `csv` formats only (the `Scraper` class only support `json`). \nHowever, this can be extended to support a custom storage or override the existing formats using the `@save()` decorator.\nThe save function should accept 2 parameters, `data` (list of dictionary of scraped data) and optional `output` (can be filename or `None`).\nTake note that the save function must return a boolean for success.\n\nThe example below prints the output to terminal using tabulate for illustration purposes only. \nYou can use the `@save()` decorator in other ways like saving the scraped data to spreadsheets, database or send it to an API.\n\n```python\nimport tabulate\n\n\n@save("table")\ndef save_table(data, output) -> bool:\n    """\n    Prints data to stdout using tabulate.\n    """\n    print(tabulate.tabulate(tabular_data=data, headers="keys", maxcolwidths=50))\n    return True\n```\n\nThe custom storage can then be called using any of these methods:\n\n1. From terminal\n    ```bash\n    dude scrape --url "<url>" path/to/file.py --format table\n    ```\n2. From python\n    ```python\n    if __name__ == "__main__":\n        import dude\n    \n        dude.run(urls=["<url>"], pages=2, format="table")\n    ```\n\nA more extensive example can be found at [examples/custom_storage.py](examples/custom_storage.py).\n\n#### Using the Scraper application class\n\nThe decorator `@select()` and the function `run()` simplifies the usage of the framework.\nIt is possible to create your own scraper application object using the example below.\n\n> 🚨 WARNING: This is not currently supported by the command line interface! \nPlease use the command `python path/to/file.py` to run the scraper application.\n\n```python\n\nfrom dude import Scraper\n\napp = Scraper()\n\n\n@app.select(selector="css=.title")\ndef result_title(element):\n    return {"title": element.text_content()}\n\n\nif __name__ == \'__main__\':\n    app.run(urls=["https://dude.ron.sh/"])\n```\n\nA more extensive example can be found at [examples/application.py](examples/application.py).\n\n#### Async Support\n\nHandler functions can be converted to async. It is not possible to mix async and sync handlers since Playwright does not support this.\nIt is however, possible to have async and sync storage handlers at the same time since this is not connected to Playwright anymore.\n\n```python\n@select(selector="css=.title")\nasync def result_title(element):\n    return {"title": await element.text_content()}\n\n@save("json")\nasync def save_json(data, output) -> bool:\n    ...\n    return True\n\n@save("xml")\ndef save_xml(data, output) -> bool:\n    # sync storage handler can be used on sync and async mode\n    ...\n    return True\n```\n\nA more extensive example can be found at [examples/async.py](examples/async.py).\n\n#### Using BeautifulSoup4\n\nOption to use BeautifulSoup4 is now available. To install, run:\n\n```bash\npip install pydude[bs4]\n```\n\nAttributes and texts from soup objects can be accessed using the examples below:\n\n```python\n@select(selector="a.url")\ndef result_url(soup):\n    return {"url": soup["href"]}\n\n\n@select(selector=".title")\ndef result_title(soup):\n    return {"title": soup.get_text()}\n```\n\nTo use BeautifulSoup4 from the command line, just add the `--bs4` argument:\n\n```bash\ndude scrape --url "<url>" --bs4 path/to/file.py\n```\n\nTo use BeautifulSoup4 from python code, just pass the parameter `parser="bs4"` to `run()` function.\n\n```python\ndude.run(urls=["https://dude.ron.sh/"], format="bs4")\n```\n\nExamples are can be found at [examples/soup.py](examples/soup.py) and [examples/async_soup.py](examples/async_soup.py).\n\n## CLI\n\n```bash\n% dude scrape -h                                                                 \nusage: dude scrape [-h] --url URL [--playwright | --bs4] [--headed] [--browser {chromium,webkit,firefox}] [--pages PAGES] [--output OUTPUT] [--format FORMAT] [--proxy-server PROXY_SERVER]\n                   [--proxy-user PROXY_USER] [--proxy-pass PROXY_PASS]\n                   PATH [PATH ...]\n\nRun the dude scraper.\n\noptions:\n  -h, --help            show this help message and exit\n\nrequired arguments:\n  PATH                  Path to python file/s containing the handler functions.\n  --url URL             Website URL to scrape. Accepts one or more url (e.g. "dude scrape --url <url1> --url <url2> ...")\n\noptional arguments:\n  --playwright          Use Playwright.\n  --bs4                 Use BeautifulSoup4.\n  --headed              Run headed browser.\n  --browser {chromium,webkit,firefox}\n                        Browser type to use.\n  --pages PAGES         Maximum number of pages to crawl before exiting (default=1). This is only valid when a navigate handler is defined.\n  --output OUTPUT       Output file. If not provided, prints into the terminal.\n  --format FORMAT       Output file format. If not provided, uses the extension of the output file or defaults to "json". Supports "json", "yaml/yml", and "csv" but can be extended using the @save()\n                        decorator.\n  --proxy-server PROXY_SERVER\n                        Proxy server.\n  --proxy-user PROXY_USER\n                        Proxy username.\n  --proxy-pass PROXY_PASS\n                        Proxy password.\n```\n\n## Why name this project "dude"?\n\n- ✅ A [Recursive acronym](https://en.wikipedia.org/wiki/Recursive_acronym) looks nice.\n- ✅ Adding "uncomplicated" (like [`ufw`](https://wiki.ubuntu.com/UncomplicatedFirewall)) into the name says it is a very simple framework. \n- ✅ Puns! I also think that if you want to do web scraping, there\'s probably some random dude around the corner who can make it very easy for you to start with it. 😊\n\n## Author\n\n[Ronie Martinez](mailto:ronmarti18@gmail.com)\n',
    'author': 'Ronie Martinez',
    'author_email': 'ronmarti18@gmail.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/roniemartinez/dude',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'extras_require': extras_require,
    'entry_points': entry_points,
    'python_requires': '>=3.7,<4.0',
}


setup(**setup_kwargs)
