"""
.. _crawl:

🕷 Python SEO Crawler / Spider
==============================

A customizable crawler to analyze SEO and content of pages and websites.

This is provided by the :func:`crawl` function which is customized for SEO and
content analysis usage, and is highly configurable. The crawler uses
`Scrapy <https://scrapy.org/>`_ so you get all the power that it provides in
terms of performance, speed, as well as flexibility and customization.

There are two main approaches to crawl:

1. **Discovery (spider mode):** You know the website to crawl, so you provide a ``url_list``
   (one or more URLs), and you want the crawler to go through the whole
   website(s) by following all available links.

2. **Pre-determined (list mode):** You have a known set of URLs that you
   want to crawl and analyze, without following links or discovering new URLs.

Discovery Crawling Approach
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The simplest way to use the function is to provide a list of one or more URLs
and the crawler will go through all of the reachable pages.

.. code-block:: python

   >>> import advertools as adv
   >>> adv.crawl('https://example.com', 'my_output_file.jl', follow_links=True)

That's it! To open the file:

.. code-block:: python

   >>> import pandas as pd
   >>> crawl_df = pd.read_json('my_output_file.jl', lines=True)


What this does:

* Check the site's robots.txt file and get the crawl rules, which means that
  your crawl will be affected by these rules and the user agent you are using.
  Check the details below on how to change settings and user agents to control
  this.
* Starting with the provided URL(s) go through all links and parse pages.
* For each URL extract the most important SEO elements.
* Save them to ``my_output_file.jl``.
* The column headers of the output file (once you import it as a DataFrame)
  would be the names of the elements (title, h1, h2, etc.).

Jsonlines is the supported output format because of its flexibility in allowing
different values for different scraped pages, and appending indepentent items
to the output files.

.. note::

    When the crawler parses pages it saves the data to the specified file by
    appending, and not overwriting. Otherwise it would have to store all the
    data in memory, which might crash your computer. A good practice is to have
    a separate ``output_file`` for every crawl with a descriptive name
    `sitename_crawl_YYYY_MM_DD.jl` for example. If you use the same file you
    will probably get duplicate data in the same file.

Extracted On-Page SEO Elements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The names of these elements become the headers (column names) of the
``output_file``.

================= =============================================================
Element           Remarks
================= =============================================================
url               The response URL that was actually crawled. This might be different from the rquested URL in case of a redirect for example. Please check the ``redirect_*`` columns for more information.
title             The <title> tag(s)
viewport          The `viewport` meta tag if available
charset           The `charset` meta tag if available
meta_desc         Meta description
canonical         The canonical tag if available
alt_href          The `href` attribute of rel=alternate tags
alt_hreflang      The language codes of the alternate links
og:*              Open Graph data
twitter:*         Twitter card data
jsonld_*          JSON-LD data if available. In case multiple snippets occur,
                  the respective column names will include a number to
                  distinguish them, `jsonld_1_{item_a}, jsonld_1_{item_b}`,
                  etc. Note that the first snippet will not contain a number,
                  so the numbering starts with "1", starting from the second
                  snippet. The same applies to OG and Twitter cards.
h1...h6           `<h1>` through `<h6>` tag(s), whichever is available
links_url         The URLs of the links on the page
links_text        The link text (anchor text)
links_nofollow    Boolean, whether or not the link is a nofllow link. Note that
                  this only tells if the link itself contains a rel="nofollow"
                  attribute. The page might indicate "nofollow" using meta
                  robots or X-Robots-Tag, which you have to check separately.
nav_links_text    The anchor text of all links in the `<nav>` tag if
                  available
nav_links_url     The links in the `<nav>` tag if available
header_links_text The anchor text of all links in the `<header>` tag if
                  available
header_links_url  The links in the `<header>` tag if available
footer_links_text The anchor text of all links in the `<footer>` tag if
                  available
footer_links_url  The links in the `<footer>` tag if available
body_text         The text in the <p>, <span>, and <li> tags within <body>
size              The page size in bytes
download_latency  The amount of time it took to get the page HTML, in seconds.
download_timout   The amount of time (in secs) that the downloader will wait
                  before timing out. Defaults to 180.
redirect_times    The number of times the pages was redirected if available
redirect_ttl      The default maximum number of redirects the crawler allows
redirect_urls     The chain of URLs from the requested URL to the one actually
                  fetched
redirect_reasons  The type of redirection(s) 301, 302, etc.
depth             The depth of the current URL, relative to the first URLs
                  where crawling started. The first pages to be crawled have a
                  depth of zero, pages linked from there, a depth of one, etc.
status            Response status code (200, 404, etc.)
img_*             All available ``<img>`` tag attributes. 'alt', 'crossorigin',
                  'height', 'ismap', 'loading', 'longdesc', 'referrerpolicy',
                  'sizes', 'src', 'srcset', 'usemap',  and 'width' (excluding
                  global HTML attributes like ``style`` and ``draggable``)
ip_address        IP address
crawl_time        Date and time the page was crawled
resp_headers_*    All available response headers (last modified, server, etc.)
request_headers_* All available request headers (user-agent, encoding, etc.)
================= =============================================================

.. note::

    All elements that may appear multiple times on a page (like heading tags,
    or images, for example), will be joined with two "@" signs `@@`. For
    example, **"first H2 tag@@second H2 tag@@third tag"** and so on.
    Once you open the file, you simply have to split by `@@` to get the
    elements as a list.

Here is a sample file of a crawl of this site (output truncated for
readability):

.. code-block:: python

    >>> import pandas as pd
    >>> site_crawl = pd.read_json('path/to/file.jl', lines=True)
    >>> site_crawl.head()
                                   url                           title                       meta_desc                              h1                              h2                              h3                        body_text  size  download_timeout              download_slot  download_latency  redirect_times  redirect_ttl                   redirect_urls redirect_reasons  depth  status                      links_href                      links_text                         img_src                         img_alt    ip_address           crawl_time              resp_headers_date resp_headers_content-type     resp_headers_last-modified resp_headers_vary    resp_headers_x-ms-request-id resp_headers_x-ms-version resp_headers_x-ms-lease-status resp_headers_x-ms-blob-type resp_headers_access-control-allow-origin   resp_headers_x-served resp_headers_x-backend resp_headers_x-rtd-project resp_headers_x-rtd-version         resp_headers_x-rtd-path  resp_headers_x-rtd-domain resp_headers_x-rtd-version-method resp_headers_x-rtd-project-method resp_headers_strict-transport-security resp_headers_cf-cache-status  resp_headers_age           resp_headers_expires resp_headers_cache-control          resp_headers_expect-ct resp_headers_server   resp_headers_cf-ray      resp_headers_cf-request-id          request_headers_accept request_headers_accept-language      request_headers_user-agent request_headers_accept-encoding          request_headers_cookie
    0   https://advertools.readthedocs            advertools —  Python  Get productive as an online ma  advertools@@Indices and tables  Online marketing productivity                              NaN   Generate keywords for SEM camp   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN  https://advertools.readthedocs            [302]    NaN     NaN  #@@readme.html@@advertools.kw_  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:35  Thu, 21 May 2020 10:39:35 GMT                 text/html  Wed, 20 May 2020 12:26:23 GMT   Accept-Encoding  720a8581-501e-0043-01a2-2e77d2                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007c                 advertools                     master  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:35 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596daca7dbaa7e9e-BUD  02d86a3cea00007e9edb0cf2000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    1   https://advertools.readthedocs            advertools —  Python                             NaN                      advertools         Change Log - advertools  0.9.1 (2020-05-19)@@0.9.0 (202   Ability to specify robots.txt    NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  index.html@@readme.html@@adver  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:36  Thu, 21 May 2020 10:39:35 GMT                 text/html  Wed, 20 May 2020 12:26:23 GMT   Accept-Encoding  4f7bea3b-701e-0039-3f44-2f1d9f                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007h                 advertools                     master  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:35 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596daca9bcab7e9e-BUD  02d86a3e0e00007e9edb0d72000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    2   https://advertools.readthedocs            advertools —  Python  Get productive as an online ma  advertools@@Indices and tables  Online marketing productivity                              NaN   Generate keywords for SEM camp   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  #@@readme.html@@advertools.kw_  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:36  Thu, 21 May 2020 10:39:35 GMT                 text/html  Wed, 20 May 2020 12:26:36 GMT   Accept-Encoding  98b729fa-e01e-00bf-24c3-2e494d                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007c                 advertools                     latest  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:35 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596daca9bf26d423-BUD  02d86a3e150000d423322742000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    3   https://advertools.readthedocs    advertools package —  Python                             NaN              advertools package     Submodules@@Module contents                             NaN   Top-level package for advertoo   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  index.html@@readme.html@@adver  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:36  Thu, 21 May 2020 10:39:35 GMT                 text/html  Wed, 20 May 2020 12:26:25 GMT   Accept-Encoding  7a28ef3b-801e-00c2-24c3-2ed585                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web000079                 advertools                     master  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:35 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596daca9bddb7ec2-BUD  02d86a3e1300007ec2a808a2000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    4   https://advertools.readthedocs   Python Module Index —  Python                             NaN             Python Module Index                             NaN                             NaN            © Copyright 2020, Eli   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  index.html@@readme.html@@adver  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@               _static/minus.png                               -  104.17.32.82  2020-05-21 10:39:36  Thu, 21 May 2020 10:39:35 GMT                 text/html  Wed, 20 May 2020 12:26:23 GMT   Accept-Encoding  75911c9e-201e-00e6-34c3-2e4ccb                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007g                 advertools                     master  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:35 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596daca9b91fd437-BUD  02d86a3e140000d437b81532000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    66  https://advertools.readthedocs  advertools.url_builders —  Pyt                             NaN  Source code for advertools.url                             NaN                             NaN            © Copyright 2020, Eli   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  ../../index.html@@../../readme  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:39  Thu, 21 May 2020 10:39:38 GMT                 text/html  Wed, 20 May 2020 12:26:36 GMT   Accept-Encoding  d99f2368-c01e-006f-18c3-2ef5ef                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007a                 advertools                     latest  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:38 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596dacbbb8afd437-BUD  02d86a494f0000d437b828b2000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    67  https://advertools.readthedocs  advertools.kw_generate —  Pyth                             NaN  Source code for advertools.kw_                             NaN                             NaN            © Copyright 2020, Eli   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  ../../index.html@@../../readme  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:39  Thu, 21 May 2020 10:39:39 GMT                 text/html  Wed, 20 May 2020 12:26:36 GMT   Accept-Encoding  85855c48-c01e-00ce-13c3-2e3b74                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007g                 advertools                     latest  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:39 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596dacbd980bd423-BUD  02d86a4a7f0000d423323b42000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    68  https://advertools.readthedocs  advertools.ad_from_string —  P                             NaN  Source code for advertools.ad_                             NaN                             NaN            © Copyright 2020, Eli   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  ../../index.html@@../../readme  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:39  Thu, 21 May 2020 10:39:39 GMT                 text/html  Wed, 20 May 2020 12:26:36 GMT   Accept-Encoding  b0aef497-801e-004a-1647-2f6d5c                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007k                 advertools                     latest  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:39 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596dacbd980cd423-BUD  02d86a4a7f0000d423209db2000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    69  https://advertools.readthedocs  advertools.ad_create —  Python                             NaN  Source code for advertools.ad_                             NaN                             NaN            © Copyright 2020, Eli   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  ../../index.html@@../../readme  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:39  Thu, 21 May 2020 10:39:39 GMT                 text/html  Wed, 20 May 2020 12:26:36 GMT   Accept-Encoding  9dfdd38a-101e-00a1-7ec3-2e93a0                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web00007c                 advertools                     latest  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:39 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596dacbd99847ec2-BUD  02d86a4a7f00007ec2a811f2000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004
    70  https://advertools.readthedocs      advertools.emoji —  Python                             NaN  Source code for advertools.emo                             NaN                             NaN            © Copyright 2020, Eli   NaN               NaN  advertools.readthedocs.io               NaN             NaN           NaN                             NaN              NaN    NaN     NaN  ../../index.html@@../../readme  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                             NaN                             NaN  104.17.32.82  2020-05-21 10:39:40  Thu, 21 May 2020 10:39:39 GMT                 text/html  Wed, 20 May 2020 12:26:36 GMT   Accept-Encoding  2ad504a1-101e-000b-03c3-2e454f                2009-09-19                       unlocked                   BlockBlob                                        *  Nginx-Proxito-Sendfile              web000079                 advertools                     latest  /proxito/media/html/advertools  advertools.readthedocs.io                              path                         subdomain         max-age=31536000; includeSubDo                          HIT               NaN  Thu, 21 May 2020 11:39:39 GMT       public, max-age=3600  max-age=604800, report-uri="ht          cloudflare  596dacbd9fb97e9e-BUD  02d86a4a7f00007e9edb13a2000000  text/html,application/xhtml+xm                              en  Mozilla/5.0 (Windows NT 10.0;                    gzip, deflate  __cfduid=d76b68d148ddec1efd004

Pre-Determined Crawling Approach (List Mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sometimes you might have a fixed set of URLs for which you want to scrape and
analyze SEO or content performance. Some ideas:

SERP Data
---------
Let's say you just ran :ref:`serp_goog <serp>` and got a bunch of top-ranking
pages that you would like to analyze, and see how that relates to their SERP
ranking.

You simply provide the ``url_list`` parameter and again specify the
``output_file``. This will only crawl the specified URLs, and will not follow
any links.

Now you have the SERP DataFrame, as well as the crawl output file. All you have
to do is to merge them by the URL columns, and end up with a richer dataset

News Articles
-------------
You want to follow the latest news of a certain publication, and you extract
their latest news URLs from their news sitemap using
:ref:`sitemap_to_df <sitemaps>` . You provide those URLs and crawl them only.

Google Analytics / Google Search Console
----------------------------------------
Since they provide reports for URLs, you can also combine them with the ones
crawled and end up with a better perspective. You might be interested in
knowing more about high bounce-rate pages, pages that convert well, pages that
get less traffic than you think they should and so on. You can simply export
those URLs and crawl them.

Any tool that has data about a set of URLs can be used.

Again running the function is as simple as providing a list of URLs, as well as
a filepath where you want the result saved.

.. code-block:: python

    >>> adv.crawl(url_list, 'output_file.jl', follow_links=False)

The difference between the two approaches, is the simple parameter
``follow_links``. If you keep it as ``False`` (the default), the crawler
will only go through the provided URLs. Otherwise, it will discover pages by
following links on pages that it crawls. So how do you make sure that the
crawler doesn't try to crawl the whole web when ``follow_links`` is `True`?
The ``allowed_domains`` parameter gives you the ability to control this,
although it is an optional parameter. If you don't specify it, then it will
default to only the domains in the ``url_list`` and their sub-domains if any.
It's important to note that you have to set this parameter if you want to only
crawl certain sub-domains.

Custom Extraction with CSS and XPath Selectors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The above approaches are generic, and are useful for exploratory SEO audits
and the output is helpful for most cases.

But what if you want to extract special elements that are not included in the
default output? This is extremely important, as there are key elements on pages
that you need to additionally extract and analyze. Some examples might be tags,
prices, social media shares, product price or availability, comments, and
pretty much any element on a page that might be of interest to you.

For this you can use two special parameters for CSS and/or XPath selectors. You
simply provide a dictionary `{'name_1': 'selector_1', 'name_2': 'selector_2'}`
where the keys become the column names, and the values (selectors) will be
used to extract the required elements.

I mostly rely on `SlectorGadget <https://selectorgadget.com/>`_ which is a
really great tool for getting the CSS/XPath selecotrs of required elements.
In some pages it can get really tricky to figure that out. Other resources for
learning more about selectors:

* `Scrapy's documentaion for selectors <https://docs.scrapy.org/en/latest/topics/selectors.html>`_
* `CSS Selector Reference on W3C <https://www.w3schools.com/cssref/css_selectors.asp>`_
* `XPath tutorial on W3C <https://www.w3schools.com/xml/xpath_intro.asp>`_

Once you have determined the elements that you want to extract and figured out
what their names are going to be, you simply pass them as arguments to
``css_selectors`` and/or ``xpath_selectors`` as dictionaries, as decribed
above.

Let's say you want to extract the links in the sidebar of this page. By default
you would get all the links from the page, but you want to put those in the
sidebar in a separate column. It seems that the CSS selector for them is
`.toctree-l1 .internal`, and the XPath equivalent is
`//*[contains(concat( " ", @class, " " ), concat( " ", "toctree-l1", " " ))]//*[contains(concat( " ", @class, " " ), concat( " ", "internal", " " ))]`.
Note that this selects the *element* (the whole link object), which is not
typically what you might be interested in.

So with CSS you need to append `::text` or `::attr(href)` if you want the text
of the links or the `href` attribute respectively. Similarly with XPath, you
will need to append `/text()` or `/@href` to the selector to get the same.

>>> adv.crawl('https://advertools.readthedocs.io/en/master/advertools.spider.html',
...           'output_file.jl',
...           css_selectors={'sidebar_links': '.toctree-l1 .internal::text',
...                          'sidebar_links_url': '.toctree-l1 .internal::attr(href)'})

Or, instead of ``css_selectors`` you can add a similar dictionary for the
``xpath_selectors`` argument:

>>> adv.crawl('https://advertools.readthedocs.io/en/master/advertools.spider.html',
...           'output_file.jl',
...           xpath_selectors={'sidebar_links': '//*[contains(concat( " ", @class, " " ), concat( " ", "toctree-l1", " " ))]//*[contains(concat( " ", @class, " " ), concat( " ", "internal", " " ))]/text()',
...                            'sidebar_links_url': '//*[contains(concat( " ", @class, " " ), concat( " ", "toctree-l1", " " ))]//*[contains(concat( " ", @class, " " ), concat( " ", "internal", " " ))]/@href'})

Customizing the Crawling Behavior while Following Links
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you set ``follow_links=True`` you might want to restrict this option for a
more customized crawl. This means you can decide whether to include or exclude
certain links based on certain conditions based on URL parameters and/or URL
regex patterns.

URL Query Parameters
--------------------

Two options are available for this:

1. ``exclude_url_params``: By providing a list of URL parameters, any link that
contains any of the parameters will not be followed. For example if you set
this option to ['price', 'country'], and the page currently being crawled
contains three links:

  * **/shoes/model_a?price=10&country=us**: Contains both parameters, will not be followed.
  * **/shoes/model_a?price=10**: Contains one of the parameters, will not be followed.
  * **/shoes/model_b?color=black**: Contains "color" which was not listed, will be followed

To make this efficient, and in case you want to skip all links that contain any
parameter, you can set this option to ``True``, and any link that contains any
URL parameter will not be followed ``exclude_url_params=True``


2. ``include_url_params``: Similarly, you can choose the parameters that links
should contain in order for them to be followed. If a link contains any of the
listed parameters, that link will be followed. Even though this option is
straightforward, it might give you unexpected results. If you set the
parameters to include as ['price'] for example, and you start crawling from a
page that doesn't have any link containing that parameter, crawling will stop.
Yet, the website might have many links with that parameter. Please keep this in
mind, and remember that reasoning about the exclude option is easier than the
include option for this reason/example.

URL Regex Patterns
------------------

You might want even more granular control over which links to follow, and might
be interested in other URL properties than their query parameters. You have two
simple options to include/exclude links based on whether they match a certain
regex pattern.

1. ``exclude_url_regex``: Enter a regex, and the links will be checked if they
match it. If they do, they will not be followed, if not they will.

2. ``include_url_regex``: This is similar but tells the crawler which links to
follow, based on whether or not they match the regex. This option also has the
same potentially tricky behavior like the ``include_url_params`` option.

Here is a simple example showing how you might control following links using
all four options:

.. code-block::

    import advertools as adv
    adv.crawl('https://example.com', 'output_file.jl', follow_links=True,

              # don't follow links containing any of these parameters:
              exclude_url_params=['price', 'region'],
              # follow links containing any of these parameters:
              include_url_params=['source'],
              # don't follow links that contain the pattern "/fr/" or "/de/":
              exclude_url_regex='/fr/|/de/',
              # follow links that contain the pattern "/shop/":
              include_url_regex='/shop/'
              )

Spider Custom Settings and Additional Functionality
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In addition to what you can control regarding the items you can extract, you
can also customize the behaviour of the spider and set rules for crawling so
you can control it even further.

This is provided by the ``custom_settings`` parameter. It is optional, and
takes a dictionary of settings and their values. Scrapy provides a very large
number of settings, and they are all available through this parameter
(assuming some conditions for some of the settings).

Here are some examples that you might find interesting:

* `CONCURRENT_REQUESTS_PER_DOMAIN` Defaults to 8, and controls the number of
  simultaneous requests to be performed for each domain. You might want to
  lower this if you don't want to put too much pressure on the website's
  server, and you probably don't want to get blocked!
* `DEFAULT_REQUEST_HEADERS` You can change this if you need to.
* `DEPTH_LIMIT` How deep your crawl will be allowed. The default has no limit.
* `DOWNLOAD_DELAY` Similar to the first option. Controls the amount of time in
  seconds for the crawler to wait between consecutive pages of the same
  website. It can also take fractions of a second (0.4, 0.75, etc.)
* `LOG_FILE` If you want to save your crawl logs to a file, which is strongly
  recommended, you can provide a path to it here.
* `USER_AGENT` If you want to identify yourself differently while crawling.
  This is affected by the robots.txt rules, so you would be potentially
  allowed/disallowed from certain pages based on your user-agent.
* `CLOSESPIDER_ERRORCOUNT`, `CLOSESPIDER_ITEMCOUNT`, `CLOSESPIDER_PAGECOUNT`,
  `CLOSESPIDER_TIMEOUT` Stop crawling after that many errors, items, pages, or
  seconds. These can be very useful to limit your crawling in certain cases.
  I particularly like to use `CLOSESPIDER_PAGECOUNT` when exploring a new
  website, and also to make sure that my selectors are working as expected. So
  for your first few crawls you might set this to five hundred for example and
  explore the crawled pages. Then when you are confident things are working
  fine, you can remove this restriction. `CLOSESPIDER_ERRORCOUNT` can also be
  very useful while exploring, just in case you get unexpected errors.

The next page contains a number of :ref:`strategies and recipes for crawling <crawl_strategies>`
with code examples and explanations.

**Usage**

A very simple dictionary to be added to your function call:

>>> adv.crawl('http://exmaple.com', 'outpuf_file.jl',
...           custom_settings={'CLOSESPIDER_PAGECOUNT': 100,
...                            'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
...                            'USER_AGENT': 'custom-user-agent'})

Please refer to the `spider settings documentation <https://docs.scrapy.org/en/latest/topics/settings.html>`_
for the full details.

"""
import datetime
import json
import logging
import platform
import re
import subprocess
from functools import reduce
from urllib.parse import parse_qs, urlparse, urlsplit

import pandas as pd
import scrapy
import scrapy.logformatter as formatter
from scrapy import Request
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import Spider
from scrapy.utils.response import get_base_url

import advertools as adv

if int(pd.__version__[0]) >= 1:
    from pandas import json_normalize
else:
    from pandas.io.json import json_normalize

from advertools import __version__ as adv_version

spider_path = adv.__path__[0] + '/spider.py'

user_agent = f'advertools/{adv_version}'

BODY_TEXT_SELECTOR = '//body//span//text() | //body//p//text() | //body//li//text()'

_IMG_ATTRS = {'alt', 'crossorigin', 'height', 'ismap', 'loading', 'longdesc',
              'referrerpolicy', 'sizes', 'src', 'srcset', 'usemap', 'width'}


def _crawl_or_not(url,
                  exclude_url_params=None,
                  include_url_params=None,
                  exclude_url_regex=None,
                  include_url_regex=None):
    qs = parse_qs(urlsplit(url).query)
    supplied_conditions = []
    if exclude_url_params is not None:
        if exclude_url_params is True and qs:
            return False
        if exclude_url_params is True and not qs:
            pass
        else:
            exclude_params_in_url = not bool(set(exclude_url_params).intersection(qs))
            supplied_conditions.append(exclude_params_in_url)

    if include_url_params is not None:
        include_params_in_url = bool(set(include_url_params).intersection(qs))
        supplied_conditions.append(include_params_in_url)

    if exclude_url_regex is not None:
        exclude_pattern_matched = not bool(re.findall(exclude_url_regex, url))
        supplied_conditions.append(exclude_pattern_matched)

    if include_url_regex is not None:
        include_pattern_matched = bool(re.findall(include_url_regex, url))
        supplied_conditions.append(include_pattern_matched)
    return all(supplied_conditions)

def _extract_images(response):
    page_has_images = response.xpath('//img')
    if page_has_images:
        img_attributes = reduce(set.union,
                                [set(x.attrib.keys())
                                 for x in page_has_images])
        img_attributes = img_attributes.intersection(_IMG_ATTRS)
        d = dict()
        for im_attr in img_attributes:
            d['img_' + im_attr] = '@@'.join([im.attrib.get(im_attr) or ''
                                            for im in response.xpath('//img')])
        return d
    return {}


def get_max_cmd_len():
    system = platform.system()
    cmd_dict = {'Windows': 7000, 'Linux': 100000, 'Darwin': 100000}
    if system in cmd_dict:
        return cmd_dict[system]
    return 6000


MAX_CMD_LENGTH = get_max_cmd_len()

formatter.SCRAPEDMSG = "Scraped from %(src)s"
formatter.DROPPEDMSG = "Dropped: %(exception)s"
formatter.DOWNLOADERRORMSG_LONG = "Error downloading %(request)s"


class MyLinkExtractor(LinkExtractor):
    def extract_links(self, response):
        base_url = get_base_url(response)
        if self.restrict_xpaths:
            docs = [
                subdoc
                for x in self.restrict_xpaths
                for subdoc in response.xpath(x)
            ]
        else:
            docs = [response.selector]
        all_links = []
        for doc in docs:
            links = self._extract_links(doc, response.url, response.encoding,
                                        base_url)
            all_links.extend(self._process_links(links))
        return all_links


le = MyLinkExtractor(unique=False)
le_nav = MyLinkExtractor(unique=False, restrict_xpaths='//nav')
le_header = MyLinkExtractor(unique=False, restrict_xpaths='//header')
le_footer = MyLinkExtractor(unique=False, restrict_xpaths='//footer')

crawl_headers = {
    'url',
    'title',
    'meta_desc',
    'viewport',
    'charset',
    'alt_href',
    'alt_hreflang',
    'h1',
    'h2',
    'h3',
    'h4',
    'h5',
    'h6',
    'canonical',
    'body_text',
    'size',
    'download_timeout',
    'download_slot',
    'download_latency',
    'redirect_times',
    'redirect_ttl',
    'redirect_urls',
    'redirect_reasons',
    'depth',
    'status',
    'links_url',
    'links_text',
    'links_nofollow',
    'img_src',
    'img_alt',
    'ip_address',
    'crawl_time',
    'blocked_by_robotstxt',
    'jsonld_errors',
    'request_headers_accept',
    'request_headers_accept-language',
    'request_headers_user-agent',
    'request_headers_accept-encoding',
    'request_headers_cookie',
}


def _split_long_urllist(url_list, max_len=MAX_CMD_LENGTH):
    """Split url_list if their total length is greater than MAX_CMD_LENGTH."""
    split_list = [[]]

    for u in url_list:
        temp_len = sum(len(temp_u) for temp_u in split_list[-1])
        if (temp_len < max_len) and (temp_len + len(u) < max_len):
            split_list[-1].append(u)
        else:
            split_list.append([u])
    return split_list


def _numbered_duplicates(items):
    """Append a number to all duplicated items starting at 1.

    ['og:site', 'og:image', 'og:image', 'og:type', 'og:image']
    becomes:
    ['og:site', 'og:image_1', 'og:image_2', 'og:type', 'og:image_3']
    """
    item_count = dict.fromkeys(items, 0)
    numbered_items = []
    for item in items:
        numbered_items.append(item + '_' + str(item_count[item]))
        item_count[item] += 1
    for i, num_item in enumerate(numbered_items):
        split_number = num_item.rsplit('_', maxsplit=1)
        if split_number[1] == '0':
            numbered_items[i] = split_number[0]
    return numbered_items


def _json_to_dict(jsonobj, i=None):
    try:
        df = json_normalize(jsonobj)
        if i:
            df = df.add_prefix('jsonld_{}_'.format(i))
        else:
            df = df.add_prefix('jsonld_')
        return dict(zip(df.columns, df.values[0]))
    except Exception as e:
        logger = logging.getLogger(__name__)
        logger.error(msg=str(e))
        return {}


tags_xpaths = {
    'title': '//title/text()',
    'meta_desc': '//meta[@name="description"]/@content',
    'viewport': '//meta[@name="viewport"]/@content',
    'charset': '//meta[@charset]/@charset',
    'h1': '//h1',
    'h2': '//h2',
    'h3': '//h3',
    'h4': '//h4',
    'h5': '//h5',
    'h6': '//h6',
    'canonical': '//link[@rel="canonical"]/@href',
    'alt_href': '//link[@rel="alternate"]/@href',
    'alt_hreflang': '//link[@rel="alternate"]/@hreflang',
}


def _extract_content(resp, **tags_xpaths):
    d = {}
    for tag, xpath in tags_xpaths.items():
        if not tag.startswith('h'):
            value = '@@'.join(resp.xpath(xpath).getall())
            if value:
                d.update({tag: value})
        else:
            value = '@@'.join([h.root.text_content()
                               for h in resp.xpath(xpath)])
            if value:
                d.update({tag: value})
    return d


class SEOSitemapSpider(Spider):
    name = 'seo_spider'
    follow_links = False
    skip_url_params = False
    css_selectors = {}
    xpath_selectors = {}
    custom_settings = {
        'USER_AGENT': user_agent,
        'ROBOTSTXT_OBEY': True,
        'HTTPERROR_ALLOW_ALL': True,
    }


    def __init__(self, url_list, follow_links=False,
                 allowed_domains=None,
                 exclude_url_params=None,
                 include_url_params=None,
                 exclude_url_regex=None,
                 include_url_regex=None,
                 css_selectors=None,
                 xpath_selectors=None, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.start_urls = json.loads(json.dumps(url_list.split(',')))
        self.allowed_domains = json.loads(json.dumps(allowed_domains.split(',')))
        self.follow_links = eval(json.loads(json.dumps(follow_links)))
        self.exclude_url_params = eval(json.loads(json.dumps(exclude_url_params)))
        self.include_url_params = eval(json.loads(json.dumps(include_url_params)))
        self.exclude_url_regex = str(json.loads(json.dumps(exclude_url_regex)))
        if self.exclude_url_regex == "None":
            self.exclude_url_regex = None
        self.include_url_regex = str(json.loads(json.dumps(include_url_regex)))
        if self.include_url_regex == "None":
            self.include_url_regex = None
        self.css_selectors = eval(json.loads(json.dumps(css_selectors)))
        self.xpath_selectors = eval(json.loads(json.dumps(xpath_selectors)))

    def start_requests(self):
        for url in self.start_urls:
            try:
                yield Request(url, callback=self.parse, errback=self.errback)
            except Exception as e:
                self.logger.error(repr(e))

    def errback(self, failure):
        if not failure.check(scrapy.exceptions.IgnoreRequest):
            self.logger.error(repr(failure))
            yield {'url': failure.request.url,
                   'crawl_time': datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S'),
                   'errors': repr(failure)}

    def parse(self, response):
        links = le.extract_links(response)
        nav_links = le_nav.extract_links(response)
        header_links = le_header.extract_links(response)
        footer_links = le_footer.extract_links(response)
        images = _extract_images(response)

        if links:
            parsed_links = dict(
                links_url='@@'.join(link.url for link in links),
                links_text='@@'.join(link.text for link in links),
                links_nofollow='@@'.join(str(link.nofollow) for link in links),
            )
        else:
            parsed_links = {}
        if nav_links:
            parsed_nav_links = dict(
                nav_links_url='@@'.join(link.url for link in nav_links),
                nav_links_text='@@'.join(link.text for link in nav_links),
                nav_links_nofollow='@@'.join(str(link.nofollow)
                                             for link in nav_links),
            )
        else:
            parsed_nav_links = {}
        if header_links:
            parsed_header_links = dict(
                header_links_url='@@'.join(link.url
                                           for link in header_links),
                header_links_text='@@'.join(link.text
                                            for link in header_links),
                header_links_nofollow='@@'.join(str(link.nofollow)
                                                for link in header_links),
            )
        else:
            parsed_header_links = {}
        if footer_links:
            parsed_footer_links = dict(
                footer_links_url='@@'.join(link.url for link in footer_links),
                footer_links_text='@@'.join(link.text
                                            for link in footer_links),
                footer_links_nofollow='@@'.join(str(link.nofollow)
                                                for link in footer_links),
            )
        else:
            parsed_footer_links = {}
        if self.css_selectors:
            css_selectors = {key: '@@'.join(response.css('{}'.format(val)).getall())
                             for key, val in self.css_selectors.items()}
            css_selectors = {k: v for k, v in css_selectors.items() if v}
        else:
            css_selectors = {}

        if self.xpath_selectors:
            xpath_selectors = {key: '@@'.join(response.xpath('{}'.format(val)).getall())
                               for key, val in self.xpath_selectors.items()}
            xpath_selectors = {k: v for k, v in xpath_selectors.items() if v}
        else:
            xpath_selectors = {}
        canonical = {'canonical': '@@'.join(response.css('link[rel="canonical"]::attr(href)').getall())}
        canonical = canonical if canonical.get('canonical') else {}
        alt_href = {'alt_href': '@@'.join(response.css('link[rel=alternate]::attr(href)').getall())}
        alt_href = alt_href if alt_href.get('alt_href') else {}
        alt_hreflang = {'alt_hreflang': '@@'.join(response.css('link[rel=alternate]::attr(hreflang)').getall())}
        alt_hreflang = alt_hreflang if alt_hreflang.get('alt_hreflang') else {}
        og_props = response.xpath('//meta[starts-with(@property, "og:")]/@property').getall()
        og_content = response.xpath('//meta[starts-with(@property, "og:")]/@content').getall()
        if og_props and og_content:
            og_props = _numbered_duplicates(og_props)
            open_graph = dict(zip(og_props, og_content))
        else:
            open_graph = {}
        twtr_names = response.xpath('//meta[starts-with(@name, "twitter:")]/@name').getall()
        twtr_content = response.xpath('//meta[starts-with(@name, "twitter:")]/@content').getall()
        if twtr_names and twtr_content:
            twtr_card = dict(zip(twtr_names, twtr_content))
        else:
            twtr_card = {}
        try:
            ld = [json.loads(s.replace('\r', '').replace('\n', ' ')) for s in
                  response.css('script[type="application/ld+json"]::text').getall()]
            if not ld:
                jsonld = {}
            else:
                if len(ld) == 1:
                    if isinstance(ld, list):
                        ld = ld[0]
                    jsonld = _json_to_dict(ld)
                else:
                    ld_norm = [_json_to_dict(x, i) for i, x in enumerate(ld)]
                    jsonld = {}
                    for norm in ld_norm:
                        jsonld.update(**norm)
        except Exception as e:
            jsonld = {'jsonld_errors': str(e)}
            self.logger.exception(' '.join([str(e), str(response.status),
                                            response.url]))
        page_content = _extract_content(response, **tags_xpaths)

        yield dict(
            url=response.request.url,
            **page_content,
            **open_graph,
            **twtr_card,
            **jsonld,
            body_text=' '.join(response.xpath(BODY_TEXT_SELECTOR).extract()),
            size=len(response.body),
            **css_selectors,
            **xpath_selectors,
            **{k: '@@'.join(str(val) for val in v) if isinstance(v, list)
               else v for k, v in response.meta.items()},
            status=response.status,
            **parsed_links,
            **parsed_nav_links,
            **parsed_header_links,
            **parsed_footer_links,
            **images,
            ip_address=str(response.ip_address),
            crawl_time=datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S'),
            **{'resp_headers_' + k: v
               for k, v in response.headers.to_unicode_dict().items()},
            **{'request_headers_' + k: v
               for k, v in response.request.headers.to_unicode_dict().items()},
        )
        if self.follow_links:
            next_pages = [link.url for link in links]
            if next_pages:
                for page in next_pages:
                    cond = _crawl_or_not(
                        page,
                        exclude_url_params=self.exclude_url_params,
                        include_url_params=self.include_url_params,
                        exclude_url_regex=self.exclude_url_regex,
                        include_url_regex=self.include_url_regex)
                    if cond:
                        yield Request(page, callback=self.parse,
                                      errback=self.errback)
                    # if self.skip_url_params and urlparse(page).query:
                    #     continue


def crawl(url_list, output_file, follow_links=False,
          allowed_domains=None,
          exclude_url_params=None,
          include_url_params=None,
          exclude_url_regex=None,
          include_url_regex=None,
          css_selectors=None, xpath_selectors=None, custom_settings=None,
          ):
    """
    Crawl a website's URLs based on the given :attr:`url_list`

    :param url,list url_list: One or more URLs to crawl. If ``follow_links``
                          is True, the crawler will start with these URLs and
                          follow all links on pages recursively.
    :param str output_file: The path to the output of the crawl. Jsonlines only
                            is supported to allow for dynamic values. Make sure
                            your file ends with ".jl", e.g. `output_file.jl`.
    :param bool follow_links: Defaults to False. Whether or not to follow links
                              on crawled pages.
    :param list,bool exclude_url_params: A list of URL parameters to exclude
                                         while following links. If a link
                                         contains any of those parameters,
                                         don't follow it. Setting it to
                                         ``True`` will exclude links
                                         containing any parameter.
    :param list include_url_params: A list of URL parameters to include while
                                    following links. If a link contains any of
                                    those parameters, follow it. Having the
                                    same parmeters to include and exclude
                                    raises an error.
    :param str exclude_url_regex: A regular expression of a URL pattern to
                                  exclude while following links. If a link
                                  matches the regex don't follow it.
    :param str include_url_regex: A regular expression of a URL pattern to
                                  include while following links. If a link
                                  matches the regex follow it.
    :param dict css_selectors: A dictionary mapping names to CSS selectors. The
                               names will become column headers, and the
                               selectors will be used to extract the required
                               data/content.
    :param dict xpath_selectors: A dictionary mapping names to XPath selectors.
                                 The names will become column headers, and the
                                 selectors will be used to extract the required
                                 data/content.
    :param dict custom_settings: A dictionary of optional custom settings that
                                 you might want to add to the spider's
                                 functionality. There are over 170 settings for
                                 all kinds of options. For details please
                                 refer to the `spider settings <https://docs.scrapy.org/en/latest/topics/settings.html>`_
                                 documentation.
    :param list allowed_domains: (optional) A list of the allowed domains to
                                 crawl. This ensures that the crawler does not
                                 attempt to crawl the whole web. If not
                                 specified, it defaults to the domains of the
                                 URLs provided in ``url_list`` and all their 
                                 sub-domains. You can also specify a list of
                                 sub-domains, if you want to only crawl those.
    :Examples:

    Crawl a website and let the crawler discover as many pages as available

    >>> import advertools as adv
    >>> adv.crawl('http://example.com', 'output_file.jl', follow_links=True)
    >>> import pandas as pd
    >>> crawl_df = pd.read_json('output_file.jl', lines=True)

    Crawl a known set of pages (on a single or multiple sites) without
    following links (just crawl the specified pages) or "list mode":

    >>> adv.crawl(['http://exmaple.com/product', 'http://exmaple.com/product2',
    ...            'https://anotherexample.com', 'https://anotherexmaple.com/hello'],
    ...            'output_file.jl', follow_links=False)

    Crawl a website, and in addition to standard SEO elements, also get the
    required CSS selectors.
    Here we will get three additional columns `price`, `author`, and
    `author_url`. Note that you need to specify if you want the text attribute
    or the `href` attribute if you are working with links (and all other
    selectors).

    >>> adv.crawl('http://example.com', 'output_file.jl',
    ...           css_selectors={'price': '.a-color-price::text',
    ...                          'author': '.contributorNameID::text',
    ...                          'author_url': '.contributorNameID::attr(href)'})
    """
    if isinstance(url_list, str):
        url_list = [url_list]
    if isinstance(allowed_domains, str):
        allowed_domains = [allowed_domains]
    if output_file.rsplit('.')[-1] != 'jl':
        raise ValueError("Please make sure your output_file ends with '.jl'.\n"
                         "For example:\n"
                         "{}.jl".format(output_file.rsplit('.', maxsplit=1)[0]))
    if (xpath_selectors is not None) and (css_selectors is not None):
        css_xpath = set(xpath_selectors).intersection(css_selectors)
        if css_xpath:
            raise ValueError("Please make sure you don't set common keys for"
                             "`css_selectors` and `xpath_selectors`.\n"
                             "Duplicated keys: {}".format(css_xpath))
    for selector in [xpath_selectors, css_selectors]:
        if selector is not None and set(selector).intersection(crawl_headers):
            raise ValueError("Please make sure you don't use names of default "
                             "headers. Avoid using any of these as keys: \n"
                             "{}".format(sorted(crawl_headers)))
    if allowed_domains is None:
        allowed_domains = {urlparse(url).netloc for url in url_list}
    if exclude_url_params is not None and include_url_params is not None:
        if exclude_url_params is True:
            raise ValueError("Please make sure you don't exclude and include "
                             "parameters at the same time.")
        common_params = set(exclude_url_params).intersection(include_url_params)
        if common_params:
            raise ValueError(f"Please make sure you don't include and exclude "
                             f"the same parameters.\n"
                             f"Common parameters entered: "
                             f"{', '.join(common_params)}")
    if include_url_regex is not None and exclude_url_regex is not None:
        if include_url_regex == exclude_url_regex:
            raise ValueError(f"Please make sure you don't include and exclude "
                             f"the same regex pattern.\n"
                             f"You entered '{include_url_regex}'.")

    settings_list = []
    if custom_settings is not None:
        for key, val in custom_settings.items():
            if isinstance(val, (dict, list, set, tuple)):
                setting = '='.join([key, json.dumps(val)])
            else:
                setting = '='.join([key, str(val)])
            settings_list.extend(['-s', setting])

    command = ['scrapy', 'runspider', spider_path,
               '-a', 'url_list=' + ','.join(url_list),
               '-a', 'allowed_domains=' + ','.join(allowed_domains),
               '-a', 'follow_links=' + str(follow_links),
               '-a', 'exclude_url_params=' + str(exclude_url_params),
               '-a', 'include_url_params=' + str(include_url_params),
               '-a', 'exclude_url_regex=' + str(exclude_url_regex),
               '-a', 'include_url_regex=' + str(include_url_regex),
               '-a', 'css_selectors=' + str(css_selectors),
               '-a', 'xpath_selectors=' + str(xpath_selectors),
               '-o', output_file] + settings_list
    if len(','.join(url_list)) > MAX_CMD_LENGTH:
        split_urls = _split_long_urllist(url_list)

        for u_list in split_urls:
            command[4] = 'url_list=' + ','.join(u_list)
            subprocess.run(command)
    else:
        subprocess.run(command)
