Atom and RSS feeds
==================

Sosse can crawl `Atom <https://en.wikipedia.org/wiki/Atom_(web_standard)>`_ and
`RSS <https://en.wikipedia.org/wiki/Rss>`_ feeds, this can be useful to crawl websites that are updated often, and skip
already indexed pages. To index a syndication feed, it needs to be :doc:`queued explicitly <new_url>`.

.. note::
   Sosse crawler does not recurse into feeds declared in the ``<head>`` element of webpages. To crawl a feed, the URL of the XML feed must be added to the crawl queue manually.

Caching for news aggregators
----------------------------

By crawling syndication feeds, Sosse can be used as an offline archive for news aggregator 🐊 softwares. After the XML
feed is indexed, archived pages from the feed can be registered in the aggregator using the
:ref:`atom feed <ui_atom_feeds>` generated by Sosse. This can be done using the
:doc:`search parameters <../user/search>`:

- Leave the keyword field empty
- Set a search parameter to ``Keep`` ``Linked by url`` ``Equal to``, and use the URL of the XML feed as the value
- Sort results by ``First crawled descending``

.. image:: ../../../tests/robotframework/screenshots/syndication_feed.png
   :class: sosse-screenshot
