
<!-- This file is autogenerated as part of avtdl at 2024-12-08 18:12:32.247431 -->

## Description and configuration of available plugins

This page provides details on format and possible values of configuration
options of the plugins and plugins' entities, as well as description of
records produced by them.

High-level overview of the configuration file structure can be found in
[README](README.md#configuration-file-format). Example configurations
for common workflows are available [here](EXAMPLES.md).

In each plugin section, *Plugin configuration options*, if present, describes
plugin-wide settings, placed under the `config` part inside the plugin config.
The *Entity configuration options* lists settings for each entity inside the
`entities` list.

A few common options used by multiple plugins, most notable the templating rules,
are described in greater details in the [Common options](README.md#common-options)
section of the README.

### Table of content:
<!-- [TOC] -->

* [Monitors:](#monitors)
  * [fc2 - Monitor for live.fc2.com](#fc2---monitor-for-livefc2com)
  * [from_file - Monitor content of a text file](#from_file---monitor-content-of-a-text-file)
  * [nitter - Monitor for Nitter instances](#nitter---monitor-for-nitter-instances)
  * [generic_rss - RSS feed monitor](#generic_rss---rss-feed-monitor)
  * [twitcast - Monitor for twitcasting.tv](#twitcast---monitor-for-twitcastingtv)
  * [twitch - Monitor for twitch.tv](#twitch---monitor-for-twitchtv)
  * [get_url - Monitor web page text](#get_url---monitor-web-page-text)
  * [rss - Youtube channel RSS feed monitor](#rss---youtube-channel-rss-feed-monitor)
  * [community - Youtube community page monitor](#community---youtube-community-page-monitor)
  * [channel - Youtube channel monitor](#channel---youtube-channel-monitor)
  * [prechat - Youtube livechat monitor](#prechat---youtube-livechat-monitor)
  * [twitter.home - Monitor for Twitter home timeline](#twitterhome---monitor-for-twitter-home-timeline)
  * [twitter.user - Monitor for user tweets](#twitteruser---monitor-for-user-tweets)
  * [rplay - Monitor livestreams on RPLAY](#rplay---monitor-livestreams-on-rplay)
  * [rplay.user - Monitor livestreams on RPLAY channel](#rplayuser---monitor-livestreams-on-rplay-channel)
* [Filters:](#filters)
  * [filter.noop - Pass everything through](#filternoop---pass-everything-through)
  * [filter.void - Drop everything](#filtervoid---drop-everything)
  * [filter.match - Keep records with specific words](#filtermatch---keep-records-with-specific-words)
  * [filter.exclude - Drop records with specific words](#filterexclude---drop-records-with-specific-words)
  * [filter.event - Filter for records with "Event" type](#filterevent---filter-for-records-with-event-type)
  * [filter.event.cause - Filter for extracting original record from Event](#filtereventcause---filter-for-extracting-original-record-from-event)
  * [filter.type - Filter for records of specific type](#filtertype---filter-for-records-of-specific-type)
  * [filter.json - Format record as JSON](#filterjson---format-record-as-json)
  * [filter.format - Format record as text](#filterformat---format-record-as-text)
  * [filter.format.event - Generate Event from record text](#filterformatevent---generate-event-from-record-text)
  * [filter.deduplicate - Drop already seen records](#filterdeduplicate---drop-already-seen-records)
  * [filter.nitter.pick - Pick NitterRecord with specified properties](#filternitterpick---pick-nitterrecord-with-specified-properties)
  * [filter.nitter.drop - Drop NitterRecord without specified properties.](#filternitterdrop---drop-nitterrecord-without-specified-properties)
  * [filter.channel - Pick YoutubeVideoRecord with specified properties](#filterchannel---pick-youtubevideorecord-with-specified-properties)
  * [filter.channel.notify - Hold upcoming stream's records until the start time](#filterchannelnotify---hold-upcoming-streams-records-until-the-start-time)
  * [filter.twitter - Filter TwitterRecord with specified properties](#filtertwitter---filter-twitterrecord-with-specified-properties)
* [Actions:](#actions)
  * [discord.hook - Send record to Discord using webhook](#discordhook---send-record-to-discord-using-webhook)
  * [execute - Run pre-defined shell command](#execute---run-pre-defined-shell-command)
  * [to_file - Write record to a text file](#to_file---write-record-to-a-text-file)
  * [download - Download a file](#download---download-a-file)
  * [xmpp - Send records as Jabber messages](#xmpp---send-records-as-jabber-messages)
  * [twitter.space - Retrieve Twitter Space metadata from tweet](#twitterspace---retrieve-twitter-space-metadata-from-tweet)

<!-- [TOC] -->
---

## Monitors:


### `fc2` - Monitor for live.fc2.com

Monitors fc2.com user with given id, produces a record when it goes live.
For user `https://live.fc2.com/24374512/`, user id would be `24374512`.

Since the endpoint used for monitoring does not provide the user's nickname,
the name of the configuration entity is used instead.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `user_id`: user id, numeric part at the end of livestream url. Required.
##### 
* `update_interval`: how often the monitored channel will be checked, in seconds. Default value is `120`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.


#### Produced records types:


<details markdown="block">
  <summary>FC2Record</summary>

Represents event of a stream going live on FC2


* `name`: name of the config entity for this user. 
* `url`: url of the user stream. 
* `user_id`: unique for the given user/channel part of the stream url. 
* `title`: stream title. 
* `info`: stream description. 
* `start`: time of the stream start. 
* `start_timestamp`: UNIX timestamp of the stream start. 
* `avatar_url`: link to the user's avatar. 
* `login_only`: whether logging in is required to view current livestream. 

</details>

---

### `from_file` - Monitor content of a text file

On specified intervals, check existence and last modification time
of target file, and if it has changed, read file contents
either line by line or as a whole and emit it as a text record(s).

When `follow` mode is enabled, current position at the end of the file
is preserved and only lines that were appended after it will be read
on consequent update.

Enabling `split_lines` option will cause file content to be split into
multiple records according to regular expressions provided with the
`record_start` and `record_end` settings.

They are set to match the beginning and then end of a line by default,
which means every line will be treated as a separate record. Note, that
both regular expressions are compiled with multiline flag, meaning that
match will span over multiple lines.

It is possible to make a single `record_start` expression to match
entire record by setting `record_end` to empty string ``.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `path`: path to the monitored file. Required.
##### 
* `update_interval`: how often the monitored file should be checked, in seconds. Default value is `60`.
* `encoding`: encoding used to open the monitored file. If not specified, default system-wide encoding is used. Not required.
* `split_lines`: split text into multiple records according to "record_start" and "record_end" patterns. If disabled, a single record with entire text will be produced. Default value is `false`.
* `record_start`: regular expression marking beginning of the record in the text, used when "split_lines" enabled. Default value is `^`.
* `record_end`: regular expression marking the end of the record in the text, used when "split_lines" enabled. Default value is `$`.
* `follow`: remember current position in the file and only read lines below it on consequent update. Default value is `false`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>TextRecord</summary>

Simplest record, containing only a single text field


* `text`: content of the record. 

</details>

---

### `nitter` - Monitor for Nitter instances

Monitors recent tweets, retweets and replies of Twitter user
by scraping and parsing data from a Nitter instance.

Examples of supported urls:

- `https://nitter.net/username`
- `https://nitter.net/username/with_replies`

Some instances might not be happy about getting automated scraping. Make sure
to use a reasonable `update_interval` and keep an eye out for 4XX and 5XX responses
in log, as they might indicate server is under high load or refuses to
communicate.

Nitter has a built-in RSS feed, though not all instances enable it, so it
can also be monitored with `generic_rss` plugin instead of this one.

Twitter Spaces appears on user feed as normal tweets with text only
containing a single link similar to `https://x.com/i/spaces/2FsjOybqEbnzR`.
It therefore can be picked up by using a regular full-text `match` filter.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url`: url that should be monitored. Required.
##### 
* `update_interval`: How often the monitored url will be checked, in seconds. Default value is `1800`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `max_continuation_depth`: when updating feed with pagination support, only continue for this many pages. Default value is `10`.
* `next_page_delay`: when updating feed with pagination support, wait this much before loading next page. Default value is `1`.
* `allow_discontinuity`: when updating feed with pagination support, if this setting is enabled and error happens when loading a page, records from already parsed pages will not be dropped. It will allow update of the feed to finish, but older records from deeper pages will then never be parsed on consecutive updates. Default value is `false`.
* `fetch_until_the_end_of_feed_mode`: when updating feed with pagination support, enables special mode, which makes a monitor try loading and parsing all pages until the end, even if they have been already parsed. Designed for purpose of archiving entire feed content. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>NitterRecord</summary>

Single post as parsed from Nitter instance

Depending on the tweet type (regular, retweet, reply, quote) some fields might be empty


* `retweet_header`: text line saying this is a retweet. 
* `reply_header`: text line saying this tweet is a reply. 
* `url`: tweet url. 
* `author`: user's visible name. 
* `username`: user's handle. 
* `avatar_url`: link to the picture used as the user's avatar. 
* `published`: tweet timestamp. 
* `text`: tweet text with stripped formatting. 
* `html`: tweet text as raw html. 
* `attachments`: list of links to attached images or video thumbnails. 
* `quote`: Nested NitterRecord containing tweet being quited. 

</details>

---

### `generic_rss` - RSS feed monitor

Monitors RSS feed for new entries. Will attempt to adjust
update interval based on HTTP response headers.

Depending on a specific feed format, fields names and content
might vary greatly. Commonly present standardized fields are
`url`, `title` and `author`, though they might be empty
in some feeds.

Before defining a command to be executed for records of a newly
added feed it is recommended to inspect the feed entity content by
forwarding records in a file in JSON format using `to_file` plugin.

Normally feeds have some kind of value to uniquely identify
feed entries, but in case there is none, parser will attempt
to create one by combining `link` and `title` or `summary` fields.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `update_interval`: how often the monitored source should be checked for new content, in seconds. Required.
* `url`: url that should be monitored. Required.
##### 
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.


#### Produced records types:


<details markdown="block">
  <summary>GenericRSSRecord</summary>

Represents RSS feed entry

Might contain additional fields if they are present in the feed.


* `uid`: value that is unique for this entry of RSS feed. 
* `url`: "href" or "link" field value of this entry. 
* `summary`: "summary" or "description" field value of this entry. 
* `author`: "author" field value. Might be empty. 
* `title`: "title" field value. Might be empty. 
* `published`: "published" or "issued" field value of this entry. 

</details>

---

### `twitcast` - Monitor for twitcasting.tv

Monitors twitcasting.tv user with a given id, produces a record when it goes live.
For user `https://twitcasting.tv/c:username` user id would be `c:username`.

Streams on Twitcasting might be set to be visible only for members of a specific group.
For monitoring such streams it is necessarily to provide login cookies of
an account being a member of the group. Password-protected and age-restricted streams
do not require that.

Rate limits for the endpoint used to check if user is live are likely relatively high,
but it is better to keep `update_interval` big enough for combined amount of updates
for all monitored users to not exceed one request per second.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `user_id`: user id that should be monitored. Required.
##### 
* `update_interval`: how often the user will be checked for being live, in seconds. Default value is `60`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.


#### Produced records types:


<details markdown="block">
  <summary>TwitcastRecord</summary>

Represents an event of a user going live on Twitcasting


* `user_id`: unique part of the channel url. 
* `movie_id`: unique id for the current livestream. 
* `url`: user (channel) url. 
* `movie_url`: current livestream url. 
* `title`: livestream title. 
* `thumbnail_url`: link to the stream thumbnail. 

</details>

---

### `twitch` - Monitor for twitch.tv

Monitors twitch.tv user with given username, produces a record when it goes live.
For user `https://www.twitch.tv/username` username would be `username`.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `username`: Twitch username of a monitored channel. Required.
##### 
* `update_interval`: how often the user will be checked for being live, in seconds. Default value is `300`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.


#### Produced records types:


<details markdown="block">
  <summary>TwitchRecord</summary>

Represents event of a user going live on Twitch


* `url`: channel url. 
* `username`: username value from configuration entity. 
* `title`: stream title. 
* `start`: timestamp of the stream start. 
* `avatar_url`: link to the user's avatar. 
* `game`: game name, if present. 

</details>

---

### `get_url` - Monitor web page text

Download contents of web page at `url` and emit it as a `TextRecord`
if it has changed since the last update. Intended for working with simple
text endpoints.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `update_interval`: how often the monitored source should be checked for new content, in seconds. Required.
* `url`: url to monitor. Required.
##### 
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.


#### Produced records types:


<details markdown="block">
  <summary>TextRecord</summary>

Simplest record, containing only a single text field


* `text`: content of the record. 

</details>

---

### `rss` - Youtube channel RSS feed monitor

Monitors channel for new uploads and livestreams using RSS feed
generated by Youtube. Requires old channel id format. In order to obtain
RSS feed url for a given channel, use "View Source" on a channel page
and search for "rss".

Example of a supported url:

- `https://www.youtube.com/feeds/videos.xml?channel_id=UCK0V3b23uJyU4N8eR_BR0QA`

RSS feed is smaller and faster to parse compared to an HTML channel page,
but by design only shows updates of a single channel and doesn't support
authentication and therefore unable to show member-only streams.

Scheduled date for upcoming streams is not present in the feed itself, so it
is obtained by fetching and parsing video page the first time it appears in
the feed. It then gets updated until stream goes live, unless `track_reschedule`
option is disabled.

No matter how often the url gets fetched, content of the feed only gets
changed once every 15 minutes, so setting `update_interval` lower than that
value is not recommended. This monitor will attempt to calculate time of the
next update from HTTP headers and schedule next request right after it. Use
`adjust_update_interval` to disable this behavior.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url`: url that should be monitored. Required.
##### 
* `update_interval`: How often the feed should be updated, in seconds. Default value is `900`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `track_reschedule`: keep track of scheduled time of upcoming streams, emit record again if it is changed to an earlier date. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>YoutubeFeedRecord</summary>

Youtube video or livestream parsed from a channel's RSS feed


* `url`: link to the video. 
* `title`: title of the video at the time of parsing. 
* `published`: published value of the feed item, usually the time when the video was uploaded or the livestream frame was set up. 
* `updated`: updated value of the feed item. If different from `published`, might indicate either a change to video title, thumbnail or description, or a change in video status, for example livestream ending. 
* `thumbnail_url`: link to the video thumbnail. 
* `author`: author's name, as shown on the channel icon. 
* `video_id`: short string identifying the video on Youtube. Part of the video url. 
* `summary`: video's description. 
* `views`: current number of views. Is zero for upcoming and ongoing livestreams. 
* `scheduled`: scheduled time for an upcoming livestream to start at, otherwise absent. 

</details>

---

### `community` - Youtube community page monitor

Monitors posts on community page of a channel, supports
member-only posts if login cookies are provided. Some features,
such as polls, are not supported.

Examples of supported url:

- `https://www.youtube.com/@ChannelName/community`
- `https://www.youtube.com/channel/UCK0V3b23uJyU4N8eR_BR0QA/community`


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url`: url of the community page of the channel. Required.
##### 
* `update_interval`: how often the community page will be checked for new posts. Default value is `1800`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `max_continuation_depth`: when updating feed with pagination support, only continue for this many pages. Default value is `10`.
* `next_page_delay`: when updating feed with pagination support, wait this much before loading next page. Default value is `1`.
* `allow_discontinuity`: when updating feed with pagination support, if this setting is enabled and error happens when loading a page, records from already parsed pages will not be dropped. It will allow update of the feed to finish, but older records from deeper pages will then never be parsed on consecutive updates. Default value is `false`.
* `fetch_until_the_end_of_feed_mode`: when updating feed with pagination support, enables special mode, which makes a monitor try loading and parsing all pages until the end, even if they have been already parsed. Designed for purpose of archiving entire feed content. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>CommunityPostRecord</summary>

Youtube community post content


* `channel_id`: channel ID in old format. 
* `post_id`: unique id of the post. 
* `author`: author's channel name. 
* `avatar_url`: link to the avatar of the channel. 
* `vote_count`: current number of upvotes. 
* `sponsor_only`: indicates whether the post is member-only. 
* `published_text`: localized text saying how long ago the video was uploaded. 
* `full_text`: post contents as plaintext. 
* `attachments`: list of links to attached images or video thumbnails. 
* `video_id`: if the post links to youtube video will contain video id, otherwise absent. 
* `original_post`: for reposts contains original post content, otherwise absent. 

</details>


<details markdown="block">
  <summary>SharedCommunityPostRecord</summary>

Youtube community post that is itself a repost of another post


* `channel_id`: channel ID in old format. 
* `post_id`: unique id of the post. 
* `author`: author's channel name. 
* `avatar_url`: link to the avatar of the channel. 
* `published_text`: localized text saying how long ago the video was uploaded. 
* `full_text`: post content. 
* `original_post`: not present in shared post. 

</details>

---

### `channel` - Youtube channel monitor

Monitors Youtube url listing videos, such as channels main page,
videos and streams tab of a channel, as well as playlists, and,
with login cookies, subscriptions feed or the main page.

Due to small differences in presentation in aforementioned
sources, same video might have slightly different appearance when
parsed from different urls. For example, video parsed from main
page or subscriptions feed will not have full description text.

Examples of supported url:

- `https://www.youtube.com/@ChannelName`
- `https://www.youtube.com/@ChannelName/videos`
- `https://www.youtube.com/@ChannelName/streams`
- `https://www.youtube.com/channel/UCK0V3b23uJyU4N8eR_BR0QA/`
- `https://www.youtube.com/playlist?list=PLWGY3fcU-ZeQmBfoJ6SmT8v2zV8NEhrB2`
- `https://www.youtube.com/feed/subscriptions` (providing cookies is necessarily)

Unlike `rss` monitor, with login cookies it can see videos and streams
with limited access (such as member-only).

While monitoring a single channel is less efficient, using this monitor with
subscriptions feed url on a dedicated account is a recommended way
to monitor a high amount (hundreds) of channels, as it only requires
loading a single page to check all of them for updates.

When main page of a channel (https://www.youtube.com/@ChannelName) is viewed
in logged in state, it might contain "For you" block, which content might
vary with subsequent updates. As a result, monitoring this url might occasionally
produce records with old videos that got showed in this block. If monitoring
without a cookies file is not an option, use a combination of "Videos" and "Streams"
tabs instead.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url`: url that should be monitored. Required.
##### 
* `update_interval`: . Default value is `1800`.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `adjust_update_interval`: change delay before next update based on response headers. This setting doesn't affect timeouts after failed requests. Default value is `true`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `max_continuation_depth`: when updating feed with pagination support, only continue for this many pages. Default value is `10`.
* `next_page_delay`: when updating feed with pagination support, wait this much before loading next page. Default value is `1`.
* `allow_discontinuity`: when updating feed with pagination support, if this setting is enabled and error happens when loading a page, records from already parsed pages will not be dropped. It will allow update of the feed to finish, but older records from deeper pages will then never be parsed on consecutive updates. Default value is `false`.
* `fetch_until_the_end_of_feed_mode`: when updating feed with pagination support, enables special mode, which makes a monitor try loading and parsing all pages until the end, even if they have been already parsed. Designed for purpose of archiving entire feed content. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>YoutubeVideoRecord</summary>

Youtube video or livestream listed among others on Youtube page

Produced by parsing a channels main page, videos and streams tab,
as well as playlists, and, with login cookies, subscriptions feed.


* `video_id`: short string identifying video on Youtube. Part of video url. 
* `url`: link to the video, uses `https://www.youtube.com/watch?v=<video_id>` format. 
* `title`: title of the video at the time of parsing. 
* `summary`: snippet of the video description. Not always available. 
* `scheduled`: scheduled date for upcoming stream or premiere. 
* `author`: channel name. 
* `avatar_url`: link to the avatar of the channel. Not always available. 
* `thumbnail_url`: link to the video thumbnail. 
* `channel_link`: link to the channel uploading the video. 
* `channel_id`: channel ID in old format (such as `UCK0V3b23uJyU4N8eR_BR0QA`). 
* `published_text`: localized text saying how long ago the video was uploaded. 
* `length`: text showing the video duration (hh:mm:ss). 
* `is_upcoming`: indicates that video is an upcoming livestream or premiere. 
* `is_live`: indicates that the video is a livestream or premiere that is currently live. 
* `is_member_only`: indicated that the video is limited to members of the channel. Note that the video status might be changed at any time. 

</details>

---

### `prechat` - Youtube livechat monitor

Monitor chat of a Youtube livestream and produce a record
for each chat message. Though it is capable of processing
chat on an ongoing stream and chat replay on a stream VOD,
the main purpose is to monitor and preserve chat of upcoming
livestreams.

Some features, such as polls, are not supported.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url`: url of a video frame with a chat. Required.
##### 
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.


#### Produced records types:


<details markdown="block">
  <summary>YoutubeChatRecord</summary>

Youtube chat message


* `author`: message author's name. 
* `channel`: message author's channel url. 
* `badges`: localized list of message author's badges (owner, moderator, member, verified and so on). 
* `timestamp`: timestamp of when the message was sent. 
* `text`: message content as plaintext. 
* `amount`: for superchats, string specifying amount and currency, otherwise empty. 
* `banner_header`: used for special objects in chat, such as pinned messages. 
* `message_header`: . 
* `sticker`: supersticker name if the message is a supersticker, otherwise empty. 
* `color`: message header color (RGB integer), if present. 
* `video_id`: video_id of the video chat message was sent to. 
* `video_title`: title of the video chat message was sent to. 
* `video_author`: name of the channel where the video chat message was sent to is published. 
* `uid`: unique id of the message. 
* `action`: internal name of message type. Used for debug purposes. 
* `renderer`: internal name of message format. Used for debug purposes. 

</details>

---

### `twitter.home` - Monitor for Twitter home timeline

Monitors tweets on Twitter Home Timeline, either the "Following"
of the "For you" tab.

Requires login cookies from a logged in Twitter account to work.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `cookies_file`: path to a text file containing cookies in Netscape format. Required.
##### 
* `update_interval`: how often the monitored url will be checked, in seconds. Default value is `1800`.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `url`: Twitter domain name. Default value is `https://twitter.com`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `max_continuation_depth`: when updating feed with pagination support, only continue for this many pages. Default value is `10`.
* `next_page_delay`: when updating feed with pagination support, wait this much before loading next page. Default value is `1`.
* `allow_discontinuity`: when updating feed with pagination support, if this setting is enabled and error happens when loading a page, records from already parsed pages will not be dropped. It will allow update of the feed to finish, but older records from deeper pages will then never be parsed on consecutive updates. Default value is `false`.
* `fetch_until_the_end_of_feed_mode`: when updating feed with pagination support, enables special mode, which makes a monitor try loading and parsing all pages until the end, even if they have been already parsed. Designed for purpose of archiving entire feed content. Default value is `false`.
* `following`: monitor tweets from the "Following" tab instead of "For you" . Default value is `true`.


#### Produced records types:


<details markdown="block">
  <summary>TwitterRecord</summary>

Single post as parsed from Twitter instance

Depending on the tweet type (regular, retweet, reply, quote) some fields might be empty


* `uid`: tweet id. 
* `url`: tweet url. 
* `author`: user's visible name. 
* `username`: user's handle. 
* `avatar_url`: link to the picture used as the user's avatar. 
* `published`: tweet timestamp. 
* `text`: tweet text. Links are unshortened if possible. 
* `attachments`: list of links to attached images or video thumbnails. 
* `images`: list of links to attached images. 
* `videos`: list of links to attached videos and gifs. 
* `replying_to_username`: for replies, name of the user that got replied to. 
* `retweet`: for retweets, nested TwitterRecord containing tweet that was retweeted. 
* `quote`: for quotes, nested TwitterRecord containing tweet being quoted. 
* `space_id`: for tweets or retweets mentioning Twitter Space url contains unique part of the url, otherwise empty. 

</details>

---

### `twitter.user` - Monitor for user tweets

Monitors timeline of a user for new tweets, including retweets and quotes.
Enabling `with_replies` will additionally include replies posted by the user.

With `only_likes` option enabled tweets from the "Likes" tab are collected
instead of user's own tweets.

Requires login cookies from a logged in Twitter account to work.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `cookies_file`: path to a text file containing cookies in Netscape format. Required.
* `user`: user handle. Required.
##### 
* `update_interval`: how often the monitored url will be checked, in seconds. Default value is `1800`.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `url`: Twitter domain name. Default value is `https://twitter.com`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `max_continuation_depth`: when updating feed with pagination support, only continue for this many pages. Default value is `10`.
* `next_page_delay`: when updating feed with pagination support, wait this much before loading next page. Default value is `1`.
* `allow_discontinuity`: when updating feed with pagination support, if this setting is enabled and error happens when loading a page, records from already parsed pages will not be dropped. It will allow update of the feed to finish, but older records from deeper pages will then never be parsed on consecutive updates. Default value is `false`.
* `fetch_until_the_end_of_feed_mode`: when updating feed with pagination support, enables special mode, which makes a monitor try loading and parsing all pages until the end, even if they have been already parsed. Designed for purpose of archiving entire feed content. Default value is `false`.
* `with_replies`: include replies by monitored user. Default value is `true`.
* `only_likes`: monitor tweets liked by the user instead of user's own tweets. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>TwitterRecord</summary>

Single post as parsed from Twitter instance

Depending on the tweet type (regular, retweet, reply, quote) some fields might be empty


* `uid`: tweet id. 
* `url`: tweet url. 
* `author`: user's visible name. 
* `username`: user's handle. 
* `avatar_url`: link to the picture used as the user's avatar. 
* `published`: tweet timestamp. 
* `text`: tweet text. Links are unshortened if possible. 
* `attachments`: list of links to attached images or video thumbnails. 
* `images`: list of links to attached images. 
* `videos`: list of links to attached videos and gifs. 
* `replying_to_username`: for replies, name of the user that got replied to. 
* `retweet`: for retweets, nested TwitterRecord containing tweet that was retweeted. 
* `quote`: for quotes, nested TwitterRecord containing tweet being quoted. 
* `space_id`: for tweets or retweets mentioning Twitter Space url contains unique part of the url, otherwise empty. 

</details>

---

### `rplay` - Monitor livestreams on RPLAY

Monitors users with `creator_oid` listed in `creators`, produces a record
when any of them starts a livestream. When `creators` list is not provided,
every livestream on the site will generate a record. `creator_oid` is the
unique part of the user's home or livestream url.
For example, `creator_oid` is `6596e71c04a7ea2fd7c36ae7`
for the following urls:

- `https://rplay.live/creatorhome/6596e71c04a7ea2fd7c36ae7`
- `https://rplay.live/live/6596e71c04a7ea2fd7c36ae7`

This monitor checks all creators with a single request,
however, because of that it does not try to retrieve the
direct `playlist url` for livestreams, and therefore does
not support providing login credentials.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `update_interval`: how often the monitored channel will be checked, in seconds. Default value is `300`.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `creators`: list of IDs of the users to monitor. Entity with no IDs will report on every user going live. Not required.


#### Produced records types:


<details markdown="block">
  <summary>RplayRecord</summary>

Ongoing livestream on RPLAY


* `url`: livestream url. 
* `title`: stream title. 
* `description`: stream description. 
* `thumbnail_url`: url of the stream thumbnail. 
* `start`: time of the stream start. 
* `user_id`: user's oid. 
* `creator_id`: creatorOid, unique id used in channel and livestream urls. 
* `name`: visible name of the user. 
* `avatar_url`: link to the user's avatar. 
* `restream_platform`: for restream, platform the stream is being hosted on. 
* `restream_url`: for restream, url of the stream on the source platform. 
* `playlist_url`: for native stream, link to the underlying hls playlist if was retrieved successfully.
Might be invalid even when present in case of insufficient permissions or network error. 

</details>

---

### `rplay.user` - Monitor livestreams on RPLAY channel

Monitors a user with given `creator_oid`, produces record
when the user starts a livestream. `creator_oid` is the
unique part of the user's home or livestream url.
For example, `creator_oid` is `6596e71c04a7ea2fd7c36ae7`
for the following urls:

- `https://rplay.live/creatorhome/6596e71c04a7ea2fd7c36ae7`
- `https://rplay.live/live/6596e71c04a7ea2fd7c36ae7`

When producing a record, this plugin will generate `playlist_url`
for the stream. If credentials are provided in the `config` section,
it will try to update it with the key required to access livestreams
limited to subscribers. Resulting `playlist_url` might still be
invalid if the update failed or account does not have permissions
to view the stream.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.
* `login`: rplay login of the account used for monitoring. Not required.
* `password`: rplay password of the account used for monitoring. Not required.
* `user_id`: userOid of the account used for monitoring. When absent can be retrieved automatically by performing a login request. Not required.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `creator_oid`: ID of the user to monitor. Required.
##### 
* `update_interval`: how often the monitored channel will be checked, in seconds. Default value is `300`.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Default value is `"Accept-Language": "en-US,en;q=0.9"`.
* `quiet_start`: throw away new records on the first update after application startup. Default value is `false`.
* `quiet_first_time`: throw away new records produced on first update of given url. Default value is `true`.
* `creators`: list of IDs of the users to monitor. Entity with no IDs will report on every user going live. Not required.


#### Produced records types:


<details markdown="block">
  <summary>RplayRecord</summary>

Ongoing livestream on RPLAY


* `url`: livestream url. 
* `title`: stream title. 
* `description`: stream description. 
* `thumbnail_url`: url of the stream thumbnail. 
* `start`: time of the stream start. 
* `user_id`: user's oid. 
* `creator_id`: creatorOid, unique id used in channel and livestream urls. 
* `name`: visible name of the user. 
* `avatar_url`: link to the user's avatar. 
* `restream_platform`: for restream, platform the stream is being hosted on. 
* `restream_url`: for restream, url of the stream on the source platform. 
* `playlist_url`: for native stream, link to the underlying hls playlist if was retrieved successfully.
Might be invalid even when present in case of insufficient permissions or network error. 

</details>


## Filters:


### `filter.noop` - Pass everything through

Lets all incoming records pass through unchanged, effectively
doing nothing with them. As any other filter it has entities,
so it can be used as a merging point to gather records from
multiple chains and process them in a single place.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.

---

### `filter.void` - Drop everything

Does not produce anything, dropping all incoming records.
Can be used to stuff multiple chains in one if the need ever arises.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.

---

### `filter.match` - Keep records with specific words

This filter lets through records that have one of the values
defined by `patterns` list found in any (or specified) field of the record.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `patterns`: list of strings to search for in the record. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `fields`: field names to search the patterns in. If not specified, all fields are checked. Not required.

---

### `filter.exclude` - Drop records with specific words

This filter lets through records that have none of the values
defined by `patterns` list found in any (or specified) field of the record.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `patterns`: list of strings to search for in the record. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `fields`: field names to search the patterns in. If not specified, all fields are checked. Not required.

---

### `filter.event` - Filter for records with "Event" type

Only lets through Events and not normal Records. Can be used to
set up notifications on events (such as errors) from, for example,
`execute` plugin within the same chain that uses it, by separating
them from regular records.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `event_types`: list of event types. See descriptions of plugins producing events for possible values. Not required.


#### Produced records types:


<details markdown="block">
  <summary>Event</summary>

Record produced by an internal event (usually error) inside the plugin


* `event_type`: text describing the nature of event, can be used to filter classes of events, such as errors. 
* `text`: text describing specific even details. 

</details>

---

### `filter.event.cause` - Filter for extracting original record from Event

Take an Event and return the record that was being processed
when it happened. For example, Event sent by `to_file`
plugin failing to write a TextRecord in a file will produce
the original TextRecord.

Regular records (not Events) are passed through unchanged.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.

---

### `filter.type` - Filter for records of specific type

Only lets through records of specified types, such as `Event` or `YoutubeVideoRecord`.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `types`: list of records class names, such as "Record" and "Event" . Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `exact_match`: whether match should check for exact record type or look in entire records hierarchy up to Record. Default value is `false`.

---

### `filter.json` - Format record as JSON

Takes record and produces a new `TextRecord` rendering fields of the
original record in JSON format, with option for pretty-print.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `prettify`: whether output should be multiline and indented or a single line. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>TextRecord</summary>

Simplest record, containing only a single text field


* `text`: content of the record. 

</details>

---

### `filter.format` - Format record as text

Takes a record and produces a new `TextRecord` by taking `template` string
and replacing "{placeholder}" with value of `placeholder` field of the
current record, where `placeholder` is any field the record might have.
If one of the placeholders is not a field of a specific record, it will be
replaced with a value defined in `missing` parameter if it is specified,
otherwise it will be left intact.

Because output record is essentially a text, timezone offset will be lost
for all fields containing date and time value. Therefore, the `timezone`
parameter is provided to allow formatting these fields in desired timezone.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `template`: template string with placeholders that will be filled with corresponding values from current record. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `missing`: if specified, will be used to fill template placeholders that do not have corresponding fields in current record. Not required.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.


#### Produced records types:


<details markdown="block">
  <summary>TextRecord</summary>

Simplest record, containing only a single text field


* `text`: content of the record. 

</details>

---

### `filter.format.event` - Generate Event from record text

Takes a record and produces a new `Event` with the `event_type` and
`text` values evaluated by filling placeholders in `type_template`
and `text_template`.

Just like with the `filter.format`, missing placeholders are filled with
value from the `missing` parameter if it is specified, otherwise they are
left unchanged.

Specifying timezone offset for the output text is also
supported, though original record might be retrieved with `filter.event.cause`.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `type_template`: template string with placeholders that will be filled with corresponding values from current record. Required.
* `text_template`: template string with placeholders that will be filled with corresponding values from current record. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `missing`: if specified, will be used to fill template placeholders that do not have corresponding fields in current record. Not required.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.


#### Produced records types:


<details markdown="block">
  <summary>Event</summary>

Record produced by an internal event (usually error) inside the plugin


* `event_type`: text describing the nature of event, can be used to filter classes of events, such as errors. 
* `text`: text describing specific even details. 

</details>

---

### `filter.deduplicate` - Drop already seen records

Checks if the `field` field value of the current record has already been
present in one of the previous records and only let it through otherwise.

`field` might be either a record field name or one of `hash` or `as_json`
for sha1 and fulltext comparison. If `field` is not present in the current
record, it will be passed through as if it's new.

This filter will work with records of any type, as long as they have defined
field (all records have `hash` and `as_json`). For example, it is possible
to ensure no multiple records for a single video will be produced
in a chain, that gather records from Youtube channel and Youtube RSS monitors,
by passing them to an entity of this filter with `field` set to `video_id`.

Note, that history is kept in memory, so it will not be persisted between
restarts.


#### Plugin configuration options:
* `history_dir`: directory to store entities history between restarts. Default value is `cache/deduplicate/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `field`: field name to use for comparison. Default value is `hash`.
* `history_size`: how many old records should be kept in memory. Default value is `10000`.

---

### `filter.nitter.pick` - Pick `NitterRecord` with specified properties

Lets through `NitterRecord` if it matches any of specified criteria.
All records from other sources pass through without filtering.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `retweet`: match retweets. Default value is `false`.
* `reply`: match replies. Default value is `false`.
* `quote`: match quotes. Default value is `false`.
* `regular_tweet`: match regular tweets that are not a retweet, reply or quote. Default value is `false`.
* `author`: match if a given string is a part of the name of the author of the tweet. Not required.
* `username`: match if a given string is a part of tweet author's username (without the "@" symbol). Not required.


#### Produced records types:


<details markdown="block">
  <summary>NitterRecord</summary>

Single post as parsed from Nitter instance

Depending on the tweet type (regular, retweet, reply, quote) some fields might be empty


* `retweet_header`: text line saying this is a retweet. 
* `reply_header`: text line saying this tweet is a reply. 
* `url`: tweet url. 
* `author`: user's visible name. 
* `username`: user's handle. 
* `avatar_url`: link to the picture used as the user's avatar. 
* `published`: tweet timestamp. 
* `text`: tweet text with stripped formatting. 
* `html`: tweet text as raw html. 
* `attachments`: list of links to attached images or video thumbnails. 
* `quote`: Nested NitterRecord containing tweet being quited. 

</details>

---

### `filter.nitter.drop` - Drop `NitterRecord` without specified properties.

Lets through `NitterRecord` if it doesn't match all of the specified criteria.
All records from other sources pass through without filtering.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `retweet`: match retweets. Default value is `false`.
* `reply`: match replies. Default value is `false`.
* `quote`: match quotes. Default value is `false`.
* `regular_tweet`: match regular tweets that are not a retweet, reply or quote. Default value is `false`.
* `author`: match if a given string is a part of the name of the author of the tweet. Not required.
* `username`: match if a given string is a part of tweet author's username (without the "@" symbol). Not required.


#### Produced records types:


<details markdown="block">
  <summary>NitterRecord</summary>

Single post as parsed from Nitter instance

Depending on the tweet type (regular, retweet, reply, quote) some fields might be empty


* `retweet_header`: text line saying this is a retweet. 
* `reply_header`: text line saying this tweet is a reply. 
* `url`: tweet url. 
* `author`: user's visible name. 
* `username`: user's handle. 
* `avatar_url`: link to the picture used as the user's avatar. 
* `published`: tweet timestamp. 
* `text`: tweet text with stripped formatting. 
* `html`: tweet text as raw html. 
* `attachments`: list of links to attached images or video thumbnails. 
* `quote`: Nested NitterRecord containing tweet being quited. 

</details>

---

### `filter.channel` - Pick `YoutubeVideoRecord` with specified properties

Filter that only lets `YoutubeVideoRecord` through if it has certain properties.
All records from other sources pass through without filtering.

If multiple settings are set to `true`, they all should match. Use multiple
entities if picking records with any of multiple properties is required.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `upcoming`: to pass the filter a record should be either upcoming livestream or scheduled premiere. Default value is `false`.
* `live`: to pass the filter a record should be an ongoing livestream. Default value is `false`.
* `member_only`: to pass the filter a record should be marked as member-only. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>YoutubeVideoRecord</summary>

Youtube video or livestream listed among others on Youtube page

Produced by parsing a channels main page, videos and streams tab,
as well as playlists, and, with login cookies, subscriptions feed.


* `video_id`: short string identifying video on Youtube. Part of video url. 
* `url`: link to the video, uses `https://www.youtube.com/watch?v=<video_id>` format. 
* `title`: title of the video at the time of parsing. 
* `summary`: snippet of the video description. Not always available. 
* `scheduled`: scheduled date for upcoming stream or premiere. 
* `author`: channel name. 
* `avatar_url`: link to the avatar of the channel. Not always available. 
* `thumbnail_url`: link to the video thumbnail. 
* `channel_link`: link to the channel uploading the video. 
* `channel_id`: channel ID in old format (such as `UCK0V3b23uJyU4N8eR_BR0QA`). 
* `published_text`: localized text saying how long ago the video was uploaded. 
* `length`: text showing the video duration (hh:mm:ss). 
* `is_upcoming`: indicates that video is an upcoming livestream or premiere. 
* `is_live`: indicates that the video is a livestream or premiere that is currently live. 
* `is_member_only`: indicated that the video is limited to members of the channel. Note that the video status might be changed at any time. 

</details>

---

### `filter.channel.notify` - Hold upcoming stream's records until the start time

Determines whether a record represents a Youtube livestream/Premiere
with a scheduled time, and holds it waiting until the time comes
instead of passing down the chain immediately if needed.

If the record is not an upcoming Youtube livestream, it gets silently dropped.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `prior`: output a record this many minutes before the scheduled start of a live broadcast. Default value is `10`.
* `include_ongoing`: whether currently live streams should be included. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>YoutubeVideoRecord</summary>

Youtube video or livestream listed among others on Youtube page

Produced by parsing a channels main page, videos and streams tab,
as well as playlists, and, with login cookies, subscriptions feed.


* `video_id`: short string identifying video on Youtube. Part of video url. 
* `url`: link to the video, uses `https://www.youtube.com/watch?v=<video_id>` format. 
* `title`: title of the video at the time of parsing. 
* `summary`: snippet of the video description. Not always available. 
* `scheduled`: scheduled date for upcoming stream or premiere. 
* `author`: channel name. 
* `avatar_url`: link to the avatar of the channel. Not always available. 
* `thumbnail_url`: link to the video thumbnail. 
* `channel_link`: link to the channel uploading the video. 
* `channel_id`: channel ID in old format (such as `UCK0V3b23uJyU4N8eR_BR0QA`). 
* `published_text`: localized text saying how long ago the video was uploaded. 
* `length`: text showing the video duration (hh:mm:ss). 
* `is_upcoming`: indicates that video is an upcoming livestream or premiere. 
* `is_live`: indicates that the video is a livestream or premiere that is currently live. 
* `is_member_only`: indicated that the video is limited to members of the channel. Note that the video status might be changed at any time. 

</details>


<details markdown="block">
  <summary>YoutubeFeedRecord</summary>

Youtube video or livestream parsed from a channel's RSS feed


* `url`: link to the video. 
* `title`: title of the video at the time of parsing. 
* `published`: published value of the feed item, usually the time when the video was uploaded or the livestream frame was set up. 
* `updated`: updated value of the feed item. If different from `published`, might indicate either a change to video title, thumbnail or description, or a change in video status, for example livestream ending. 
* `thumbnail_url`: link to the video thumbnail. 
* `author`: author's name, as shown on the channel icon. 
* `video_id`: short string identifying the video on Youtube. Part of the video url. 
* `summary`: video's description. 
* `views`: current number of views. Is zero for upcoming and ongoing livestreams. 
* `scheduled`: scheduled time for an upcoming livestream to start at, otherwise absent. 

</details>

---

### `filter.twitter` - Filter `TwitterRecord` with specified properties

Lets through `TwitterRecord` if it matches any of specified criteria.
Enabling `reversed` setting reverses the conditions, making record match
if none of the criteria apply.

All records from other sources pass through without filtering.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `retweet`: match retweets. Default value is `false`.
* `reply`: match replies. Default value is `false`.
* `quote`: match quotes. Default value is `false`.
* `regular_tweet`: match regular tweets that are not a retweet, reply or quote. Default value is `false`.
* `author`: match if a given string is a part of the name of the author of the tweet. Not required.
* `username`: match if a given string is a part of tweet author's username (without the "@" symbol). Not required.
* `reversed`: drop record instead of letting it through if any of the properties matches. Default value is `false`.


## Actions:


### `discord.hook` - Send record to Discord using webhook

To generate webhook url follow instructions in "Making a Webhook" section of
<https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks>

Some record types support rich formatting when sent to Discord, such as
showing the author's avatar and links to attached images. Youtube videos will
show thumbnail, however embedding video itself is not supported.

Records coming within six seconds one after another will be batched together into a single message.
When too many records are received at once, they will be sent with delays to conform to Discord
rate limits. Records deemed to be too long to fit in a Discord message
[length limits](https://discord.com/developers/docs/resources/channel#create-message-jsonform-params)
will be dropped with a warning.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url`: webhook url. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `consume_record`: whether record should be consumed or passed down the chain after processing. Disabling it allows chaining multiple Actions. Default value is `true`.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.

---

### `execute` - Run pre-defined shell command

Take `command` string, replace keywords wrapped in curly braces with corresponding fields
of currently processed record. For example, if `command` is set to

    "yt-dlp {url}"`

and currently processed record comes from Youtube RSS feed and has `url` field value
`https://www.youtube.com/watch?v=L692Sxz3thw`, then with default `placeholders`
resulting command will be

    yt-dlp https://www.youtube.com/watch?v=L692Sxz3thw

See `Formatting templates` section of README.md for more detailed description
of the formatting options.

`command` string is not treated as raw shell command. Instead, it is split into list
 of elements, where first element specifies the program executable, and the rest
 specify the arguments. It is therefore not possible to use shell features such as pipes
 or execute multiple commands in one line.

Make sure the executable the command uses (`yt-dlp` in this case) is installed and
can be run from the working directory by current user. It is advised to confirm that
the command can be executed manually and it finishes without errors before automating it.

For each entity, a separate working directory can be configured. Output is shown
in the same window by default, but can be redirected to a file, with either static
or autogenerated name.

Produces Events at startup and successful or erroneous termination of the executed command
if corresponding entity settings are enabled. They can be used to send Discord
or Jabber notifications or execute another command when it happens.

Processed record itself can also be passed down the chain if the command failed,
providing a way to try a different one as a fallback. For example, record with
Youtube url could be first handled by `ytarchive` and passed to `yt-dlp` if
it happens to fail due to video link not being a livestream.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `command`: shell command to be executed on every received record. Supports placeholders that will be replaced with currently processed record fields values. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `consume_record`: whether record should be consumed or passed down the chain after processing. Disabling it allows chaining multiple Actions. Default value is `true`.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.
* `working_dir`: path to the directory where command will be executed. If not set current working directory is used. Supports templating with {...}. Not required.
* `log_dir`: write executed process output to a file in this directory if set. If it is not set, output will not be redirected to file. Not required.
* `log_filename`: filename to write executed process output to. If not defined, it is generated automatically based on command and entity name. Not required.
* `static_placeholders`: parts of `command` string that will be replaced with provided values, defined as mapping `'placeholder': 'replacement string'`. Intended to allow reusing same `command` template for multiple entities. Not required.
* `forward_failed`: emit currently processed record down the chain if the subprocess returned non-zero exit code. Can be used to define fallback command in case this one fails. Default value is `false`.
* `report_failed`: emit Event with type "error" if the subprocess returned non-zero exit code or raised exception. Default value is `true`.
* `report_finished`: emit Event with type "finished" if the subprocess returned zero as exit code. Default value is `false`.
* `report_started`: emit Event with type "started" before starting a subprocess. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>Event</summary>

Record produced by an internal event (usually error) inside the plugin


* `event_type`: text describing the nature of event, can be used to filter classes of events, such as errors. 
* `text`: text describing specific even details. 

</details>

---

### `to_file` - Write record to a text file

Takes a record coming from a Chain, converts it to text representation,
and writes to a file in given directory. When a file already exists,
new records can be appended to the end of the file or overwrite it.

Output file name can be static or generated dynamically based on the template
filled with values from the record fields: every occurrence of `{text}`
in filename will be replaced with the value of the `text` field of processed
record, if the record has one.

Allows writing the record as human-readable text representation or as names and
values of the record fields in json format. For text representation it is possible
to provide a custom format template.

Produces `Event` with `error` type if writing to target file fails.

Note discrepancy between default value of `encoding` setting between `from_file`
and `to_file` plugins. Former is expected to be able to read files produced by
different software and therefore relies on system-wide settings. It would make
sense to do the same in the latter, but it would introduce possibility of failing
to write records containing text with Unicode codepoints that cannot be represented
using system-wide encoding.


#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `filename`: name of the output file. Supports templating with {...}. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `consume_record`: whether record should be consumed or passed down the chain after processing. Disabling it allows chaining multiple Actions. Default value is `true`.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.
* `path`: directory where output file should be created. Default is current directory. Supports templating with {...}. Not required.
* `encoding`: output file encoding. Default value is `utf8`.
* `output_format`: one of `str`, `repr`, `json`, `pretty_json`, `hash`. Default value is `text`.
* `output_template`: if provided, it will be used as a template to format processed record. Only works with `output_format` set to plain text. Not required.
* `missing`: if specified, will be used  to fill template placeholders that do not have corresponding fields in current record. Not required.
* `overwrite`: whether file should be overwritten in if it already exists. Default value is `true`.
* `append`: if true, new record will be written at the end of the file without overwriting already present lines. Default value is `true`.
* `prefix`: string that will be appended before the record text. Can be used to separate records from each other or for simple templating. Not required.
* `postfix`: string that will be appended after the record text. Not required.


#### Produced records types:


<details markdown="block">
  <summary>Event</summary>

Record produced by an internal event (usually error) inside the plugin


* `event_type`: text describing the nature of event, can be used to filter classes of events, such as errors. 
* `text`: text describing specific even details. 

</details>

---

### `download` - Download a file

Take an url from a field of a processed record with a name specified in `url_field`
and download it as a file to specified location.
The field must be present in the record and  must contain a valid url with a scheme
(such as "https") or a list of such urls.

Primarily designed for downloading images attached to a post, or thumbnails.
Does not support resuming interrupted downloads or detecting that this exact file is
already stored at target location without downloading it again.

File extension and name are inferred from HTTP headers and path part of the url,
unless provided explicitly with `extension` and `filename` parameters.
Since final file name is determined as a part of the download process, the file
is initially stored under a temporary name (currently an SHA1 of the url) in the
download directory, and then renamed to target filename.

If a file with given name already exists, depending on an `overwrite` setting a new file will either
overwrite it or get stored under different name, generated by combining
the base name with a number added as part of `rename_suffix`.
If, however, an exact copy of the new file is found among the files in target directory
sharing the base name, the new file will be deleted, giving preference to the existing copy.


#### Plugin configuration options:
* `max_concurrent_downloads`: limit for simultaneously active download tasks among all entities. Note that each entity will still process records sequentially regardless of this setting. Default value is `1`.
* `partial_file_suffix`: appended to a name of the file that is not yet completely downloaded. Default value is `.part`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `url_field`: field in the incoming record containing url of file to be downloaded. Required.
* `path`: directory where downloaded file should be created. Supports templating with {...}. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `consume_record`: whether record should be consumed or passed down the chain after processing. Disabling it allows chaining multiple Actions. Default value is `true`.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.
* `cookies_file`: path to a text file containing cookies in Netscape format. Not required.
* `headers`: custom HTTP headers as pairs "key": value". "Set-Cookie" header will be ignored, use `cookies_file` option instead. Not required.
* `filename`: name downloaded file should be stored under. If not provided will be inferred from HTTP headers or download url. Supports templating with {...} (additionally, "{source_name}" placeholder will be replaced with the inferred value). Not required.
* `extension`: normally file extension will be inferred from HTTP headers. This option allows to overwrite it. Not required.
* `overwrite`: whether file should be overwritten in if it already exists. If set to false will cause suffix with a number be added to the newly downloaded file name. Default value is `false`.
* `rename_suffix`: when overwriting is disabled, this suffix is attached to the base filename with the "{i}" part replaced with a number. Must contain "{i}" exactly once. Default value is ` [{i}]`.

---

### `xmpp` - Send records as Jabber messages

Converts records to a text representation and sends them as messages
to specified recipients. Sends each record in a separate message,
does not impose any limits on frequency or size of messages, leaving
it to server side.


#### Plugin configuration options:
* `xmpp_username`: JID of the account to be used to send messages, including resource. Required.
* `xmpp_pass`: password of the account to be used to send messages. Required.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `jid`: JID to send message to. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `consume_record`: whether record should be consumed or passed down the chain after processing. Disabling it allows chaining multiple Actions. Default value is `true`.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.

---

### `twitter.space` - Retrieve Twitter Space metadata from tweet

Take a record (normally coming from a Twitter monitor), check if it
has a link to a Twitter Space, and if so try to retrieve
additional information on the space, such as title and start time.

Produces a TwitterSpaceRecord if currently processed record
contains a link to a Space and the metadata was retrieved successfully.

It is possible to produce additional records with updated metadata at the
beginning and/or at the end of the space by toggling the `emit_*` settings.
However, a single state change should only produce one record. For example,
if a space has already ended before the first update, only a single record is
produced with all `emit_*` options enabled.


#### Plugin configuration options:
* `db_path`: path to the sqlite database file keeping history of old records.
Might specify a path to a directory containing the file (with trailing slash)
or a direct path to the file itself (without a slash). If special value `:memory:` is used,
database is kept in memory and not stored on disk at all, providing a clean database on every startup. Default value is `db/`.



#### Entity configuration options:
* `name`: name of a specific entity. Used to reference it in `chains` section. Must be unique within a plugin. Required.
* `cookies_file`: path to a text file containing cookies in Netscape format. Required.
##### 
* `reset_origin`: treat throughput records as if they have originated from this entity, 
emitting them in every Chain this entity is used. Default value is `false`.
* `consume_record`: whether record should be consumed or passed down the chain after processing. Disabling it allows chaining multiple Actions. Default value is `true`.
* `timezone`: takes timezone name from <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones> (or local time if omitted), converts record fields containing date and time to this timezone. Not required.
* `url`: Twitter domain name. Default value is `https://twitter.com`.
* `emit_immediately`: whether record should be produced immediately, regardless of media_url presence.
When disabled, only records with valid media_url are produced, depending on the following settings. Default value is `true`.
* `emit_on_live`: if enabled, a record is produced when livestream media_url becomes available. Default value is `true`.
* `emit_on_archive`: if enabled, a record is produced when archive media_url becomes available. Default value is `true`.
* `emit_on_end`: if enabled, a record is produced when the space ends, even if there is no archive. Default value is `false`.


#### Produced records types:


<details markdown="block">
  <summary>TwitterSpaceRecord</summary>

* `uid`: space id. 
* `url`: url of the space. 
* `state`: description of current status of a space: upcoming, ongoing, ended. 
* `media_key`: id that can be used to fetch url of the underlying HLS stream. 
* `media_url`: link to the underlying stream, when available. 
* `title`: space title. 
* `author`: user's visible name. 
* `username`: user's handle. 
* `avatar_url`: link to the picture used as the user's avatar. 
* `published`: timestamp of the space creation. 
* `scheduled`: scheduled time for an upcoming space to start at, otherwise absent. 
* `started`: timestamp of the space start, empty for upcoming spaces. 
* `ended`: timestamp of the space end, empty for not yet ended spaces. 
* `updated`: timestamp of the last update. 
* `recording_enabled`: whether host enabled recording of the space. When true, archive is likely to be available. 

</details>

