RELIQ(1)

NAME
reliq - simple html searching tool

SYNOPSIS
reliq [OPTION...] PATTERNS [FILE...]

DESCRIPTION
reliq searches for PATTERNS in each FILE. Typically PATTERNS should be quoted when reliq is used in shell command.

When FILE isn't specified FILE will become standard input.

OPTIONS

  Generic Program Information

-h ,  --help
    Output a usage message and exit.

-v, --version
    Output the version number of reliq and exit.

  Modes of execution

--html
    execute expression on provided html (default)

--urljoin
    join urls passed in arguments with first argument url

-f, --file FILE
    Obtain pattern from FILE. The empty file contains zero patterns, and therefore matches nothing.

-e, --expression PATTERN
    use PATTERN instead of first input argument

--encode
    encode '&', '<', '>', '"', '\'' to html entities in input files

--encode-full
    encode all possible characters to html entities in input files

--decode
    decode html entities, while translating &nbsp; to space in input files

--decode-exact
    decode html entities in input files

  General Output Control

-o, --output FILE
    Change output to a FILE instead of stdout.

-E, --error-file FILE
    Change output of errors to a FILE instead of stderr.

  File and Directory Selection

-r, --recursive
    Read all files under each directory, recursively.

-R, --dereference-recursive
    Read all files under each directory, recursively. Follow all symbolic links, unlike -r.

  Other Options

-l ,  --list-structure
    set PATTERN to '| "%n%Ua - desc(%c) lvl(%L) size(%s) pos(%I)\n"'.

SYNTAX

  PATTERN
Consists of text that can be specified in ' and " brackets. It can be preceded by characters specifying the way of matching ended with '>' character. By default it's set to "tWfncs". After '>' character a range might be specified, which will match the number of characters in matched string. In such case specifying text is not mandatory. If unquoted text is set to a single '*' character matching of text will be omitted, leaving only matching by range.

TEXT
"TEXT"
'TEXT'
>'TEXT'
>TEXT             - match TEXT, same as above
[a-z]>TEXT        - match TEXT depending on options
[a-z]>[URANGE]TEXT
*                 - match to everything
>*                - same as above
>[URANGE]"TEXT"
>[URANGE]          - match length of pattern
>[URANGE]*         - same as above

Matching options:

t - Match trimmed
u - Match untrimmed
c - Be case sensitive
i - Be case insensitive
v - Invert matching
n - Match normally, without inversion
a - Match if it occurs at any position
f - Match if it fully matches
b - Match if it matches at the begining
e - Match if it matches at the end
w - Match it to any of the words in it
W - Match it to the Whole thing
s - Use string matching
B - Use BRE matching
E - Use ERE matching

  ATTRIBUTE_NAME
Attribute's name PATTERN. Starting by '+' or '-' will ensure that it will be treated as attribute, but in most cases can be omitted. '-' signifies its negation. It can be followed by [RANGE] which matches attribute's position in tag. ATTRIB_NAME has 'uWnsfi' pattern flags, PATTERN has 'tWncfs'. Attribute name can be shortened to '.' or '#' (see below). ATTRIB_NAME flags to 'uWnsfi' and 'uwncsf' for PATTERN.

PATTERN          - existing attribute
+PATTERN         - existing attribute
-PATTERN         - non existing attribute
.PATTERN         - same as i>class=w>PATTERN
#PATTERN         - same as i>id=w>PATTERN
[RANGE]PATTERN   - existing attribute at position determined by RANGE
-[RANGE]PATTERN
+.PATTERN
-[RANGE]#PATTERN

  ATTRIBUTE
Consists of ATTRIBUTE_NAME followed by '=' and PATTERN of attribute's value. Specifying only ATTRIBUTE_NAME without specifying its value equals to ignoring its value.

ATTRIBUTE - ignore value of attribute
ATTRIBUTE_NAME=PATTERN

  RANGE
Is always embedded in square brackets. Consists of groups of four numbers separated by ':', that can be practically endlessly separated by ',' if any of the matching succeedes the matching will stop. Specifying only one value equals to matching only to that value. Specifying two values equals to matching range between, and of them. If '!' is found before the first value the matching will be inversed. Empty values will be treated as infinity. Specifying three values additionally matches only values of which modulo of third value is equal to 0. Forth value is an offset to value from which modulo is calculated from.

  RRANGE
Relative range matches arrays. Putting '-' before two first values (even if they are not specified) makes them subtracted from the maximal value.

[x1,!x2,x3,x4]        - match to x1 or anything that isn't x2 or x3 or x4.
[x1:y1,x2:y2,!x3:y4]  - match to values between, and x1 and y1 or ...
[x:]                  - match values that are x or higher
[:y]                  - match values that are y or lower
[:]                   - match everything
[-]                   - match to last index of the array
[-x]                  - match to last index of the array subtracted by x
[:-y]                 - match to range from 0 to y'th value from end
[::w]                 - match to values from which modulo of w is equal to 0
[x:y:w]               - match to range from x to y from which modulo of w is equal to 0
[x:y:w:z]             - match to range from x to y with value increased by z from which modulo of w is equal to 0
[::2:1]               - match to uneven values

  URANGE
Unsigned range matches unsigned integers. Putting '-' before first value is the same as '0', before the second value is the same as matching to infinity.

[x1,x1:y]             - match to x1 or between, and x1 and y
[-x]                  - match to nothing
[-x:-y]               - match to everything
[-x:y]                - match from 0 to y
[:y]                  - match from 0 to y
[x:-y]                - match from x to infinity
[x:]                  - match from x to infinity
[x::2]                - match to even values starting from x

  SRANGE
Signed range matches signed integers.

[-x,-y]               - match from -x to -y
[:-y]                 - match from negative infinity to -y
[:y]                  - match from negative infinity to y
[x:]                  - match from x to infinity
[:-1:2:1]             - match to uneven values until -1

  HOOK
Begins with a name of function followed by '@' and ended with argument which can be a RANGE, EXPRESSION, PATTERN or nothing. Name of matching hook can be preceded with '+' or '-' character. If it's '-' matching of hook will be inverted.

NAME@PATTERN
NAME@[RANGE]
NAME@"EXPRESSION"
NAME@
+NAME@[RANGE]
-NAME@"EXPRESSION"

List of implemented hooks:
Global hooks, which can be used for any type:

L,  level [URANGE]
    Get nodes that are on level within URANGE.

l,  levelrelative [SRANGE]
    Get nodes that are on level relative to parent within the SRANGE.

c,  count [URANGE]
    Get nodes with count of descending tag nodes that is within the URANGE.

Ct,  counttext [URANGE]
    Get nodes with count of descending text nodes that is within the URANGE.

Cc,  countcomments [URANGE]
    Get nodes with count of descending comment nodes that is within the URANGE.

Ca,  countall [URANGE]
    Get nodes with count of descending nodes that is within the URANGE.

P,  position [URANGE]
    Get nodes with position within URANGE.

p,  positionrelative [SRANGE]
    Get nodes with position relative to parent within SRANGE.

I,  index [URANGE]
    Get nodes with starting index of tag in file that is within URANGE.

Hooks for tag type:

A,  all "PATTERN"
    Get tags with text creating them matching to PATTERN.
    Pattern flags set to "uWcnas".

i,  insides "PATTERN"
    Get tags with insides that match PATTERN.
    Pattern flags set to "tWncas".

S,  start "PATTERN"
    Get tags which head tag matches to PATTERN.
    Pattern flags set to "uWcnas".

e,  endstrip "PATTERN"
    Get tags which tail tag stripped from <> matches to PATTERN.
    Pattern flags set to "tWcnfs".

E,  end "PATTERN"
    Get tags which tail tag matches to PATTERN.
    Pattern flags set to "uWcnas".

n,  name "PATTERN"
    Get tags which name matches PATTERN. Used as TAG matcher.
    Pattern flags set to "uWinfs".

a,  attributes [URANGE]
    Get tags with attributes that are within the URANGE.

has "EXPRESSION"
    Get tags in which chained EXPRESSION matches at least one of its children.

Hooks for comment type:

A,  all "PATTERN"
    Get comments with text creating them matching to PATTERN. Used as TAG matcher.
    Pattern flags set to "tWncas".

i,  insides "PATTERN"
    Get comments with insides that match PATTERN.
    Pattern flags set to "tWncas".

Hooks for text types:

A,  all "PATTERN"
    Get text matching to PATTERN. Used as TAG matcher.
    Pattern flags set to "tWncas".

Access hooks specify what nodes will be matched, specifying multiple will not preserve order:

@   Specifying hook with no name is the same as self@

full
    Matches node itself and everything below it (set by default).

self
    Matches only node itself (similar to l@[0]).

child
    Matches nodes that are only one level higher then self (similar to l@[1]).

desc,    descendant
    Matches nodes that are lower than self (similar to l@[1:]).

ancestor
    Matches nodes that are ancestors of self, relative level of 0 matches to parent.

parent
    Matches node that is a parent of self.

rparent, relative_parent
    Matches node that matched self in script e.g. 'TAG1; TAG2; * rparent@' will return TAG1.

sibl,    sibling
    Matches siblings of self.

spre,    sibling_preceding.
    Matches preceding siblings of self.

ssub,    sibling_subsequent
    Matches subsequent siblings of self.

fsibl,   full_sibling
    Matches siblings of self and nodes below them.

fspre,   full_sibling_preceding
    Matches preceding siblings of self and nodes below them.

fssub,   full_sibling_subsequent
    Matches subsequent siblings of self and nodes below them.

everything
    Matches every node possible

preceding
    Matches nodes before current that are not it's ancestors

before
    Matches nodes before current

subsequent
    Matches nodes after current that are not it's descendants

after
    Matches nodes after current

Type hooks:

tag Matches tag nodes.

comment
    Matches comment nodes.

text
    Matches text nodes that are not of textempty type.

textempty
    Matches text nodes consisting of only whitespaces.

texterr
    Matches text nodes in which error parsing html has occurred

textnoerr
    Matches text nodes that are not textempty or texterr

textall
    Matches all text nodes

  TAG
At the begining each TAG must contain PATTERN of html tag and that can be followed by a number of ATTRIBUTEs and HOOKs. Range separated by spaces will match the position of results relative to parent nodes, or if specified before tag PATTERN absolute to all results.

PATTERN
PATTERN ATTRIBUTE... HOOK... [RANGE] - match RANGE to results relative to parent nodes
[RANGE] PATTERN - match RANGE to results

TAG, ATTRIBUTEs and HOOKs can be grouped in '(' ')' brackets. ')' has to be preceded by space otherwise it will be treated as part of argument.

    (... ) - correct
    ( ... ) - correct
    ( ...) - incorrect
    ( ( ... )(... )) - correct

If brackets are 'touching' themselves they will match if at least one of them matches. Groups cannot contain position or access hooks definitions. If TAG pattern is not specified before any groups then all of the first groups will specify it.

    TAG ( ATTRIB1 HOOK )( ATTRIB2 ( ATTRIB3 ATTRIB4 )( ATTRIB5 ) ) - TAG having either ATTRIB1 with HOOK or ATTRIB2 which has ATTRIB3 and ATTRIB4 or ATTRIB5
    TAG ( ATTRIB1 ) ( ATTRIB2 ) - TAG having both ATTRIB1 and ATTRIB2. Since groups have space in between they are not 'touching'.
    ( TAG1 HOOK )( TAG2 ) ATTRIB - either TAG1 with HOOK or TAG2 both having ATTRIB.

To match multiple types you have to specify type hooks as the first thing in the group. You can specify multiple types but only if they are in separate groups or can be inherited from each other.

    ( div )( comment@ DOCTYPE )( text@ have ) l@[0] - this matches to multiple types, since l@ is a global hook it can be specified outside the groups
    textall@ have ( @text seen )( texterr@ you ) - text and texterr inherit from textall
    (( p )( tag@ li ))( textall@ li ( text@ look ( textnoerr@ list )( texterr@ raise ))( textempty@ "  " ))

  TAG_FORMAT
It has to be specified in '"' or '\'' quotes. If format is not specified it will be set to "%C\n". i, t, T, C, a, v S, E, e directives can be preceded with 'U' and 'D' characters that change their output e.g. '%Ui', '%(href)DUa', '%1UDa'. 'U' causes output to be untrimmed (by default they are trimmed), 'D' decodes html escape codes. Prints output according to FORMAT interpreting '\' escapes and `%' directives. The escapes and directives are:

    \a  Alarm bell.

    \b  Backspace.

    \f  Form feed.

    \n  Newline.

    \r  Carriage return.

    \t  Horizontal tab.

    \v  Vertical tab.

    \0  ASCII NUL.

    \0XXX
        Byte in octal.

    \xXX
        Byte in hexadecimal.

    \uXXXX
        UTF-8 character (uses hexadecimal digits).

    \UXXXXXXXX
        UTF-8 character (uses hexadecimal digits).

    \\  A literal backslash (`\').

    %%  A literal percent sign.

    %l  Level in html structure relative to parent.

    %L  Level in html structure.

    %P  Position of node.

    %p  Position of node relative to parent.

    %I  Index of node's starting position in file.

    %c  Node's count of tags nodes descending.

    %Ct Node's count of text nodes descending.

    %Cc Node's count of comment nodes descending.

    %Ca Node's count of descendants, basically sum of the previous ones.

    %A  Contents of node.

    %i  Node's insides.

    %n  Tag's name.

    %S  Starting tag.

    %E  Ending tag.

    %e  Stripped ending tag.

    %t  Tag's text.

    %T  Tag's text, recursive.

    %s  Node's size.

    %a  All of the tag's attributes.

    %v  Values of tag's attributes separated with '"'.

    %kv Value of tag's attribute, where k is its position counted from zero.

    %(k)v
        Value of tag's attribute, where k is its case insensitive name.

  FUNCTION
Begins with name followed by arguments separated by whitespaces. FUNCTION can have up to 4 arguments that are clearly defined in [] brackets or in "",'' quotes.

NAME - function with no arguments
NAME [list] "text1" "text2" - function with first argument as a list, and second and third as text

List of implemented functions:
line [SELECTED] "DELIM"

    Return selected lines.
    Lines are separated by DELIM (by default '\n').

trim "DELIM"

    Trim whitespaces at the beginning and end of the whole input.
    Input can be split by DELIM and trimmed separately.

sort "FLAGS" "DELIM"

    Sort input delimited by DELIM (by default '\n').
    Flags:
        r - reverse the results of comparison
        u - omit repeated lines

uniq "DELIM"

    Filter out repeating lines from input delimited by DELIM (by default '\n').

echo "TEXT1" "TEXT2"

    Print TEXT1 before the input and TEXT2 after.

tr "STR1" "STR2" "FLAGS"

    Translate characters in STR1 to STR2.
    If STR2 is not specified characters in STR1 will be deleted.
    Special STR syntax:
        CHAR1-CHAR2 - all characters from CHAR1 to CHAR2 in descending order
        [CHAR*REPEAT] - REPEAT copies of CHAR
        [:space:] - support for common character types, written in lower case
    Flags:
        s - replace repeating sequences of characters with only one
        c - use the complement of STR1

cut [SELECTED] "DELIMS" "FLAGS" "DELIM"

    Return selected parts from input delimited by DELIM (by default '\n').
    DELIMS specifies delimiters for fields and selects fields, if none are specified selection is based on bytes.
    DELIMS have the same syntax as tr STR.
    Flags:
        s - do not return lines with no delimiters
        c - complement LIST
        z - sets DELIM to '\0'

sed "SCRIPT" "FLAGS" "DELIM"

    Implementation of sed(1).
    Lines are delimited by DELIM (by default '\n').
    Flags:
        n - suppress automatic printing of pattern space
        z - set DELIM to '\0'
        E - use extended regexp
    Deviations from standard:
        i c a commands do nothing even though they take arguments
        l r R Q w W are not implemented

rev "DELIM"

    Reverse order of characters in every line delimited by DELIM (by default '\n').

tac "DELIM"

    Reverse order of input lines delimited by DELIM (by default '\n').

wc "FLAGS" "DELIM"

    Print count of lines, words, bytes.
    Input is delimited by DELIM (by default '\n').
    Flags:
        l - lines count
        w - words count
        c - bytes count
        L - size of the longest line
    If multiple values are returned each will be separated with '\t'.
    If no flags are given then flags are set to "lwc".

decode "FLAGS"

    Decode html entities while translating &nbsp; to space.
    Flags:
        e - decode &nbsp; correctly

encode "FLAGS"

    Encode '&', '<', '>', '"', '\'' to html entities
    Flags:
        f - encode all possible characters to html entities

  FORMAT
Consists of FUNCTIONs separated by whitespace. Output of the tag is passed to FUNCTION, and its output is passed to the next until the last one which will print it. If specified after '|' FORMAT will be executed separately for each matched tag. If after '/' FORMAT will be executed for the whole output.

FUNCTION FUNCTION...

  NODE
Consists of TAGs and EXPRESSIONs separated by ';' which makes them pass result from previous node to the next. Output FORMAT can be specified after '|' and '/' characters, everything after it will be taken as FORMAT.

TAG1; TAG2; NODE - matches results of TAG1 by TAG2 and by NODE
NODE1; NODE2 - process the results of NODE1 by NODE2

  EXPRESSION
Consists of NODEs separated by ',' and grouped in '{' '}' brackets (which accumulate their output and increases their level).

NODE1, NODE2 - two expressions
EXPRESSION1; { EXPRESSION2; {EXPRESSION3, EXPRESSION4}, EXPRESSION5}; EXPRESSION6

  OUTPUT_FORMAT
Is changed based on output FORMAT and can be specified only to the last EXPRESSIONs, or if they are children of groups having FORMAT. Output FORMAT can be specified after '|' and '/' characters, everything after it will be taken as FORMAT. If the first thing in FORMAT specified after '|' character of a node is a "TEXT" it will be used as TAG_FORMAT. FORMAT after '|' character is executed on each element matched, and after '/' to the whole result. Groups with format after '|' will execute their EXPRESSIONs for each element in input independently, contrary to normal behavior where the child processes all the input at once before going to the next.

NODE1; NODE2 | TAG_FORMAT FORMAT1 / FORMAT2 - matches NODE2 to results of NODE1, then prints them one by one with TAG_FORMAT and processes WITH FORMAT1, then processes everything by FORMAT2
NODE1 | FORMAT, NODE2 / FORMAT
NODE1; { NODE1 | FORMAT, NODE2 / FORMAT }
NODE1; { NODE1 | FORMAT, NODE2 / FORMAT } / FORMAT - process results of GROUP
NODE1; { NODE1 | FORMAT, NODE2 / FORMAT } | FORMAT / FORMAT - process results of GROUP one by one, and then as a whole

  OUTPUT_FIELD
Accumulates output and prints it in json format. Begins before EXPRESSION, starts with '.' character and is followed by name, which can be defined as [A-Za-z0-9_-]+. If field doesn't have a name it will be a protected field i.e. if the EXPRESSION matches nothing a newline will be printed (deprecated feature that will be removed in the future). To specify type of field the name has to be followed by '.' and type name, the same rules of naming apply as to the field names. List of types:
    .s string, returns the exact input, set by default for fields without types
    .s(uint) string that has size greater than first argument
    .s(uint,uint) string that has size within range of first and second argument
    .b boolean value, return true only if first found non whitespace character is 'y', 'Y', 't', 'T' or number higher than 0, otherwise returns false. Fails only if input is empty
    .n number, return the first found floating point number, if none found or input is empty fails with 0
    .n(num) same as above but found number is greater than the first argument
    .n(num,num) same as above but found number is within range of first and second arguments
    .i integer, analogous to .n
    .i(int)
    .i(int,int)
    .u unsigned integer, analogous to .n
    .u(uint)
    .u(uint,uint)
    .d date, fails returning input
    .d(str...) date, fails if input doesn't match to any of arguments that are strptime(3) format otherwise converts it into iso 8081 date
    .U url, join url in input with main url specified by --url option
    .U(str) url, join url in input with argument
    .N null, never fails
    .e error, never fails
    .e(...) error, arguments are used for annotation
    .a array, of strings, delimited by '\n'
    .a("\t") array of strings, delimited by the first character in the argument, i.e. '\t'
    .a.type array of type, delimited by '\n'
    .a("-").type array of type, delimited by the first character in the argument (only one can be specified) Examples:
    if field doesn't return to a field is globally available, and even if div .author is not found the fields will be printed.
    .title h2, div .author; { .image img, .bolded b }
    {"title":"...","image":"...","bolded":"..."}
    if field has fields as an input it is an object.
    .title h2, .author div .author; { .image img, .bolded b }
    {"title":"...","author":{"image":"...","bolded":"..."}}
    if field has fields as an input and expressions without fields, first it prints out result of expressions, and then prints the object.
    div .author; { .image img, .bolded b, a }
    <a objects>
    {"author":{"image":"...","bolded":"..."}}
    blocks ended with '|' character are treated as arrays. If such block has no input it returns "[]".
    .authors div .author; { .images img, .bolded b } |
    {"authors":[{"images":"...","bolded":"..."}]} If after type name is '|' character another type can be specified. In this case if field fails type after it is attempted. If input expression matched to nothing type will always fail and output of the last type in such chain will be returned. Examples:
    .tag2 and .tag3 have results of num and null types failing since expression did not match to anything
    .tag.s nothing, .tag2.s|n nothing, .tag3.s|N nothing
    {"tag":"","tag2":0,"tag3":null}
    .title2 is matched by int since title input doesn't fit the assertion
    .title.s(1)|N h2, .title2.s(10,50)|i h2
    {"title":"...","title2":-123}
    after specifying type for array all alternative types will be assigned for array's elements since array can't fail
    .title.s(9999)|a("\t").s(2,5)|u|N h2
    {"title":[52,"...","...",null,0]} If immediately after field a string in '\'' or '"' brackets is specified it'll be treated as annotation e.g. '.u"annotation"', '.t.n(12)|N"annotation"'

  COMMENTS
Basic C style comments are implemented, although they have to be preceded by whitespace. Since sed quite often has 's/ *//g', haskell comments are implemented, they also have to be preceded by whitespace.

  CONDITIONALS
Are placed in between expressions and have to be surrounded by whitespace. They influence execution of the second expression based on success of the first.

& - execute second expression if first succeeds
&& - execute second expression if first succeeds and ignore output of the first
|| - execute second expression if first fails
^&
^&&
^||

Expression succeeds if anything inside of it matches html, e.g. { div, li | "" } && tr | "%i" will print insides of tr tag if either div or li tags are found, even though li tag outputs nothing it will match because only matching html matters.
If the described conditionals are preceded by '^' they will succeed only if everything matches, e.g. { div, li | "" } ^&& tr | "%i" both div and li would have to match for tr | "%li" to happen.
Output fields take precedence over conditionals so .name a .user | "%i" / tr "\n" || h1 | "%i" is the same as .name { a .user | "%i" / tr "\n" || h1 | "%i" }.
Output fields cannot be embedded in conditionals e.g. { .likes.u * #likes | "%i" } & { .dlikes.u * #dislikes | "%i" }.

NOTES
UTF-8 \u and \U escape directives don't work in any place where delimiter is to be specified i.e. array separator in fields, in translation tables of tr, in y sed command, in delimiters of sed, tr, line, trim, uniq, sort, echo, cut, rev, tac, wc commands.

EXAMPLES
Get tags 'a' with attribute 'href' at position 0 of value ending with '.org' or '.com', from result of matching tags 'div' with attribute 'id', and without attribute 'class', from file 'index.html'.

$ reliq 'div id -class; a [0]href=E>".*\\.(org|com)"' index.html

Get tags that don't have any tags inside them

$ reliq '* c@[0]' index.html

Get empty tags.

$ reliq '* m@>[0]' index.html

Get hyperlinks from level greater or equal to 6 (counting from zero).

$ reliq 'a href @l[6:] | %(href)v\\n ' index.html

Get all urls from 'a' and 'img' tags, where images end with '.png'.

$ reliq 'img src=e>.png | %(src)v\\n , a href | %(href)v\\n ' index.html

Get all urls in div with class 'index' or ul both with title attribute.

$ reliq '( div .index )( ul ) title; a href | %(href)v\\n' index.html

Decode html entities from index.html

$ reliq --decode index.html

Join urls with 'https://wikipedia.com/'

$ reliq --urljoin 'https://wikipedia.com/' '/' 'index.html' './name/../../list/genres/a.html'

EXIT STATUS

0
    success
5
    system call failure
10
    mangled html
15
    incorrect script

AUTHOR
Dominik Stanisław Suchora <hexderm@gmail.com>