Metadata-Version: 2.1
Name: cwb-ccc
Version: 0.9.4
Summary: CWB wrapper to extract concordances and collocates
Home-page: https://gitlab.cs.fau.de/pheinrich/ccc
Author: Philipp Heinrich
Author-email: philipp.heinrich@fau.de
License: UNKNOWN
Description: # Collocation and Concordance Computation #
        
        ## Introduction ##
        This module is a wrapper around the [IMS Open Corpus Workbench
        (CWB)]([http://cwb.sourceforge.net/]).  Main purpose of the module is
        to extract concordance lines, calculate collocates, and run queries
        with several anchor points.
        
        If you want to extract the results of anchored queries with more than
        two anchor points, the module requires CWB version 3.4.16 or newer.
        
        ## Installation ##
        You can install this module with pip from PyPI:
        
        	pip3 install cwb-ccc
        
        You can also clone the repository from
        [gitlab.cs.fau.de](https://gitlab.cs.fau.de/pheinrich/ccc) and use
        `setup.py`:
        
            python3 setup.py install
        
        You can also just install all requirements specified in `setup.py` and
        make sure the `ccc` subfolder can be found by Python by including it
        in your `PYTHONPATH`.
        
        
        ## Usage ##
        
        ### CWBEngine
        All methods rely on the `CWBEngine` from `ccc.cwb`, which you first
        have to initialize with your system specific settings:
        
        ```python
        from ccc.cwb import CWBEngine
        engine = CWBEngine(
        	corpus_name="EXAMPLE_CORPUS",
        	registry_path="/path/to/your/cwb/registry"
        )
        ```
        
        This will raise a KeyError if the named corpus is not in the specified
        registry.
        
        You can use the `cqp_bin` to point the engine to a specific version of
        `cqp` (this is also helpful if `cqp` is not in your `PATH`) and change
        the `cache_path`:
        
        ```python
        engine = CWBEngine(
        	corpus_name="EXAMPLE_CORPUS",
        	registry_path="/path/to/your/cwb/registry", 
        	cqp_bin="/usr/local/cwb-3.4.16/bin/cqp",
        	cache_path="/tmp/ccc-cache"
        )
        ```
        
        If you are using macros and wordlists, you have to store them in a
        separate folder (with subfolders `wordlists` and `macros`).  Make sure
        you specify this folder via `lib_path` when initializing the
        engine:
        
        ```python
        engine = CWBEngine(
        	corpus_name="EXAMPLE_CORPUS", 
        	registry_path="/path/to/your/cwb/registry",
        	lib_path="/path/to/your/lib/"
        )
        ```
        
        ### Concordancing ###
        
        You can use the `Concordance` class from `ccc.concordances` for
        concordancing. The concordancer has to be initialized with the engine
        and accepts valid CQP queries:
        	
        ```python
        from ccc.concordances import Concordance
        # initialize the concordancer with the engine
        concordance = Concordance(engine)
        # extract concordance lines
        concordance.query('[lemma="Angela"] [lemma="Merkel"]')
        ```
        
        The result will be a dictionary with the _cpos_ of the match as keys
        and the entries one concordance line each. Each concordance line is
        formatted as a `pandas.DataFrame` with the _cpos_ of each token as
        index:
        
        | *cpos*    | word    | match | offset |
        |-----------|---------|-------|--------|
        | 188530363 | ,       | False | -5     |
        | 188530364 | dass    | False | -4     |
        | 188530365 | die     | False | -3     |
        | 188530366 | Tage    | False | -2     |
        | 188530367 | von     | False | -1     |
        | 188530368 | Angela  | True  | 0      |
        | 188530369 | Merkel  | True  | 0      |
        | 188530370 | gezählt | False | 1      |
        | 188530371 | sind    | False | 2      |
        | 188530372 | .       | False | 3      |
        
        The queries _must not_ end on a "within" clause.  If you want to
        restrict your concordance lines by a structural attribute, use the
        `s_break` parameter (defaults to "text"). The default context window
        is 20 tokens to the left and 20 tokens to the right of the query match
        and matchend, respectively.
        
        ```python
        concordance = Concordance(engine, context=50, s_break='s')
        concordance.query('[lemma="Angela"] [lemma="Merkel"]')
        ```
        
        Further parameters for the `Concordance` class are `order` (one of
        "random", "first", or "last"), `cut_off` (for the number of
        concordance lines to extract), and `p_show` (a `list` of additional
        p-attributes besides the primary word layer to show, e.g. "lemma" or
        "pos"; these will be added as additional columns).
        
        ### Collocation Analyses ###
        You can use the `Collocates` class to extract collocates for a given
        window size (symmetric windows around the corpus matches):
        
        ```python
        from ccc.collocates import Collocates
        # initialize the collocation calculator with the engine
        collocates = Collocates(engine)
        # extract collocates
        collocates.query('[lemma="Angela"] [lemma="Merkel"]', window=5)
        ```
        
        The result will be a `DataFrame` with lexical items (lemmas by
        default) as index and frequency signatures and association measures as
        columns:
        
        | *item*          | O11  | f2       | N         | f1    | O12   | O21      | O22       | E11         | E12          | E21          | E22          | log_likelihood | ... |
        |-----------------|------|----------|-----------|-------|-------|----------|-----------|-------------|--------------|--------------|--------------|----------------|-----|
        | die             | 1799 | 25125329 | 300917702 | 22832 | 21033 | 25123530 | 275771340 | 1906.373430 | 20925.626570 | 2.512342e+07 | 2.757714e+08 | -2.459194      | ... |
        | Bundeskanzlerin | 1491 | 8816     | 300917702 | 22832 | 21341 | 7325     | 300887545 | 0.668910    | 22831.331090 | 8.815331e+03 | 3.008861e+08 | 1822.211827    | ... |
        | .               | 1123 | 13677811 | 300917702 | 22832 | 21709 | 13676688 | 287218182 | 1037.797972 | 21794.202028 | 1.367677e+07 | 2.872181e+08 | 2.644804       | ... |
        | ,               | 814  | 17562059 | 300917702 | 22832 | 22018 | 17561245 | 283333625 | 1332.513602 | 21499.486398 | 1.756073e+07 | 2.833341e+08 | -14.204447     | ... |
        | Kanzlerin       | 648  | 17622    | 300917702 | 22832 | 22184 | 16974    | 300877896 | 1.337062    | 22830.662938 | 1.762066e+04 | 3.008772e+08 | 559.245198     | ... |
        
        For notation and further information, see
        [collocations.de](http://www.collocations.de/AM/index.html).
        
        By default, the dataframe is sorted by co-occurrence frequency (O11)
        and annotated with *z-score*, *t-score*, *dice*, *log-likelihood*,
        and *mutual information*. You can change these and further parameters
        when initializing the `Collocates` class:
        
        ```python
        collocates = Collocates(
        	engine,
        	max_window_size=20, 
        	s_break='text',
        	order='O11',
        	p_query='lemma', 
        	cut_off=100,
        	ams=['z_score', 't_score', 'dice', 'log_likelihood', 'mutual_information']
        )
        ```
        
        Available association measures depend on their implementation in the
        [association-measures](https://pypi.org/project/association-measures/)
        module.
        
        Note that the `max_window_size` parameter is an internal parameter
        that determines the result of the initial query via the
        `CWBEngine`. The result of the initial query is cached, which means
        that you can extract collocates for windows from 0 to
        `max_window_size` quickly.
        
        
        ### Anchored Queries ###
        
        The `Concordance` class detects anchored queries by default. The following query
        ```python
        concordance.query(
        	'@0[lemma="Angela"]? @1[lemma="Merkel"] [word="\\("] @2[lemma="CDU"] [word="\\)"]'
        )
        ```
        will thus return `DataFrame`s with an additional column indicating the
        anchor positions:
        
        | *cpos*    | word       | match | offset | anchor |
        |-----------|------------|-------|--------|--------|
        | 298906425 | auch       | False | -5     | None   |
        | 298906426 | das        | False | -4     | None   |
        | 298906427 | Handy      | False | -3     | None   |
        | 298906428 | von        | False | -2     | None   |
        | 298906429 | Kanzlerin  | False | -1     | None   |
        | 298906430 | Angela     | True  | 0      | 0      |
        | 298906431 | Merkel     | True  | 0      | 1      |
        | 298906432 | (          | True  | 0      | None   |
        | 298906433 | CDU        | True  | 0      | 2      |
        | 298906434 | )          | True  | 0      | None   |
        | 298906435 | sowie      | False | 1      | None   |
        | 298906436 | ihres      | False | 2      | None   |
        | 298906437 | Vorgängers | False | 3      | None   |
        | 298906438 | Gerhard    | False | 4      | None   |
        | 298906439 | Schröder   | False | 5      | None   |
        
        
        ### Argument Queries
        Argument queries are anchored queries with additional information. (1)
        Each anchor can be modified by an offset (usually used to capture
        underspecified tokens near an anchor point). (2) Anchors can be mapped
        to external identifiers for further logical processing. (3) Anchors
        may be given a clear name:
        
        
        | anchor | offset | idx  | clear name |
        |--------|--------|------|------------|
        | 0      | 0      | None | None       |
        | 1      | -1     | None | None       |
        | 2      | 0      | None | None       |
        | 3      | -1     | None | None       |
        
        
        Furthermore, several anchor points can be combined to form regions,
        which in turn can be mapped to identifiers and be given a clear name:
        
        | start | end | idx | clear name |
        |-------|-----|-----|------------|
        | 0     | 1   | "0" | "person X" |
        | 2     | 3   | "1" | "person Y" |
        
        
        Example: Given the definition of anchors and regions above as well as
        suitable wordlists, the following complex query extracts corpus
        positions where there's some correlation between "person X" (the
        region from anchor 0 to anchor 1) and "person Y" (anchor 2 to 3):
        
        ```python
        query = (
        	"<np> []* /ap[]* [lemma = $nouns_similarity] "
        	"[]*</np> \"between\" @0:[::](<np>[pos_simple=\"D|A\"]* "
        	"([pos_simple=\"Z|P\" | lemma = $nouns_person_common | "
        	"lemma = $nouns_person_origin | lemma = $nouns_person_support | "
        	"lemma = $nouns_person_negative | "
        	"lemma = $nouns_person_profession] |/region[ner])+ "
        	"[]*</np>)+@1:[::] \"and\" @2:[::](<np>[pos_simple=\"D|A\"]* "
        	"([pos_simple=\"Z|P\" | lemma = $nouns_person_common | "
        	"lemma = $nouns_person_origin | lemma = $nouns_person_support | "
        	"lemma = $nouns_person_negative | "
        	"lemma = $nouns_person_profession] | /region[ner])+ "
        	"[]*</np>) (/region[np] | <vp>[lemma!=\"be\"]</vp> | "
        	"/region[pp] |/be_ap[])* @3:[::]"
        )
        ```
        
        NB: the set of identifiers defined in the table of anchors and in the
        table of regions, respectively, should not overlap.
        
        It is customary to store these queries in json query files such as the
        [example](tests/gold/query-example.json) in the repository. You can
        directly process these files using the `process_argmin_file` method
        from `ccc.argmin`:
        
        ```python
        from ccc.argmin import process_argmin_file
        query_path = "tests/gold/query-example.json"
        result = process_argmin_file(engine, query_path)
        ```
        
        The result is a `dict` with the same keys as specified in the query
        file as well as an entry "result" with the following keys:
        
        - "nr_matches": the number of query matches in the corpus.
        - "matches": a list of concordance lines. Each concordance line
          contains 
          - the corpus position of the match (entry "position")
          - the actual concordance line as returned from
          `Concordance().query()` (see above) converted to a `dict` (entry
          "df")
          - a mapping from the idx specified in the anchor and region tables
          to the tokens or token sequences, respectively (default: lemma
          layer) (entry "holes")
          - a reconstruction of the concordance line as a sequence of tokens
            (word layer) (entry "full")
        - "holes": a global list of all tokens of the entities specified in
          the "idx" columns (default: lemma layer).
        
        
        ## Acknowledgements ##
        The module relies on
        [cwb-python](https://pypi.org/project/cwb-python/), special thanks to
        Yannick Versley and Jorg Asmussen for the implementation. Thanks to
        Markus Opolka for the implementation of
        [association-measures](https://pypi.org/project/association-measures/)
        and for forcing me to write tests.
        
        This work has been funded by the Deutsche Forschungsgemeinschaft (DFG)
        within the project "Reconstructing Arguments from Noisy Text", grant
        number 377333057, as part of the Priority Program "Robust
        Argumentation Machines (RATIO)" (SPP-1999).
        
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Operating System :: OS Independent
Requires-Python: >=3.5
Description-Content-Type: text/markdown
