# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['sirmordred', 'sirmordred.bin', 'sirmordred.utils', 'tests']

package_data = \
{'': ['*'], 'sirmordred.utils': ['logs/*'], 'tests': ['archives/*', 'data/*']}

install_requires = \
['cereslib>=0.3',
 'colorlog==4.1.0',
 'elasticsearch-dsl==6.3.1',
 'elasticsearch==6.3.1',
 'file-read-backwards==2.0.0',
 'graal>=0.3',
 'grimoire-elk>=0.102',
 'grimoirelab-panels>=0.1',
 'grimoirelab-toolkit>=0.3',
 'kidash>=0.5',
 'perceval-mozilla>=0.3',
 'perceval-opnfv>=0.2',
 'perceval-puppet>=0.2',
 'perceval-weblate>=0.2',
 'perceval>=0.19',
 'sortinghat>=0.7.20']

entry_points = \
{'console_scripts': ['healthcheck.py = sirmordred.utils.healthcheck:main',
                     'micro.py = sirmordred.utils.micro:main',
                     'panels_config.py = sirmordred.utils.panels_config:main',
                     'sirmordred = sirmordred.bin.sirmordred:main']}

setup_kwargs = {
    'name': 'sirmordred',
    'version': '0.5.1',
    'description': 'Drive GrimoireLab tools to produce a dashboard',
    'long_description': '# SirMordred [![Build Status](https://github.com/chaoss/grimoirelab-sirmordred/workflows/tests/badge.svg)](https://github.com/chaoss/grimoirelab-sirmordred/actions?query=workflow:tests+branch:master+event:push) [![Coverage Status](https://coveralls.io/repos/github/chaoss/grimoirelab-sirmordred/badge.svg?branch=master)](https://coveralls.io/github/chaoss/grimoirelab-sirmordred?branch=master) [![PyPI version](https://badge.fury.io/py/sirmordred.svg)](https://badge.fury.io/py/sirmordred)\n\nSirMordred is the tool used to coordinate the execution of the GrimoireLab platform, via two main configuration files, the `setup.cfg` and `projects.json`, which are summarized in their corresponding sections.\n\n## Contents\n\n* [Setup.cfg](#setupcfg-)\n* [Projects.json](#projectsjson-)\n* [Supported data sources](#supported-data-sources-)\n* [Micro Mordred](#micro-mordred-)\n* [Getting started](/Getting-Started.md#getting-started-)\n* [Troubleshooting](/Getting-Started.md#troubleshooting-)\n* [How to](/Getting-Started.md#how-to-)\n\n\n## Setup.cfg [&uarr;](#contents)\n\nThe setup file holds the configuration to arrange all processes underlying GrimoireLab. It is composed of sections that allow defining the general settings such as which phases to activate (e.g., collection, enrichment) and where to store the logs, as well as the location and credentials for SortingHat and the ElasticSearch instances where the raw and enriched data is stored. Furthermore, it also includes backend sections to set up the parameters used by Perceval to access the software development tools (e.g., GitHub tokens, Gerrit username) and fetch their data.\n\nDashboards can be automatically uploaded via the `setup.cfg` if the phase `panels` is enabled. The `Data Status` and `Overview` dashboards will contain\nwidgets that summarize the information of the data sources declared in the `setup.cfg`. Note that the widgets are not updated when adding\nnew data sources, thus you need to manually delete the dashboards `Data Status` and `Overview` (under **Stack Management > Saved Objects** in Kibiter), and restart mordred again (making sure that the option `panels` is enabled).\n\n### [es_collection]\n\n * **url** (str: http://172.17.0.1:9200): Elasticsearch URL (**Required**)\n### [es_enrichment]\n\n * **autorefresh** (bool: True): Execute the autorefresh of identities\n * **autorefresh_interval** (int: 2): Time interval (days) to autorefresh identities\n * **url** (str: http://172.17.0.1:9200): Elasticsearch URL (**Required**)\n### [general]\n\n * **bulk_size** (int: 1000): Number of items to write in Elasticsearch using bulk operations\n * **debug** (bool: True): Debug mode (logging mainly) (**Required**)\n * **logs_dir** (str: logs): Directory with the logs of sirmordred (**Required**)\n * **min_update_delay** (int: 60): Short delay between tasks (collect, enrich ...)\n * **scroll_size** (int: 100): Number of items to read from Elasticsearch when scrolling\n * **short_name** (str: Short name): Short name of the project (**Required**)\n * **update** (bool: False): Execute the tasks in loop (**Required**)\n * **aliases_file** (str: ./aliases.json): JSON file to define aliases for raw and enriched indexes\n * **menu_file** (str: ./menu.yaml): YAML file to define the menus to be shown in Kibiter\n * **global_data_sources** (list: bugzilla, bugzillarest, confluence, discourse, gerrit, jenkins, jira): List of data sources collected globally, they are declared in the section \'unknown\' of the projects.json\n * **retention_time** (int: None): the maximum number of minutes wrt the current date to retain the data\n * **update_hour** (int: None): The hour of the day the tasks will run ignoring `min_update_delay` (collect, enrich ...)\n### [panels]\n\n * **community** (bool: True): Include community section in dashboard\n * **kibiter_default_index** (str: git): Default index pattern for Kibiter\n * **kibiter_time_from** (str: now-90d): Default time interval for Kibiter\n * **kibiter_url** (str): Kibiter URL (**Required**)\n * **kibiter_version** (str: None): Kibiter version\n * **kafka** (bool: False): Include KIP section in dashboard\n * **github-comments** (bool: False): Enable GitHub comments menu. Note that if enabled, the github2:issue and github2:pull sections in the setup.cfg and projects.json should be declared\n * **github-events** (bool: False): Enable GitHub events menu. Note that if enabled, the github:event section in the setup.cfg and projects.json should be declared\n * **github-repos** (bool: False): Enable GitHub repo stats menu. Note that if enabled, the github:repo section in the setup.cfg and projects.json should be declared\n * **gitlab-issues** (bool: False): Enable GitLab issues menu. Note that if enabled, the gitlab:issue section in the setup.cfg and projects.json should be declared\n * **gitlab-merges** (bool: False): Enable GitLab merge requests menu. Note that if enabled, the gitlab:merge sections in the setup.cfg and projects.json should be declared\n * **mattermost** (bool: False): Enable Mattermost menu\n * **code-license** (bool: False): Enable code license menu. Note that if enabled, colic sections in the setup.cfg and projects.json should be declared\n * **code-complexity** (bool: False): Enable code complexity menu. Note that if enabled, cocom sections in the setup.cfg and projects.json should be declared\n * **strict** (bool: True): Enable strict panels loading\n * **contact** (str: None): Support repository URL\n### [phases]\n\n * **collection** (bool: True): Activate collection of items (**Required**)\n * **enrichment** (bool: True): Activate enrichment of items (**Required**)\n * **identities** (bool: True): Do the identities tasks (**Required**)\n * **panels** (bool: True): Load panels, create alias and other tasks related (**Required**)\n\n### [projects]\n\n * **projects_file** (str: projects.json): Projects file path with repositories to be collected grouped by projects\n * **projects_url** (str: None): Projects file URL, the projects_file is required to store the file locally\n### [sortinghat]\n\n * **affiliate** (bool: True): Affiliate identities to organizations (**Required**)\n * **autogender** (bool: False): Add gender to the profiles (executes autogender)\n * **autoprofile** (list: [\'customer\', \'git\', \'github\']): Order in which to get the identities information for filling the profile (**Required**)\n * **database** (str: sortinghat_db): Name of the Sortinghat database (**Required**)\n * **host** (str: mariadb): Host with the Sortinghat database (**Required**)\n * **identities_api_token** (str: None): API token for remote operation with GitHub and Gitlab\n * **identities_export_url** (str: None): URL in which to export the identities in Sortinghat\n * **identities_file** (list: []): File path with the identities to be loaded in Sortinghat\n * **identities_format** (str: sortinghat): Format of the identities data to be loaded\n * **load_orgs** (bool: False):\n * **matching** (list: [\'email\']): Algorithm for matching identities in Sortinghat (**Required**)\n * **orgs_file** (str: None): File path with the organizations to be loaded in Sortinghat\n * **password** (str: ): Password to access the Sortinghat database (**Required**)\n * **reset_on_load** (bool: False): Unmerge and remove affiliations for all identities on load\n * **sleep_for** (int: 3600): Delay between task identities executions (**Required**)\n * **strict_mapping** (bool: True): rigorous check of values in identities matching (i.e, well formed email addresses, non-overlapping enrollment periods)\n * **unaffiliated_group** (str: Unknown): Name of the organization for unaffiliated identities (**Required**)\n * **user** (str: root): User to access the Sortinghat database (**Required**)\n### [backend-name:tag] (tag is optional)\n\n* **collect** (bool: True): enable/disable collection phase\n* **raw_index** (str: None): Index name in which to store the raw items (**Required**)\n* **enriched_index** (str: None): Index name in which to store the enriched items (**Required**)\n* **studies** (list: []): List of studies to be executed\n* **anonymize** (bool: False): enable/disable anonymization of personal user information\n* **backend-param-1**: ..\n* **backend-param-2**: ..\n* **backend-param-n**: ..\n\nThe template of a backend section is shown above.\nFurther information about Perceval backends parameters are available at:\n\n* Params details: https://perceval.readthedocs.io/en/latest/perceval/perceval-backends.html\n* Examples: https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfg\n\nNote that some backend sections allow to specify specific enrichment options, which are listed below.\n\n### [jenkins]\n* **node_regex**: regular expression for extracting node name from `builtOn` field. This\n  regular expression **must contain at least one group**. First group will be used to extract\n  node name. More groups are allowed but not used to extract anything else.\n\n### [studies-name:tag] (tag is optional)\n\n* **study-param-1**: ..\n* **study-param-2**: ..\n* **study-param-n**: ..\n\nA template of a study section is shown above. A complete list of studies parameters is available at:\n\n* https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfg\n\n## Projects.json [&uarr;](#contents)\n\nThe projects.json aims at describing the repositories grouped by a project that will be shown on the dashboards.\n\nThe project file enables the users to list the instances of the software development tools to analyse, such\nas local and remote Git repositories, the URLs of GitHub and GitLab issue trackers, and the name of Slack channels.\nFurthermore, it also allows the users to organize these instances into nested groups, which structure is reflected in\nthe visualization artifacts (i.e., documents and dashboards). Groups can be useful to represent projects within a single\ncompany, sub-projects within a large project such as Linux and Eclipse, or the organizations within a collaborative project.\n\n1. First level: project names\n2. Second level: data sources and metadata\n3. Third level: data source URLs\n\nThere are some filters, labels, and a special section:\n* Filter `--filter-no-collection=true`: This filter is used to show old enriched data within the dashboards from\nrepositories that don\'t exist anymore in upstream.\n* Filter `--filter-raw` and the section `unknown`: The data sources will only collected at the section `unknown`\nbut this allow to add the same source in different sections to enrich using the filter `--filter-raw`.\n* Label ` --labels=[example]`: The data source will have the label of `example` which can be used to create visualisations for specific sets of data\n* Section `unknown`: If the data source is only under this section it will be enriched as project `main`.\n\n```\n{\n    "Chaoss": {\n        "gerrit": [\n            "gerrit.chaoss.org --filter-raw=data.projects:CHAOSS"\n        ]\n        "git": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https://<username>:<api-token>@github.com/chaoss/grimoirelab-sirmordred"\n        ],\n        "github": [\n            "https:/github.com/chaoss/grimoirelab-perceval --filter-no-collection=true",\n            "https:/github.com/chaoss/grimoirelab-sirmordred  --labels=[example]"\n        ]\n    },\n    "GrimoireLab": {\n        "gerrit": [\n            "gerrit.chaoss.org --filter-raw=data.projects:GrimoireLab"\n        ],\n        "meta": {\n            "title": "GrimoireLab",\n            "type": "Dev",\n            "program" : "Bitergia",\n            "state": "Operating"\n        },\n    }\n    "unknown": {\n        "gerrit": [\n            "gerrit.chaoss.org"\n        ],\n        "confluence": [\n            "https://wiki.chaoss.org"\n        ]\n}\n```\nIn the projects.json above:\n* The data included in the repo `gerrit.chaoss.org` will be collected entirely since the\nrepo is listed in the `unknown` section. However only the project `GrimoireLab` will be enriched as declared in the\n`GrimoireLab` section.\n* In the section `Chaoss` in the data source `github` the repository `grimoirelab-perceval` is not collected for\nraw index but it will enriched in the enriched index.\n* In the section `GrimoireLab` the metadata will showed in the enriched index as extra fields.\n* In the section `unknown` the data source `confluence` will be enriched as the project `main`.\n\n## Supported data sources [&uarr;](#contents)\n\nThese are the data sources GrimoireLab supports: [askbot](#askbot-), [bugzilla](#bugzilla-), [bugzillarest](#bugzillarest-), [cocom](#cocom-), [colic](#colic-), [confluence](#confluence-), [crates](#crates-), [discourse](#discourse-), [dockerhub](#dockerhub-), [dockerdeps](#dockerdeps-), [dockersmells](#dockersmells-), [functest](#functest-), [gerrit](#gerrit-), [git](#git-), [github](#github-), [github2](#github2-), [gitlab](#gitlab-), [gitter](#gitter-), [google_hits](#google_hits-), [groupsio](#groupsio-), [hyperkitty](#hyperkitty-), [jenkins](#jenkins-), [jira](#jira-), [kitsune](#kitsune-), [mattermost](#mattermost-), [mbox](#mbox-), [mediawiki](#mediawiki-), [meetup](#meetup-), [mozillaclub](#mozillaclub-), [nntp](#nntp-), [pagure](#pagure-), [phabricator](#phabricator-), [pipermail](#pipermail-), [puppetforge](#puppetforge-), [redmine](#redmine-), [remo](#remo-), [rocketchat](#rocketchat-), [rss](#rss-), [slack](#slack-), [stackexchange](#stackexchange-), [supybot](#supybot-), [telegram](#telegram-), [twitter](#twitter-)\n\n#### askbot [&uarr;](#supported-data-sources-)\nQuestions and answers from Askbot site\n- projects.json\n```\n{\n    "Chaoss": {\n        "askbot": [\n            "https://askbot.org/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[askbot]\nraw_index = askbot_raw\nenriched_index = askbot_enriched\n```\n#### bugzilla [&uarr;](#supported-data-sources-)\nBugs from Bugzilla\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "bugzilla": [\n            "https://bugs.eclipse.org/bugs/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[bugzilla]\nraw_index = bugzilla_raw\nenriched_index = bugzilla_enriched\nbackend-user = yyyy (optional)\nbackend-password = xxxx (optional)\nno-archive = true (suggested)\n```\n#### bugzillarest [&uarr;](#supported-data-sources-)\nBugs from Bugzilla server (>=5.0) using its REST API\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "bugzillarest": [\n            "https://bugzilla.mozilla.org"\n        ]\n    }\n}\n```\n\n- setup.cfg\n```\n[bugzillarest]\nraw_index = bugzillarest_raw\nenriched_index = bugzillarest_enriched\nbackend-user = yyyy (optional)\nbackend-password = xxxx (optional)\nno-archive = true (suggested)\n```\n#### cocom [&uarr;](#supported-data-sources-)\nCode complexity integration.\nSome graal dependencies like `cloc` might be required, https://github.com/chaoss/grimoirelab-graal#how-to-installcreate-the-executables\n\n- projects.json\n```\n{\n    "Chaoss":{\n        "cocom": [\n            "https://github.com/chaoss/grimoirelab-toolkit"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[cocom]\nraw_index = cocom_chaoss\nenriched_index = cocom_chaoss_enrich\ncategory = code_complexity_lizard_file\nstudies = [enrich_cocom_analysis]\nbranches = master\nworktree-path = /tmp/cocom/\n```\n#### colic [&uarr;](#supported-data-sources-)\nCode license backend.\n- projects.json\n```\n{\n    "Chaoss":{\n        "colic": [\n            "https://github.com/chaoss/grimoirelab-toolkit"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[colic]\nraw_index = colic_chaoss\nenriched_index = colic_chaoss_enrich\ncategory = code_license_nomos\nstudies = [enrich_colic_analysis]\nexec-path = /usr/share/fossology/nomos/agent/nomossa\nbranches = master\nworktree-path = /tmp/colic\n```\n#### confluence [&uarr;](#supported-data-sources-)\ncontents from Confluence\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "confluence": [\n            "https://wiki.open-o.org/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[confluence]\nraw_index = confluence_raw\nenriched_index = confluence_enriched\nno-archive = true (suggested)\n```\n#### crates [&uarr;](#supported-data-sources-)\npackages from Crates.io\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "crates": [\n            ""\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[crates]\nraw_index = crates_raw\nenriched_index = crates_enriched\n```\n#### discourse [&uarr;](#supported-data-sources-)\nTopics from Discourse\n- projects.json\n```\n{\n    "Chaoss": {\n        "discourse": [\n            "https://foro.mozilla-hispano.org/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[discourse]\nraw_index = discourse_raw\nenriched_index = discourse_enriched\nno-archive = true (suggested)\n```\n#### dockerhub [&uarr;](#supported-data-sources-)\nRepositories info from DockerHub\n- projects.json\n```\n{\n    "Chaoss": {\n        "dockerhub": [\n            "bitergia kibiter"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[dockerhub]\nraw_index = dockerhub_raw\nenriched_index = dockerhub_enriched\nno-archive = true (suggested)\n```\n#### dockerdeps [&uarr;](#supported-data-sources-)\nDependencies extracted from Dockerfiles. Requires https://github.com/crossminer/crossJadolint\n- projects.json\n```\n{\n    "Chaoss": {\n        "dockerdeps": [\n            "https://github.com/chaoss/grimoirelab"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[dockerdeps]\nraw_index = dockerdeps_raw\nenriched_index = dockerdeps_enrich\ncategory = code_dependencies_jadolint\nexec-path = <jadolint-local-path>/jadolint.jar\nin-paths = [Dockerfile, Dockerfile-full, Dockerfile-secured, Dockerfile-factory, Dockerfile-installed]\n```\n#### dockersmells [&uarr;](#supported-data-sources-)\nSmells extracted from Dockerfiles. Requires https://github.com/crossminer/crossJadolint\n- projects.json\n```\n{\n    "Chaoss": {\n        "dockersmells": [\n            "https://github.com/chaoss/grimoirelab"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[dockersmells]\nraw_index = dockersmells_raw\nenriched_index = dockersmells_enrich\ncategory = code_quality_jadolint\nexec-path = <jadolint-local-path>/jadolint.jar\nin-paths = [Dockerfile, Dockerfile-full, Dockerfile-secured, Dockerfile-factory, Dockerfile-installed]\n```\n#### functest [&uarr;](#supported-data-sources-)\nTests from functest\n- projects.json\n```\n{\n    "Chaoss": {\n        "functest": [\n            "http://testresults.opnfv.org/test/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[functest]\nraw_index = functest_raw\nenriched_index = functest_enriched\nno-archive = true (suggested)\n```\n#### gerrit [&uarr;](#supported-data-sources-)\nReviews from Gerrit\n\nYou have to add your public key in the gerrit server.\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "gerrit": [\n            "review.opendev.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[gerrit]\nraw_index = gerrit_raw\nenriched_index = gerrit_enriched\nuser = xxxx\nno-archive = true (suggested)\nblacklist-ids = [] (optional)\nmax-reviews = 500 (optional)\nstudies = [enrich_demography:gerrit, enrich_onion:gerrit, enrich_demography_contribution:gerrit] (optional)\n\n[enrich_demography:gerrit] (optional)\n\n[enrich_onion:gerrit] (optional)\nin_index = gerrit_enriched\nout_index = gerrit-onion_enriched\n\n[enrich_demography_contribution:gerrit] (optional)\ndate_field = grimoire_creation_date\nauthor_field = author_uuid\n```\n#### git [&uarr;](#supported-data-sources-)\nCommits from Git\n\n**Note:** If you want to analyze private git repositories, make sure you pass the credentials directly in the URL.\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "git": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https://<username>:<api-token>@github.com/chaoss/grimoirelab-sirmordred"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[git]\nraw_index = git_raw\nenriched_index = git_enriched\nlatest-items = true (suggested)\nstudies = [enrich_demography:git, enrich_git_branches:git, enrich_areas_of_code:git, enrich_onion:git, enrich_extra_data:git] (optional)\n\n[enrich_demography:git] (optional)\n\n[enrich_git_branches:git] (optional)\nrun_month_days = [1, 23] (optional)\n\n[enrich_areas_of_code:git] (optional)\nin_index = git_raw\nout_index = git-aoc_enriched\n\n[enrich_onion:git] (optional)\nin_index = git_enriched\nout_index = git-onion_enriched\n\n[enrich_extra_data:git]\njson_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt\n\n[enrich_forecast_activity]\nout_index = git_study_forecast\n```\n#### github [&uarr;](#supported-data-sources-)\nIssues and PRs from GitHub\n\n##### issue\n- projects.json\n```\n{\n    "Chaoss": {\n        "github:issue": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https:/github.com/chaoss/grimoirelab-sirmordred"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[github:issue]\nraw_index = github_raw\nenriched_index = github_enriched\napi-token = xxxx\ncategory = issue\nsleep-for-rate = true\nno-archive = true (suggested)\nstudies = [enrich_onion:github,\n           enrich_geolocation:user,\n           enrich_geolocation:assignee,\n           enrich_extra_data:github,\n           enrich_backlog_analysis,\n           enrich_demography:github] (optional)\n\n[enrich_onion:github] (optional)\nin_index_iss = github_issues_onion-src\nin_index_prs = github_prs_onion-src\nout_index_iss = github-issues-onion_enriched\nout_index_prs = github-prs-onion_enriched\n\n[enrich_geolocation:user] (optional)\nlocation_field = user_location\ngeolocation_field = user_geolocation\n\n[enrich_geolocation:assignee] (optional)\nlocation_field = assignee_location\ngeolocation_field = assignee_geolocation\n\n[enrich_extra_data:github]\njson_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt\n\n[enrich_backlog_analysis]\nout_index = github_enrich_backlog\ninterval_days = 7\nreduced_labels = [bug,enhancement]\nmap_label = [others, bugs, enhancements]\n\n[enrich_demography:github]\n```\n##### pull request\n- projects.json\n```\n{\n    "Chaoss": {\n        "github:pull": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https:/github.com/chaoss/grimoirelab-sirmordred"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[github:pull]\nraw_index = github-pull_raw\nenriched_index = github-pull_enriched\napi-token = xxxx\ncategory = pull_request\nsleep-for-rate = true\nno-archive = true (suggested)\nstudies = [enrich_geolocation:user,\n           enrich_geolocation:assignee,\n           enrich_extra_data:github,\n           enrich_demography:github] (optional)\n\n[enrich_geolocation:user]\nlocation_field = user_location\ngeolocation_field = user_geolocation\n\n[enrich_geolocation:assignee]\nlocation_field = assignee_location\ngeolocation_field = assignee_geolocation\n\n[enrich_extra_data:github]\njson_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt\n\n[enrich_demography:github]\n```\n##### repo\nThe number of forks, stars, and subscribers in the repository.\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "github:repo": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https:/github.com/chaoss/grimoirelab-sirmordred"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[github:repo]\nraw_index = github-repo_raw\nenriched_index = github-repo_enriched\napi-token = xxxx\ncategory = repository\nsleep-for-rate = true\nno-archive = true (suggested)\nstudies = [enrich_extra_data:github, enrich_demography:github]\n\n[enrich_extra_data:github]\njson_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt\n\n[enrich_demography:github]\n```\n#### githubql [&uarr;](#supported-data-sources-)\nEvents from GitHub\n\nThe corresponding dashboards can be automatically uploaded by setting `github-events`\nto `true` in the `panels` section within the `setup.cfg`\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "githubql": [\n            "https://github.com/chaoss/grimoirelab-toolkit"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[panels]\ngithub-events = true\n\n[githubql]\nraw_index = github_event_raw\nenriched_index = github_event_enriched\napi-token = xxxxx\nsleep-for-rate = true\nsleep-time = "300" (optional)\nno-archive = true (suggested)\nstudies = [enrich_duration_analysis:kanban, enrich_reference_analysis] (optional)\n\n[enrich_duration_analysis:kanban]\nstart_event_type = MovedColumnsInProjectEvent\nfltr_attr = board_name\ntarget_attr = board_column\nfltr_event_types = [MovedColumnsInProjectEvent, AddedToProjectEvent]\n\n[enrich_duration_analysis:label]\nstart_event_type = UnlabeledEvent\ntarget_attr = label\nfltr_attr = label\nfltr_event_types = [LabeledEvent]\n\n[enrich_reference_analysis] (optional)\n```\n\n#### github2 [&uarr;](#supported-data-sources-)\nComments from GitHub\n\nThe corresponding dashboards can be automatically uploaded by setting `github-comments`\nto `true` in the `panels` section within the `setup.cfg`\n\n##### issue\n- projects.json\n```\n{\n    "Chaoss": {\n        "github2:issue": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https:/github.com/chaoss/grimoirelab-sirmordred"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[github2:issue]\napi-token = xxxx\nraw_index = github2-issues_raw\nenriched_index = github2-issues_enriched\nsleep-for-rate = true\ncategory = issue\nno-archive = true (suggested)\nstudies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github2, enrich_feelings] (optional)\n\n[enrich_geolocation:user] (optional)\nlocation_field = user_location\ngeolocation_field = user_geolocation\n\n[enrich_geolocation:assignee] (optional)\nlocation_field = assignee_location\ngeolocation_field = assignee_geolocation\n\n[enrich_extra_data:github2]\njson_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt\n\n[enrich_feelings]\nattributes = [title, body]\nnlp_rest_url = http://localhost:2901\n```\n##### pull request\n- projects.json\n```\n{\n    "Chaoss": {\n        "github2:pull": [\n            "https:/github.com/chaoss/grimoirelab-perceval",\n            "https:/github.com/chaoss/grimoirelab-sirmordred"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[github2:pull]\napi-token = xxxx\nraw_index = github2-pull_raw\nenriched_index = github2-pull_enriched\nsleep-for-rate = true\ncategory = pull_request\nno-archive = true (suggested)\nstudies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:git, enrich_feelings] (optional)\n\n[enrich_geolocation:user] (optional)\nlocation_field = user_location\ngeolocation_field = user_geolocation\n\n[enrich_geolocation:assignee] (optional)\nlocation_field = assignee_location\ngeolocation_field = assignee_geolocation\n\n[enrich_extra_data:github2]\njson_url = https://gist.githubusercontent.com/zhquan/bb48654bed8a835ab2ba9a149230b11a/raw/5eef38de508e0a99fa9772db8aef114042e82e47/bitergia-example.txt\n\n[enrich_feelings]\nattributes = [title, body]\nnlp_rest_url = http://localhost:2901\n```\n\n#### gitlab [&uarr;](#supported-data-sources-)\nIssues and MRs from GitLab\n\nGitLab issues and merge requests need to be configured in two different sections.\nThe corresponding dashboards can be automatically uploaded by setting `gitlab-issue` and `gitlab-merge`\nto `true` in the `panels` section within the `setup.cfg`\n\nIf a given GitLab repository is under more than 1 level, all the slashes `/` starting from the second level have to be\nreplaced by `%2F`. For instance, for a repository with a structure similar to this one\n`https://gitlab.com/Molly/lab/first`.\n##### issue\n- projects.json\n```\n{\n    "Chaoss": {\n        "gitlab:issue": [\n            "https://gitlab.com/Molly/first",\n            "https://gitlab.com/Molly/lab%2Fsecond"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[panels]\ngitlab-issues = true\n\n[gitlab:issue]\ncategory = issue\nraw_index = gitlab-issues_raw\nenriched_index = gitlab-issues_enriched\napi-token = xxxx\nsleep-for-rate = true\nno-archive = true (suggested)\nstudies = [enrich_onion:gitlab-issue] (optional)\n\n[enrich_onion:gitlab-issue] (optional)\nin_index = gitlab-issues_enriched\nout_index = gitlab-issues-onion_enriched\ndata_source = gitlab-issues\n```\n##### merge request\n- projects.json\n```\n{\n    "Chaoss": {\n        "gitlab:merge": [\n            "https://gitlab.com/Molly/first",\n            "https://gitlab.com/Molly/lab%2Fsecond"\n        ],\n    }\n}\n```\n- setup.cfg\n```\n[panels]\ngitlab-merges = true\n\n[gitlab:merge]\ncategory = merge_request\nraw_index = gitlab-mrs_raw\nenriched_index = gitlab-mrs_enriched\napi-token = xxxx\nsleep-for-rate = true\nno-archive = true (suggested)\nstudies = [enrich_onion:gitlab-merge] (optional)\n\n[enrich_onion:gitlab-merge] (optional)\nin_index = gitlab-mrs_enriched\nout_index = gitlab-mrs-onion_enriched\ndata_source = gitlab-merges\n\n```\n#### gitter [&uarr;](#supported-data-sources-)\nMessages from gitter rooms\n\nYou have to join the rooms you want to mine.\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "gitter": [\n            "https://gitter.im/jenkinsci/jenkins",\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[gitter]\nraw_index = gitter_raw\nenriched_index = gitter_enriched_raw\napi-token = xxxxx\nsleep-for-rate = true\nsleep-time = "300" (optional)\nno-archive = true (suggested)\n```\n\n#### google_hits [&uarr;](#supported-data-sources-)\nNumber of hits for a set of keywords from Google\n- projects.json\n```\n{\n    "Chaoss": {\n        "google_hits": [\n            "bitergia grimoirelab"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[google_hits]\nraw_index = google_hits_raw\nenriched_index =google_hits_enriched\n```\n#### groupsio [&uarr;](#supported-data-sources-)\nMessages from Groupsio\n\nTo know the lists you are subscribed to: https://gist.github.com/valeriocos/ad33a0b9b2d13a8336230c8c59df3c55\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "groupsio": [\n            "group1",\n            "group2"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[groupsio]\nraw_index = groupsio_raw\nenriched_index = groupsio_enriched\nemail = yyyy\npassword = xxxx\n```\n#### hyperkitty [&uarr;](#supported-data-sources-)\nMessages from a HyperKitty\n- projects.json\n```\n{\n    "Chaoss": {\n        "hyperkitty": [\n            "https://lists.mailman3.org/archives/list/mailman-users@mailman3.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[hyperkitty]\nraw_index = hyperkitty_raw\nenriched_index = hyperkitty_enriched\n```\n#### jenkins [&uarr;](#supported-data-sources-)\nBuilds from a Jenkins\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "jenkins": [\n            "https://build.opnfv.org/ci"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[jenkins]\nraw_index = jenkins_raw\nenriched_index = jenkins_enriched\nno-archive = true (suggested)\n```\n#### jira [&uarr;](#supported-data-sources-)\nIssues data from JIRA issue trackers\n\n- projects.json\n```\n{\n    "Chaoss":{\n        "jira": [\n            "https://jira.opnfv.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[jira]\nraw_index = jira_raw\nenriched_index = jira_enriched\nno-archive = true (suggested)\nbackend-user = yyyy (optional)\nbackend-password = xxxx (optional)\n```\n#### kitsune [&uarr;](#supported-data-sources-)\nQuestions and answers from KitSune\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "kitsune": [\n            ""\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[kitsune]\nraw_index = kitsune_raw\nenriched_index = kitsune_enriched\n```\n\n#### mattermost [&uarr;](#supported-data-sources-)\nMessages from Mattermost channels\n- projects.json\n```\n{\n    "Chaoss": {\n        "mattermost": [\n            "https://chat.openshift.io 8j366ft5affy3p36987pcugaoa"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[mattermost]\nraw_index = mattermost_raw\nenriched_index = mattermost_enriched\napi-token = xxxx\nstudies = [enrich_demography:mattermost] (optional)\n\n[enrich_demography:mattermost] (optional)\n```\n#### mbox [&uarr;](#supported-data-sources-)\nMessages from MBox files\n\nFor mbox files, it is needed the name of the mailing list and the path where the mboxes can be found. In the example\nbelow, the name of the mailing list is set to "mirageos-devel".\n- projects.json\n```\n{\n    "Chaoss": {\n        "mbox": [\n            "mirageos-devel /home/bitergia/mbox/mirageos-devel/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[mbox]\nraw_index = mbox_raw\nenriched_index = mbox_enriched\n```\n#### mediawiki [&uarr;](#supported-data-sources-)\nPages and revisions from MediaWiki\n\n-projects.json\n```\n{\n    "Chaoss": {\n        "mediawiki": [\n            "https://www.mediawiki.org/w https://www.mediawiki.org/wiki"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[mediawiki]\nraw_index = mediawiki_raw\nenriched_index = mediawiki_enriched\nno-archive = true (suggested)\n```\n#### meetup [&uarr;](#supported-data-sources-)\nEvents from Meetup groups\n\nFor meetup groups it is only needed the identifier of the meetup group\nand an API token: https://chaoss.github.io/grimoirelab-tutorial/gelk/meetup.html#gathering-meetup-groups-data\n- projects.json\n```\n{\n    "Chaoss": {\n        "meetup": [\n        "Alicante-Bitergia-Users-Group",\n        "South-East-Bitergia-User-Group"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[meetup]\nraw_index = meetup_raw\nenriched_index = meetup_enriched\napi-token = xxxx\nsleep-for-rate = true\nsleep-time = "300" (optional)\nno-archive = true (suggested)\n\n```\n#### mozillaclub [&uarr;](#supported-data-sources-)\nEvents from Mozillaclub\n- projects.json\n```\n{\n    "Chaoss": {\n        "mozillaclub": [\n            "https://spreadsheets.google.com/feeds/cells/1QHl2bjBhMslyFzR5XXPzMLdzzx7oeSKTbgR5PM8qp64/ohaibtm/public/values?alt=json"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[mozillaclub]\nraw_index = mozillaclub_raw\nenriched_index = mozillaclub_enriched\n```\n#### nntp [&uarr;](#supported-data-sources-)\nArticles from NNTP newsgroups\n\nThe way to setup netnews is adding the server and the news channel to be monitored. In the example below,\nthe `news.myproject.org` is the server name.\n- projects.json\n```\n{\n    "Chaoss": {\n        "nntp": [\n            "news.myproject.org mozilla.dev.tech.crypto.checkins",\n            "news.myproject.org mozilla.dev.tech.electrolysis",\n            "news.myproject.org mozilla.dev.tech.gfx",\n            "news.myproject.org mozilla.dev.tech.java"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[nntp]\nraw_index = nntp_raw\nenriched_index =  nntp_enriched\n```\n#### pagure [&uarr;](#supported-data-sources-)\nIssues from Pagure repositories\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "pagure": [\n            "https://pagure.io/Test-group/Project-example-namespace"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[pagure]\nraw_index = pagure_raw\nenriched_index = pagure_enriched\napi-token = xxxx\nsleep-for-rate = true\nsleep-time = "300" (optional)\nno-archive = true (suggested)\n```\n#### phabricator [&uarr;](#supported-data-sources-)\nTasks from Phabricator\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "phabricator": [\n            "https://phabricator.wikimedia.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[phabricator]\nraw_index = phabricator_raw\nenriched_index = phabricator_enriched\napi-token = xxxx\nno-archive = true (suggested)\n```\n#### pipermail [&uarr;](#supported-data-sources-)\nMessages from Pipermail\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "pipermail": [\n            "https://lists.linuxfoundation.org/pipermail/grimoirelab-discussions/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[pipermail]\nraw_index = pipermail_raw\nenriched_index = pipermail_enriched\n```\n#### puppetforge [&uarr;](#supported-data-sources-)\nModules and their releases from Puppet\'s forge\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "puppetforge": [\n            ""\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[puppetforge]\nraw_index = puppetforge_raw\nenriched_index = puppetforge_enriched\n```\n#### redmine [&uarr;](#supported-data-sources-)\nIssues from Redmine\n- project.json\n```\n{\n    "Chaoss": {\n        "redmine": [\n            "http://tracker.ceph.com/"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[redmine]\nraw_index = redmine_raw\nenriched_index = redmine_enriched\napi-token = XXXXX\n```\n#### remo [&uarr;](#supported-data-sources-)\nEvents, people and activities from ReMo\n- project.json\n```\n{\n    "Chaoss": {\n        "remo": [\n            "https://reps.mozilla.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[remo]\nraw_index = remo_raw\nenriched_index = remo_enriched\n```\n#### rocketchat [&uarr;](#supported-data-sources-)\nMessages from Rocketchat channels\n- projects.json\n```\n{\n    "Chaoss": {\n        "rocketchat": [\n            "https://open.rocket.chat general"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[rocketchat]\nraw_index = rocketchat_raw\nenriched_index = rocketchat_enriched\napi-token = xxxx\nsleep-for-rate = true\nuser-id = xxxx\nno-archive = true (suggested)\n```\n#### rss [&uarr;](#supported-data-sources-)\nEntries from RSS feeds\n\n- project.json\n```\n{\n    "Chaoss": {\n        "remo": [\n            "https://reps.mozilla.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[rss]\nraw_index = rss_raw\nenriched_index = rss_enriched\n```\n#### slack [&uarr;](#supported-data-sources-)\nMessages from Slack channels\n\nThe information needed to monitor slack channels is the channel id.\n- projects.json\n```\n{\n    "Chaoss": {\n        "slack": [\n            "A195YQBLL",\n            "A495YQBM2"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[slack]\nraw_index = slack_raw\nenriched_index = slack_enriched\napi-token = xxxx\nno-archive = true (suggested)\n```\n#### stackexchange [&uarr;](#supported-data-sources-)\nQuestions, answers and comments from StackExchange\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "stackexchange": [\n            "http://stackoverflow.com/questions/tagged/chef",\n            "http://stackoverflow.com/questions/tagged/chefcookbook",\n            "http://stackoverflow.com/questions/tagged/ohai",\n            "http://stackoverflow.com/questions/tagged/test-kitchen",\n            "http://stackoverflow.com/questions/tagged/knife"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[stackexchange]\nraw_index = stackexchange_raw\nenriched_index = stackexchange_enriched\napi-token = xxxx\nno-archive = true (suggested)\n```\n#### supybot [&uarr;](#supported-data-sources-)\nMessages from Supybot log files\n\nFor supybot files, it is needed the name of the IRC channel and the path where the logs can be found. In the example\nbelow, the name of the channel is set to "irc://irc.freenode.net/atomic".\n- projects.json\n```\n{\n    "Chaoss": {\n        "supybot": [\n            "irc://irc.freenode.net/atomic /home/bitergia/irc/percevalbot/logs/ChannelLogger/freenode/#atomic"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[supybot]\nraw_index = supybot_raw\nenriched_index = supybot_enriched\n```\n\n#### telegram [&uarr;](#supported-data-sources-)\nMessages from Telegram\n\nYou need to have an API token: https://github.com/chaoss/grimoirelab-perceval#telegram\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "telegram": [\n            "Mozilla_analytics"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[telegram]\nraw_index = telegram_raw\nenriched_index = telegram_enriched\napi-token = XXXXX\n```\n#### twitter [&uarr;](#supported-data-sources-)\nMessages from Twitter\n\nYou need to provide a [search query](https://developer.twitter.com/en/docs/tweets/search/guides/build-standard-query) and an API token (which requires to create an [app](https://developer.twitter.com/en/docs/basics/apps/overview)). The script at https://gist.github.com/valeriocos/7d4d28f72f53fbce49f1512ba77ef5f6 helps obtaining a token.\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "twitter": [\n            "bitergia"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[twitter]\nraw_index = twitter_raw\nenriched_index = twitter_enriched\napi-token = XXXX\n```\n\n#### weblate [&uarr;](#supported-data-sources-)\nChanges from Weblate\n\nYou need to have an API token: The token can be obtained after registering to a weblate\ninstance (e.g., https://translations.documentfoundation.org/), via the page <instance>/accounts/profile/#api\n\n- projects.json\n```\n{\n    "Chaoss": {\n        "weblate": [\n            "https://translations.documentfoundation.org"\n        ]\n    }\n}\n```\n- setup.cfg\n```\n[weblate]\nraw_index = weblate_raw\nenriched_index = weblate_enriched\napi-token = XXXX\nno-archive = true (suggested)\nsleep-for-rate = true (suggested)\nstudies = [enrich_demography:weblate] (optional)\n\n[enrich_demography:weblate] (optional)\n```\n\n## Micro Mordred [&uarr;](#contents)\n\nMicro Mordred is a simplified version of Mordred which omits the use of its scheduler. Thus, Micro Mordred allows running single Mordred tasks (e.g., raw collection, enrichment) per execution.\n\nMicro Mordred is located in the [sirmordred/utils](https://github.com/chaoss/grimoirelab-sirmordred/tree/master/sirmordred/utils/micro.py) folder of this same repository. It can be executed via command line, its parameters are summarized below:\n```\n--debug: execute Micro Mordred in debug mode\n\n--raw: activate raw task\n\n--enrich: activate enrich task\n\n--identities: activate merge identities task\n\n--panels: activate panels task\n\n--cfg: path of the configuration file\n\n--backends: list of cfg sections where the active tasks will be executed\n\n--logs-dir: single parameter denoting the path of folder in which logs are to be stored\n```\n\nExamples of possible executions are shown below:\n```\ncd .../grimoirelab-sirmordred/sirmordred/utils/\nmicro.py --raw --enrich --cfg ./setup.cfg --backends git # execute the Raw and Enrich tasks for the Git cfg section\nmicro.py --panels # execute the Panels task to load the Sigils panels to Kibiter\nmicro.py --raw --enrich --debug --cfg ./setup.cfg --backends groupsio --logs-dir logs # execute the raw and enriched tasks for the groupsio cfg section with debug mode on and logs being saved in the folder logs in the same directory as micro.py\n```\n',
    'author': 'GrimoireLab Developers',
    'author_email': 'None',
    'maintainer': 'None',
    'maintainer_email': 'None',
    'url': 'https://chaoss.github.io/grimoirelab/',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'entry_points': entry_points,
    'python_requires': '>=3.7.1,<4.0.0',
}


setup(**setup_kwargs)
