Metadata-Version: 2.1
Name: sirmordred
Version: 0.2.17
Summary: Drive GrimoireLab tools to produce a dashboard
Home-page: https://github.com/chaoss/grimoirelab-sirmordred
Author: CHAOSS
Author-email: acs@bitergia.com
License: GPLv3
Keywords: development repositories analytics git github bugzilla jira jenkins
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Requires-Dist: PyMySQL
Requires-Dist: grimoire-elk (>=0.32)
Requires-Dist: kidash (>=0.4.13)
Requires-Dist: kingarthur (>=0.1.12)
Requires-Dist: manuscripts (>=0.2.16)
Requires-Dist: pyyaml
Requires-Dist: redis
Requires-Dist: sortinghat (>=0.7)

SirMordred `Build Status <https://travis-ci.org/chaoss/grimoirelab-sirmordred>`__\ `Coverage Status <https://coveralls.io/github/chaoss/grimoirelab-sirmordred?branch=master>`__
================================================================================================================================================================================

SirMordred is the tool used to coordinate the execution of the
GrimoireLab platform, via a configuration file. Below you can find
details about the different sections composing the configuration file.

General Sections
----------------

[es_collection]
~~~~~~~~~~~~~~~

-  **arthur** (bool: False): Use arthur for collecting items from
   perceval
-  **arthur_url** (str: None): URL for the arthur service
-  **password** (str: None): Password for connection to Elasticsearch
-  **redis_url** (str: None): URL for the redis service
-  **url** (str: http://172.17.0.1:9200): Elasticsearch URL
   (**Required**)
-  **user** (str: None): User for connection to Elasticsearch ###
   [es_enrichment]

-  **autorefresh** (bool: True): Execute the autorefresh of identities
-  **autorefresh_interval** (int: 2): Time interval (days) to
   autorefresh identities
-  **password** (str: None): Password for connection to Elasticsearch
-  **url** (str: http://172.17.0.1:9200): Elasticsearch URL
   (**Required**)
-  **user** (str: None): User for connection to Elasticsearch ###
   [general]

-  **bulk_size** (int: 1000): Number of items to write in Elasticsearch
   using bulk operations
-  **debug** (bool: True): Debug mode (logging mainly) (**Required**)
-  **log_backup_count** (int: 5): Number of rotate logs files to
   preserve
-  **log_handler** (str: file): use rotate for rotating the logs
   automatically
-  **log_max_bytes** (int: 104857600): Max number of bytes per log file
-  **logs_dir** (str: logs): Directory with the logs of sirmordred
   (**Required**)
-  **min_update_delay** (int: 60): Short delay between tasks (collect,
   enrich …)
-  **scroll_size** (int: 100): Number of items to read from
   Elasticsearch when scrolling
-  **short_name** (str: Short name): Short name of the project
   (**Required**)
-  **update** (bool: False): Execute the tasks in loop (**Required**)
-  **aliases_file** (str: ./aliases.json): JSON file to define aliases
   for raw and enriched indexes
-  **menu_file** (str: ./menu.yaml): YAML file to define the menus to be
   shown in Kibiter
-  **global_data_sources** (list: bugzilla, bugzillarest, confluence,
   discourse, gerrit, jenkins, jira): List of data sources collected
   globally, they are declared in the section ‘unknown’ of the
   projects.json
-  **retention_time** (int: None): the maximum number of minutes wrt the
   current date to retain the data ### [panels]

-  **community** (bool: True): Include community section in dashboard
-  **kibiter_default_index** (str: git): Default index pattern for
   Kibiter
-  **kibiter_time_from** (str: now-90d): Default time interval for
   Kibiter
-  **kibiter_url** (str): Kibiter URL (**Required**)
-  **kibiter_version** (str: None): Kibiter version
-  **kafka** (bool: False): Include KIP section in dashboard
-  **github-repos** (bool: False): Enable GitHub repo stats menu
-  **gitlab-issues** (bool: False): Enable GitLab issues menu
-  **gitlab-merges** (bool: False): Enable GitLab merge requests menu
-  **mattermost** (bool: False): Enable Mattermost menu
-  **strict** (bool: True): Enable strict panels loading ### [phases]

-  **collection** (bool: True): Activate collection of items
   (**Required**)
-  **enrichment** (bool: True): Activate enrichment of items
   (**Required**)
-  **identities** (bool: True): Do the identities tasks (**Required**)
-  **panels** (bool: True): Load panels, create alias and other tasks
   related (**Required**)
-  **report** (bool: False): Generate the PDF report for a project
   (alpha)
-  **track_items** (bool: False): Track specific items from a gerrit
   repository ### [projects]

-  **load_eclipse** (bool: False): Load the projects from Eclipse
-  **projects_file** (str: projects.json): Projects file path with
   repositories to be collected group by projects
-  **projects_url** (str: None): Projects file URL ### [report]

-  **config_file** (str: report.cfg): Config file for the report
   (**Required**)
-  **data_dir** (str: report_data): Directory in which to store the
   report data (**Required**)
-  **end_date** (str: 2100-01-01): End date for the report
   (**Required**)
-  **filters** (list: []): General filters to be applied to all queries
-  **interval** (str: quarter): Interval for the report (**Required**)
-  **offset** (str: None): Date offset to be applied to start and end
-  **start_date** (str: 1970-01-01): Start date for the report
   (**Required**) ### [sortinghat]

-  **affiliate** (bool: True): Affiliate identities to organizations
   (**Required**)
-  **autogender** (bool: False): Add gender to the profiles (executes
   autogender)
-  **autoprofile** (list: [‘customer’, ‘git’, ‘github’]): Order in which
   to get the identities information for filling the profile
   (**Required**)
-  **bots_names** (list: []): Name of the identities to be marked as
   bots
-  **database** (str: sortinghat_db): Name of the Sortinghat database
   (**Required**)
-  **host** (str: mariadb): Host with the Sortinghat database
   (**Required**)
-  **identities_api_token** (str: None): API token for remote operation
   with GitHub and Gitlab
-  **identities_export_url** (str: None): URL in which to export the
   identities in Sortinghat
-  **identities_file** (list: []): File path with the identities to be
   loaded in Sortinghat
-  **identities_format** (str: sortinghat): Format of the identities
   data to be loaded
-  **load_orgs** (bool: False):
-  **matching** (list: [‘email’]): Algorithm for matching identities in
   Sortinghat (**Required**)
-  **no_bots_names** (list: []): Name of the identities to be unmarked
   as bots
-  **orgs_file** (str: None): File path with the organizations to be
   loaded in Sortinghat
-  **password** (str: ): Password to access the Sortinghat database
   (**Required**)
-  **reset_on_load** (bool: False): Unmerge and remove affiliations for
   all identities on load
-  **sleep_for** (int: 3600): Delay between task identities executions
   (**Required**)
-  **strict_mapping** (bool: True): rigorous check of values in
   identities matching (i.e, well formed email addresses)
-  **unaffiliated_group** (str: Unknown): Name of the organization for
   unaffiliated identities (**Required**)
-  **user** (str: root): User to access the Sortinghat database
   (**Required**) ### [track_items]

-  **project** (str: TrackProject): Gerrit project to track
   (**Required**)
-  **raw_index_gerrit** (str: ): Name of the gerrit raw index
   (**Required**)
-  **raw_index_git** (str: ): Name of the git raw index (**Required**)
-  **upstream_raw_es_url** (str: ): URL with the file with the gerrit
   reviews to track (**Required**) ## Backend Sections

In this section, a template of a backend section is shown. Further
information about Perceval backends parameters are available at:

-  Params details:
   http://perceval.readthedocs.io/en/latest/perceval.backends.core.html
-  Examples:
   https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfg

[backend-name:tag] # :tag is optional
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  **collect** (bool: True): enable/disable collection phase
-  **raw_index** (str: None): Index name in which to store the raw items
   (**Required**)
-  **enriched_index** (str: None): Index name in which to store the
   enriched items (**Required**)
-  **studies** (list: []): List of studies to be executed
-  **backend-param-1**: ..
-  **backend-param-2**: ..
-  **backend-param-n**: ..

Enrichment params
^^^^^^^^^^^^^^^^^

Some backend sections allow to specify specific enrichment options,
listed below.

[jenkins]
'''''''''

-  **node_regex**: regular expression for extracting node name from
   ``builtOn`` field. This regular expression **must contain at least
   one group**. First group will be used to extract node name. More
   groups are allowed but not used to extract anything else.

Studies Sections
----------------

In this section, a template of a study section is shown. A complete list
of studies parameters is available at:

-  https://github.com/chaoss/grimoirelab-sirmordred/blob/master/tests/test_studies.cfg

[studies-name:tag] # :tag is optional
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  **study-param-1**: ..
-  **study-param-2**: ..
-  **study-param-n**: ..

Micro Mordred
-------------

Micro Mordred is a simplified version of Mordred which omits the use of
its scheduler. Thus, Micro Mordred allows to run single Mordred tasks
(e.g., raw collection, enrichment) per execution.

Micro Mordred is located in the
`/utils <https://github.com/chaoss/grimoirelab-sirmordred/tree/master/utils/micro.py>`__
folder of this same repository. It can be executed via command line, its
parameters are summarized below:

::

   --debug: execute Micro Mordred in debug mode

   --raw: activate raw task
   --arthur: use Arthur to collect the raw data

   --enrich: activate enrich task

   --identities: activate merge identities task

   --panels: activate panels task

   --cfg: path of the onfiguration file
   --backends: list of cfg sections where the active tasks will be executed

Examples of possible executions are shown below:

::

   cd .../grimoirelab-sirmordred/utils/
   micro.py --raw --enrich --cfg ./setup.cfg --backends git #  execute the Raw and Enrich tasks for the Git cfg section
   micro.py --panels # execute the Panels task to load the Sigils panels to Kibiter

Getting started
---------------

There are 3 options to get started with SirMordred:

Source code
~~~~~~~~~~~

You will need to install ElasticSearch (6.1.0), Kibana (6.1.4) and a
MySQL/MariaDB database, and the following components:

-  `SirModred <https://github.com/chaoss/grimoirelab-sirmordred>`__
-  `ELK <https://github.com/chaoss/grimoirelab-elk>`__
-  `King Arthur <https://github.com/chaoss/grimoirelab-kingarthur>`__
-  `Perceval <https://github.com/chaoss/grimoirelab-perceval>`__
-  `Perceval for
   Mozilla <https://github.com/chaoss/grimoirelab-perceval-mozilla>`__
-  `Perceval for
   OPNFV <https://github.com/chaoss/grimoirelab-perceval-opnfv>`__
-  `Perceval for
   Puppet <https://github.com/chaoss/grimoirelab-perceval-puppet>`__
-  `Perceval for
   FINOS <https://github.com/Bitergia/grimoirelab-perceval-finos>`__
-  `SortingHat <https://github.com/chaoss/grimoirelab-sortinghat>`__
-  `Sigils <https://github.com/chaoss/grimoirelab-sigils>`__
-  `Kidash <https://github.com/chaoss/grimoirelab-kidash>`__
-  `Toolkit <https://github.com/chaoss/grimoirelab-toolkit>`__
-  `Cereslib <https://github.com/chaoss/grimoirelab-cereslib>`__
-  `Manuscripts <https://github.com/chaoss/grimoirelab-manuscripts>`__

Source code and docker
~~~~~~~~~~~~~~~~~~~~~~

You will have to install the GrimoireLab components listed above, and
use the following docker-compose to have ElasticSearch, Kibana and
MariaDB. Note that you can omit the ``mariadb`` section in case you have
MySQL/MariaDB already installed in your system.

::

   elasticsearch:
     restart: on-failure:5
     image: bitergia/elasticsearch:6.1.0-secured
     command: elasticsearch -Enetwork.bind_host=0.0.0.0 -Ehttp.max_content_length=2000mb
     environment:
       - ES_JAVA_OPTS=-Xms2g -Xmx2g
     ports:
       - 9200:9200

   kibiter:
     restart: on-failure:5
     image: bitergia/kibiter:secured-v6.1.4-2
     environment:
       - PROJECT_NAME=Development
       - NODE_OPTIONS=--max-old-space-size=1000
       - ELASTICSEARCH_URL=https://elasticsearch:9200
     links:
       - elasticsearch
     ports:
       - 5601:5601

   mariadb:
     restart: on-failure:5
     image: mariadb:10.0
     expose:
       - "3306"
     ports:
       - "3306:3306"
     environment:
       - MYSQL_ROOT_PASSWORD=
       - MYSQL_ALLOW_EMPTY_PASSWORD=yes
       - MYSQL_DATABASE=test_sh
     command: --wait_timeout=2592000 --interactive_timeout=2592000 --max_connections=300
     log_driver: "json-file"
     log_opt:
         max-size: "100m"
         max-file: "3"

Only docker
~~~~~~~~~~~

Follow the instruction in the GrimoireLab tutorial to have `SirMordred
in a
container <https://chaoss.github.io/grimoirelab-tutorial/sirmordred/container.html>`__


