# ansible-content-parser

## Overview

`ansible-content-parser` used for analyze Ansible files, such
as playbooks, task files, etc. in a given directory.

It runs `ansible-lint` internally against a given
source directory and
updates Ansible files (the `--fix` option of `ansible-lint`)
and generates the `lint-result.json` file, which summarizes
files found in the directory and lint errors.

## Build

Execute the `tox` command. Installable images are created under
the `dist` directory.

## Installation

```commandline
pip install ansible-content-parser
```

## Execution

`ansible-content-parser` accepts two positional parameters (`source` and `output`)
with a few optional parameters.

```commandline
$ ansible-content-parser --help
usage: ansible-content-parser [-h] [--config-file CONFIG_FILE]
                              [--profile {min,basic,moderate,safety,shared,production}] [--skip-transform]
                              [--skip-ansible-lint] [--no-exclude] [-v] [--source-license SOURCE_LICENSE]
                              [--source-description SOURCE_DESCRIPTION] [--repo-name REPO_NAME] [--repo-url REPO_URL]
                              [--version]
                              source output

Parse Ansible files in the given repository by running ansible-lint and generate a training dataset for Ansible
Lightspeed.

positional arguments:
  source                source, which can be an zip/tar archive, a git URL or a local directory
  output                output directory

options:
  -h, --help            show this help message and exit
  --config-file CONFIG_FILE
                        Specify the configuration file to use for ansible-lint. By default it will look for '.ansible-
                        lint', '.config/ansible-lint.yml', or '.config/ansible-lint.yaml' in the source repository.
  --profile {min,basic,moderate,safety,shared,production}
                        Specify which rules profile to be used for ansible-lint
  --skip-transform      Skip the transform step of ansible-lint. If this option is not specified, ansible-lint is
                        executed with the --fix option and files are transformed according to the rules specified.
  --skip-ansible-lint   Skip the execution of ansible-lint.
  --no-exclude          Do not rerun ansible-lint with excluding files that caused syntax check errors. If one or more
                        syntax check errors were found, execution fails without generating the training dataset.
  -v, --verbose         Explain what is being done
  --source-license SOURCE_LICENSE
                        Specify the license that will be included in the training dataset.
  --source-description SOURCE_DESCRIPTION
                        Specify the description of the source that will be included in the training dataset.
  --repo-name REPO_NAME
                        Specify the repository name that will be included in the training dataset. If it is not
                        specified, it is generated from the source name.
  --repo-url REPO_URL   Specify the repository url that will be included in the training dataset. If it is not
                        specified, it is generated from the source name.
  --version             show program's version number and exit
```

### `source` positional argument

The first positional parameter is `source`, which specifies
the source repository to be used. Following three types of sources are supported:

1. File directory.
2. Archive file in the following table:

| File Format      | File Extension                                |
| ---------------- | --------------------------------------------- |
| ZIP              | .zip                                          |
| Uncompressed TAR | .tar                                          |
| Compressed TAR   | .tar.gz, .tgz, .tar.bz2, .tbz2, .tar.xz, .txz |

3. Git URL, e.g. `git@github.com:ansible/workshop-examples.git` or `https://github.com/ansible/workshop-examples.git`

### `output` positional argument

The second positional parameter is `output`, which specifies a writable
directory. If the directory already exists, it has to be
an empty directory. If it does not exist, it will be newly created with
the given name.

`ansible-content-parser` creates the`repository` subdirectory in the
`output` directory and copies the contents of the `source` repository
to it. The copied contents may be changed by during the execution
of the Content Parser.

## Outputs

Following directory structure is created in the directory specified with the `output`
positional argument.

```
output/
  |-- ftdata.json  # Training dataset
  |-- report.txt   # A human-readable report
  |
  |-- repository/
  |     |-- (files copied from the source repository)
  |
  |-- metadata/
        |-- lint-result.json     # Metadata generated by ansible-lint
        |-- sarif.json           # ansible-lint results in SARIF format
        |-- (other metadata files generated)
```

### ftdata.json

This is the training dataset file, which is the main output of `ansible-content-parser`.

It is in the JSONL format, each of whose line represents a JSON object

### report.txt

This is a human-readable report that provides the summary information of the run
of `ansible-content-parser`, which contains sections like:

1. File counts per type
2. List of Ansible files identified
3. Issues found by ansible-lint
4. List of Ansible modules found in tasks

Note: When the `--skip-ansible-lint` option is specified, the first three sections do
not appear in the report.

### metadata directory

This subdirectory contains a few files that contain metadata generated
in the Content Parser run.

#### lint-result.json

`lint-result.json` is created in the `metadata` subdirectory
as the result of the execution
of `ansible-content-parser`. The file contains a dictionary, which
has two key/value pairs:

1. `files` This is for the list of files that were found
   in the execution. The format of each file entry is explained below.

Each file entry is represented as a dictionary that contains following keys

| Key               | Description                                                   |
| ----------------- | ------------------------------------------------------------- |
| `base_kind`       | MIME type of the file, for example, `text/yaml`               |
| `dir`             | Directory where the file resides.                             |
| `exc`             | Exception found while processing this file. It can be null.   |
| `filename`        | File name                                                     |
| `kind`            | File type, for example, `playbook`, `tasks` or `role`         |
| `name`            | File name (Usually same as `filename`)                        |
| `parent`          | Name of the parent, like a role. It can be null               |
| `role`            | Ansible role. It can be null                                  |
| `stop_processing` | Identifies whether processing was stopped on this file or not |
| `updated`         | Identifies whether contents were updated by `ansible-lint`    |

Following shows an example of a file entry:

```json
{
  "base_kind": "text/yaml",
  "dir": "/mnt/input/roles/delete_compute_node/tasks",
  "exc": null,
  "filename": "roles/delete_compute_node/tasks/main.yaml",
  "kind": "tasks",
  "name": "roles/delete_compute_node/tasks/main.yaml",
  "parent": "roles/delete_compute_node",
  "role": "delete_compute_node",
  "stop_processing": false,
  "updated": false
}
```

2.  `excluded` This is for the list of file paths, which were excluded in the second `ansible-lint`
    execution because syntax check errors were found in those files on the first execution.
    The files included in the list will not appear in the entries associated with the `files` key.

- **Note:** If `ansible-content-parser` is executed with the `--no-exclude` option, the second execution
  does not occur even if syntax check errors were found on the first execution and
  the training dataset will not be created.

#### sarif.json

This is the output of `ansible-lint` with the `--sarif-file` option.
The `report.txt` contains a summary generated
from this file in the "Issues found by ansible-lint" section.
