Git as a Data Source - Nautobot Documentation

Skip to content

# Git as a Data Source

The "Git™ as a Data Source" feature was developed to provide the ability to populate existing data, templates, scripts, and much more into Nautobot from Git repositories; leveraging the benefits that tools such as GitHub and GitLab already provide, including issue tracking, discussions, pipelines, and approvals. For just a few examples, it is possible to:

- have users approve the YAML data that is used for Nautobot `config contexts` along with running tests on that data.
- have users approve Jinja2 templates that are used for Nautobot `export templates`.

Like other Nautobot features, it is possible to create, read, update, and delete Git repository objects via the REST API as well as via the GUI. In addition, it is also possible to trigger a resync of an existing repository in either interface. For more detail on this and other technical aspects of using this feature, please see the documentation on Git Repositories.

## Supported Providers

This feature uses the concept of a `provides` field to identify the provided data types and therefore map a repository to a use case. A list of the supported options is provided below.

### Core Functionality

Name | Summary
--- | ---
Export Templates | Allows users to define custom templates that can be used when exporting objects.
Jobs | A way for users to execute custom logic on demand from within the Nautobot UI, to accomplish various Nautobot data creation, modification, and validation tasks.
Config Contexts | Config contexts can be used to provide additional data that you can't natively store in Nautobot.
Config Context Schemas | Schemas enforce data validation on config contexts.
GraphQL queries | GraphQL reduces the complexity of performing multiple API calls

### Examples of Apps Defining Additional Providers

Additional Git providers can be added by using Nautobot's flexible App system.

Name | Summary | Related App
--- | --- | ---
Backup Configs | Backup configuration data. | Golden Config
Intended Configs | Stores the intended configurations, this grabs Nautobot data and runs through Jinja Templates. | Golden Config
Jinja Templates | Repository that holds Jinja templates to be used to generate intended configs. | Golden Config

## Repository Details

This table defines repository parameters that are required to establish a repository connection.

Field | Explanation
--- | ---
Name | User friendly name for the repo.
Slug | Computer-friendly name for the repo. Auto-generated based on the `name` provided, but you can change it if you wish.
Remote URL | The URL pointing to the Git repo. Current git url usage is limited to `http` or `https`.
Branch | The branch in the Git repo to use. Defaults to `main`.
Secrets Group | (Optional) Grouping containing a HTTP token and/or HTTP username as needed to access the repository.
Provides | Resource type(s) provided by this Git repo.

Tip

The `slug` cannot be changed without removing and recreating the Git Repository record. Also, because a repository can potentially provide Jobs, Nautobot requires you to provide a `slug` that could be used as a Python module name and will not conflict with any existing Python module or package installed in Nautobot. For example, you can't create a repository with the slug `nautobot`.

## Token Requirements

The token in the `Secrets Group` field above should be generated on the provider side, usually on their website. These links will help with some of the common platforms:

- GitHub Personal Access Token
- GitLab Personal Access Token
- Bitbucket Personal Access Token

Removed in version 2.0.0

In Nautobot 1.x it was possible to configure the secrets (`username` and/or `token`) for a private Git Repository directly in Nautobot's database. Due to security concerns and maintainability challenges, this option has been removed. To access a private Git repository you now must use Secrets Groups.

## Using Git Data Sources

This section will focus on examples, and uses the `user-guide` branch on the `demo-git-datasources` repo: `https://github.com/nautobot/demo-git-datasource/tree/user-guide`.

### Export Templates

Export Templates allow a user to export Nautobot objects based on a custom template. Export templates can change over time depending on the needs of a user. Allowing export templates to reference a Git repo makes managing templates easier.

A template can be used to put objects into a specific format for ingestion into another system, tool, or report. It is possible that different templates are needed depending on specific users or teams. This can lead to sprawl of export templates. To keep accurate templates synced with Nautobot the Git Data Sources extensibility feature can be used.

#### Add a Repository

Navigate to the Data Sources Git integration. Extensibility -> Git Repositories.

Click [+] or [Add]

That loads a default page to add a repository.

Note

By default only config contexts, config context schemas, export templates, and jobs are available resource types. Others may be added when specific Nautobot Apps are installed.

#### Fill out Repository Details

Fill out the details for the Git repository. More information on the inputs can be found in the fields section.

As soon as you click on Create & Sync, Nautobot will clone and sync the repository and provide status of the job.

Note

If you are using a self-signed Git repository, the Server Administrator will need to ensure the `GIT_SSL_NO_VERIFY` environment variable is set to permit this.

The repository will now be displayed on the main Git Repository page.

Once the repository is synced each template will now be available in the Export Templates section. Extensibility -> Export Templates.

Note

If the templates don't populate, make sure the Git directory is named `export_templates` and the sub-directory and sub-sub-directory names correctly match the Nautobot `content type`.

Example below:

    ▶ tree export_templates
    export_templates
    └── dcim
        └── device
            ├── markdown_export.md
            ├── text_export.txt
            └── yaml_export.yml

    2 directories, 3 files

#### Modify a File and Sync Changes

Now that the export templates have been loaded into Nautobot they can be utilized as normal. For example navigate to Devices -> Devices and click on Export in the top right corner, the dropdown will now include the templates loaded from the Git repository.

The power of having export templates utilizing the Git integration comes with the native source control features that Git comes with. To illustrate a simple Git sync within Nautobot assume the following template needs to be updated.

Filename: `/export_templates/dcim/device/yaml_export.yml`

Current contents:

    ---
    {% for device in queryset %}
    {% if device.status %}
    - {{ device.name }}:
    {% endif %}
    {% endfor %}
    ...

The template needs to be modified to provide more information than just a list of hostnames. The location needs to be added.

The updated template is now:

    ---
    {% for device in queryset %}
    {% if device.status %}
    - {{ device.name }}:
      location: {{ device.location }}
    {% endif %}
    {% endfor %}
    ...

Once the contributor updates the Git repository via normal Git processes and it is reviewed and merged into the branch that was used, a sync process from Nautobot needs to be completed. This can be done from the default Git view, or within a specific detailed view of a Git repository.

From the default Git repositories view:

From the detailed view:

Tip

Once the repository has been synced it's easy to check the history for the templates. Navigate to Git Repositories and select the repository in question. Once you're in the detailed view you can look through the Synchronization Status or Change Log tabs.

Now that the Git repository is linked for export templates it can be controlled via the normal Git operations workflow, which allows users or groups of users to perform code reviews using Pull Requests etc.

### Jobs

Jobs are a way for users to execute custom logic on demand from within the Nautobot UI. Jobs can interact directly with Nautobot data to accomplish various data creation, modification, and validation tasks.

For technical details on jobs, please see the documentation on jobs.

Jobs allow a user to write scripts in Python. By integrating the scripts with Git, a user can utilize Git workflows to manage source control, versioning, and pipelines.

Setting up the repository can be done following the same steps from Export Templates. The only differences is the `provides` selection changes to `jobs`.

Jobs need to be defined in a `/jobs/` directory or `jobs.py` at the root of a Git repository. Any job classes defined in these files that have been registered during import will be discovered by Nautobot and made available to be run as a job. See the section on Job registration for more information.

An example tree for `/jobs/`.

    ▶ tree .
    __init__.py
    jobs
    ├── __init__.py
    └── get_device_connection.py

Note

As shown in the above example, the repository root must contain a file called `__init__.py` and the `/jobs/` directory must contain a file called `__init__.py`.

Once the repository is created in Nautobot.

Tip

The same repository and branch can be used for the different `provides` methods. Nautobot Git as a data source will look for specific root directory names.

Once the scripts have been pushed into the repository, a sync needs to be executed, after which navigating to Jobs via Jobs -> Jobs will show the new jobs loaded from the Git repository.

Jobs now shows the job from the Git repository.

At this point all changes, and history can be kept using Git. A simple `sync` operation can be done from Nautobot to pulldown any changes.

### Config Contexts

Detailed information on config contexts in Git Repositories.

Config contexts may be provided as JSON or YAML files located in the `/config_contexts/` folder, which must be in the root of the Git repository.

Config contexts can be used to provide additional details to different automation tooling. For example Ansible variables, or any other data that you can't natively store in Nautobot. It can also be used in the Nautobot Golden Configuration App to provide extra details to generate configuration templates.

A few simple examples of Configuration Context data might be:

- DNS Servers
- NTP Servers
- ACL Data
- Routing Information such as BGP ASNs etc.

Similar to the other data sources, the repository can be added by navigating to Extensibility -> Git repositories. Click on Add, and fill out the repository details.

Once the repository syncs the details can be found in the Synchronization Status tab. For example, the platform specifics were synced:

The repository structure is:

    ▶ tree config_contexts
    config_contexts
    ├── devices
    │   ├── location-a-bb-01.yml
    │   ├── location-a-rtr-01.yml
    │   ├── location-a-rtr-02.yml
    │   ├── location-a-spine-01.yml
    │   ├── location-a-spine-02.yml
    │   ├── location-b-bb-01.yml
    │   ├── location-b-leaf-01.yml
    │   ├── location-b-leaf-02.yml
    │   ├── location-b-rtr-01.yml
    │   ├── location-b-rtr-02.yml
    │   ├── location-b-spine-01.yml
    │   └── location-b-spine-02.yml
    ├── platform_eos.yml
    ├── platform_junos.yml
    ├── platform_nxos.yml
    └── role_spine.yml

    1 directory, 16 files

Configuration Context details:

- Follows an inheritance methodology similar to what Ansible implements. Global contexts can be overwritten by local contexts at both a group level, as well as at a device specific level.
- Nautobot UI provides a simple view to see merged config contexts. It can be visualized by navigating to a device and clicking on the config contexts tab.

Here's an example, with some of the details omitted for brevity.

There is a huge benefit to having `config contexts` managed by a Git workflow. This type of data can be modified often, especially platform specifics, or new device roles. Utilizing a standard Git workflow allows for all the proper reviews and approvals to be accomplished before accepting the changes into Nautobot for use.

### Config Context Schemas

Detailed information on config context schemas in Git Repositories.

Config context schemas are used to enforce data validation on config contexts. These schema are managed via the config context schema model and are optionally linked to config context instances, in addition to devices and virtual machines for the purpose of validating their local context data.

    ▶ tree config_contexts
    config_context_schemas
    ├── schema_1.yaml
    ├── schema_2.json

### GraphQL Queries

See also the GraphQL feature guide.

GraphQL queries can be used to reduce the complexity of performing multiple API calls while also correlating results by empowering the user to create their own query that provides the user exactly what they want and nothing that they don't, in a single API call.

    ▶ tree graphql_queries
    graphql_queries
    ├── query1.gql
    ├── query2.gql

## Additional Git Data Sources

As seen in Fill out Repository Details, the standard installation of Nautobot will come natively with export templates, jobs, and config contexts. Additional data sources can be incorporated using the Nautobot App system. For example, the Golden Configuration App implements four additional data sources.

- Config Contexts
- Backup Configs
- Intended Configs
- Jinja Templates

For more information for the Golden Configuration specific data sources, navigate to Nautobot Golden Config Docs.

## Common Issues and Troubleshooting

- Repository is linked, but data is not properly loaded into Nautobot.
  - Validate the path in the repository is correct for each data type:

Name | Path
--- | ---
Export Templates | `export_templates`
Jobs | `jobs`
Config Contexts | `config_contexts`
Config Context Schemas | `config_context_schemas`
GraphQL Queries | `graphql_queries`

- Synchronization Status Failures.
  - Validate branch is correct and exists in the remote repository.
  - Validate the remote url is correct and is the `http(s)` url. `ssh` urls are not currently supported.

- Authentication Issues.

  - Check repository permissions.
  - Ensure the password is the Personal Access Token (PAT) for the username supplied.
  - Ensure the PAT permissions are setup properly.
    - At a minimum the `repo` option should be checked or access.