===============
Troubleshooting
===============

***********
Firewalling
***********

Due to the nature of firewall settings and customizations, bifrost does
**not** change any local firewalling on the node. Users must ensure that
their firewalling for the node running bifrost is such that the nodes that
are being booted can connect to the following ports::

    67/UDP for DHCP requests to be serviced
    69/UDP for TFTP file transfers (Initial iPXE binary)
    6385/TCP for the ironic API
    8080/TCP for HTTP File Downloads (iPXE, Ironic-Python-Agent)

If you encounter any additional issues, use of ``tcpdump`` is highly
recommended while attempting to deploy a single node in order to capture
and review the traffic exchange between the two nodes.

*****************
NodeLocked Errors
*****************

This is due to node status checking thread in ironic, which is a locking
action as it utilizes IPMI.  The best course of action is to retry the
operation.  If this is occurring with a high frequency, tuning might be
required.

Example error::

    NodeLocked: Node 00000000-0000-0000-0000-046ebb96ec21 is locked by
    host $HOSTNAME, please retry after the current operation is completed.

*********************
Building an IPA image
*********************

Troubleshooting issues involving IPA can be time consuming.  The IPA
developers **HIGHLY** recommend that users build their own custom IPA
images in order to inject things such as SSH keys, and turn on agent
debugging which must be done in a custom image as there is no mechanism
to enable debugging via the kernel command line at present.

Custom IPA images can be built a number of ways, the most generally useful
mechanism is with diskimage-builder.

DIB images:
  https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html

For documentation on diskimage-builder, See::
  https://docs.openstack.org/diskimage-builder/latest/.

It should be noted that the steps for diskimage-builder installation and
use to create an IPA image for Bifrost are the same as for ironic. See:
https://docs.openstack.org/ironic/latest/install/deploy-ramdisk.html

Once your build is completed, you will need to copy the images files into
the ``/var/lib/ironic/httpboot`` folder.

*********************************************
Unexpected/Unknown failure with the IPA Agent
*********************************************

Many failures due to the IPA agent can be addressed by building a custom
IPA Image.  See `Building an IPA image`_ for information on building
your own IPA image.

Obtaining IPA logs via the console
==================================

1) By default, bifrost sets the agent journal to be logged to the system
   console. Due to the variation in hardware, you may need to tune the
   parameters passed to the deployment ramdisk.  This can be done, as shown
   below in ironic.conf::

    kernel_append_params=nofb nomodeset vga=normal console=ttyS0 systemd.journald.forward_to_console=yes

   Parameters will vary by your hardware type and configuration,
   however the ``systemd.journald.forward_to_console=yes`` setting is
   a default, and will only work for systemd based IPA images.

   The example above, effectively disables all attempts by the kernel to set
   the video mode, defines the console as ttyS0 or the first serial port, and
   instructs systemd to direct logs to the console.

2) Once set, restart the ironic service, e.g.
   ``systemctl restart ironic`` and attempt to redeploy the node.
   You will want to view the system console occurring. If possible, you
   may wish to use ``ipmitool`` and write the output to a log file.

Gaining access via SSH to the node running IPA for custom images
================================================================

Custom built images will require a user to be burned into the image.
Typically a user would use the diskimage-builder ``devuser`` element
to achieve this. More detail on this can be located at
https://docs.openstack.org/diskimage-builder/latest/elements/devuser/README.html.

Example::

  export DIB_DEV_USER_USERNAME=customuser
  export DIB_DEV_USER_PWDLESS_SUDO=yes
  export DIB_DEV_USER_AUTHORIZED_KEYS=$HOME/.ssh/id_rsa.pub
  ironic-python-agent-builder -o /path/to/custom-ipa -e devuser debian

************************************
``ssh_public_key_path is not valid``
************************************

Bifrost requires that the user who executes bifrost have an SSH key in
their user home, or that the user defines a variable to tell bifrost where
to identify this file.  Once this variable is defined to a valid file, the
deployment playbook can be re-run.

Generating a new ssh key
========================

See the manual page for the ``ssh-keygen`` command.

Defining a specific public key file
===================================

A user can define a specific public key file by utilizing the
``ssh_public_key_path`` variable.  This can be set in the
``group_vars/inventory/all`` file, or on the ``ansible-playbook`` command
line utilizing the ``-e`` command line parameter.

Example::

  ansible-playbook -i inventory/bifrost_inventory.py deploy-dynamic.yaml -e ssh_public_key_path=~/path/to/public/key/id_rsa.pub

NOTE: The matching private key will need to be utilized to login to the
machine deployed.

******************************************************
Verifying Creation of Test Virtual Machines (VMs)
******************************************************

When using the Bifrost test environment, the ``./bifrost-cli testenv`` command
is expected to automatically create two virtual machines (VMs) for testing
purposes. These VMs simulate baremetal systems that Ironic can enroll and
deploy during provisioning. If the virtual machines are not created, subsequent
commands such as ``enroll`` or ``deploy`` will fail with errors indicating that
no nodes are available for registration.

Check for the presence of the test VMs using the following commands

.. code-block:: bash

      virsh list --all

If the setup is successful, two VMs (for example, ``testvm1`` and ``testvm2``)
should appear in the list. If no VMs are listed, the test environment creation
was incomplete. This is often caused by insufficient host resources or issues
with the virtualization service (libvirt/KVM).

Each test VM typically uses the following default resources:

* 1 vCPU
* ~3 GB of RAM
* ~11 GB of virtual storage

If the test VMs are not created, Ironic will not have real instances to
interact with, and only placeholder nodes will exist in the database.
As a result, any attempt to deploy will fail. Always verify that the test
VMs exist before proceeding with node enrollment and deployment.

***********************************************************
Node Enrollment and Provisioning in a Testing Environment
***********************************************************

Here are a few steps to troubleshoot node enrollment and provisioning in
testing environment.

Node Power and Provision State
================================

If the nodes enrollment fails, ensure the
power state of the nodes is 'On'. By default, the power state might be 'None'.
You can use the following commands to power on the nodes
and make them available:

.. code-block:: bash

      baremetal node power on <NODE>
      baremetal node manage <NODE>
      baremetal node provide <NODE>

Once the nodes are powered on and provision state is ``available``
re-run the enrollment process:

.. code-block:: bash

   ./bifrost-cli enroll baremetal-inventory.json

After successful enrollment, proceed with the deployment:

.. code-block:: bash

   ./bifrost-cli deploy baremetal-inventory.json

For documentation on the testing environment,
see the `Bifrost test environment guide <https://docs.openstack.org/bifrost/latest/contributor/testenv.html>`_.

For more on using the Baremetal CLI,
refer to the `Bifrost user HOWTO <https://docs.openstack.org/bifrost/latest/user/howto.html#baremetal-cli>`_.
