===============
Troubleshooting
===============

Often fragment calculations fail. When this happens it's frequently an issue with just a single job that has an ill-conditioned initial guess or an SCF convergence issue. It can also be the result of a typo in your driver setup. We'll use a small built-to-fail example project to demonstrate the broad strokes of debugging a fragment calculation.

Our system will be a water hexamer

.. code-block:: text
    :caption: ``water6.xyz``

    18
    TIP4 water fragment
    O -1.126149 -1.748387 -0.423240
    H -0.234788 -1.493897 -0.661862
    H -1.062789 -2.681331 -0.218819
    O -0.254210  1.611495 -1.293845
    H -1.001520  1.163510 -1.690129
    H -0.153399  2.411746 -1.809248
    O 1.694541  -0.226287  1.705739
    H 0.785920   0.073487  1.677909
    H 2.047134   0.150917  2.511706
    O -0.864533  0.522472  1.218817
    H -0.694120  1.093542  0.469789
    H -1.131418 -0.310426  0.829702
    O -2.312727  0.054310 -2.138177
    H -2.002512 -0.686744 -1.617757
    H -3.266931  0.004778 -2.081349
    O 1.292355  -0.704768 -0.977342
    H 1.642300  -0.601085 -0.092386
    H 0.948122   0.159656 -1.202018

and we'll try running an MBE(2) calculation with BLYP in Q-Chem.

.. code-block:: yaml
    :caption: ``strategy.yaml``

    systems:
      -
        name: water6
        source: ./water6.xyz
    
    stages:
      -
        name: blyp
        type: QChem
        note: blyp through QChem
        options:
          template: |
            ! Name: {name}
            $molecule
            {charge} {mult}
            {geometry}
            $end
            $rem
              method          blyp
              basis           6-31g
              jobtype         sp
              max_scf_cycles  1 ! This is not enough cycles
            $end
      - name: cc
        type: CovalentComponentsFragmenter 
      -
        name: mbe
        type: FullFragmenter
    
    calculations:
      -
        name: mbe2
        systems: [water6]
        steps:
          - cc
          - mbe(2)
          - blyp

Run this calcuation with

.. code-block:: text

    $ fragment init strategy.yaml
    $ fragment calc run water6--mbe2
    ... It will fail. See the next section for output

Identifying Failed Calculations
===============================

You can use the ``fragment calc list --with-status`` command to identify failed calculations. They will have ``(FAILED)`` appended to the name.

.. code-block:: text

    $ fragment calc list --with-status
    water6--mbe2 (FAILED)

Determining the Failing System
==============================

Details about why a calculation failed can be found with ``fragment calc info "calc_name"``

.. code-block:: text

    $ fragment calc info water6--mbe2 
    Calculation water6--mbe2: 
    ID: 86ff7c4f-e948-4eba-a68d-d9f5144658fd
    Status: FAILED
    Created: 2024-07-15T13:29
    Steps:
        - _get_system(water6)
        - cc
        - mbe(2)
        - blyp
        Meta:
        completed_jobs: 5
        error: |
            ...-05
            ---------------------------------------
            Cycle       Energy         DIIS error
            ---------------------------------------
                1     -76.4047740048      2.38e-01  
            gen_scfman_exception: SCF failed to converge

            Q-Chem fatal error occurred in module libgscf/gen_scfman/gen_scfman_main.C, line 244:

            Error in gen_scfman


            Please submit a crash report at q-chem.com/reporter 
    
    

        failed_system: 54add0941c894724b56d9c63e09fa6ea0c3d68dd
        total_jobs: 21

This output tells us three important bits of information. The first is the error message. Different drivers have their own conventions for how to report errors; Q-Chem reports the last few lines of the output file in the ``error`` section of the record's metadata. In this case it's clearly an SCF convergence issue. 

The ``completed_jobs`` and ``total_jobs`` entries indicate approximately how many calculations have been run (not necessarily successfuly) and the total number of calculations for this fragmentation calculation.

The most important piece is the ``failed_system`` entry which gives the fingerprint for the system which produced the error. We'll use this for further debugging. 

Calculation Debug Mode
======================

Any system can be run with any driver with ``fragment calc debug --driver driver_name system_identifyer``. In debug mode the input and output files generated by the driver are stored in the ``debug`` folder inside the project directory.

.. note::

    This will `run` the calculation in the ``debug`` folder of your project. If your project folder is on an NFS mount or a mechanical hard drive you may experience performance issues. This may be problematic for large systems.

.. code-block:: text

    $ fragment calc debug --driver blyp 54add0941c894724b56d9c63e09fa6ea
    0c3d68dd
    DEBUGGING blyp:
    sys-54add094
        System Record 7a990ae9-d907-43f3-bd5f-14fca9eba5eb:
          ... The same error as above
          Meta:
            work_path: /path/to/your/project/debug/blyp/random_string


The important new information is the ``work_path`` metadata entry. Here you will find the input/output files generated by the diver. When fragment runs calculations in debug mode it does not save the results to the database, so you will have to re-run in production mode once the issue has been resolved. Now you can modify the input file to find a configuration that works. You can always rerun the debug command to generate a fresh copy of the input file if needed. 

Updating the Driver
===================

Drivers and stages can be edited using the ``fragment strategy edit-options driver_name`` CLI command. Only the options of the driver can be edited. Links between drivers cannot be changed (for now?).

.. warning ::

    Use caution when editing drivers. Changes to thresholds, basis sets, and methods may invalidate your data. It's up to you to know how your changes will effect a particular QM program.

.. code-block:: text

    $ fragment strategy edit-options blyp
    ... Interactivly prompts you to edit the stage options

In this case changing ``max_scf_cycles`` from 1 to 25 is all we need to do!

If you can't find a set of options which won't materially alter your results, you will have to create a new driver and rerun all your calculations.

Continue Where You Left Off
===========================

Now that your driver has been updated you need to mark all the FAILED jobs as PENDING. Run ``sqlite3 fragment.db "update record set status = 0 where status = 3;"``; a way to do this through the fragment CLI hasn't been added yet.

You are now good to run `fragment calc run ...` again. This will run any pending a previously failed jobs but will keep all the successful jobs run with the old driver options.