
Synchronisation
================

Heron, as an event based system, running over multiple processes and machines, does not rely on a single clock to generate
synchronised behaviour. Yet, as an experiment pipeline generator, it needs to allow users to fully time match the
different events and behaviours spawned by the different Nodes. To do this, Heron uses a number of
logging systems, which, when combined, offer the required time matching capacity.
The individual logging systems have been described in the :doc:`debugging` section for the logging system keeping
information on the communication between Nodes and the :doc:`saving_state` section for the system keeping
information on the internal activity of each Node.

In what follows we will describe how those two systems can be used in conjunction and together with some external
(hardware based) logging for the case of multiple machines, can generate a full record of the timing of every event that
a Heron pipeline generates.

Hardware
________
Within a single machine, Heron timestamps all logged events using the machine's clock. This of course breaks down when
the pipeline includes Nodes running in multiple machines. The only way to time match events that are
generated in different machines (at the required accuracy of an experimental setup) is to have an external data
acquisition (DAQ) device that will register the timing of different events that happen on separate machines. Such a DAQ
should save its data through a Heron Node running in the Heron GUI machine and have the saved data packets timestamped
through Heron's logging system.

Software
________
A Node can use Heron's Node Save State System (see :doc:`saving_state`) to record information available to the worker
process every time the worker function runs (for Transforms and Sinks) or goes around once in its infinite loop (for
Sources). That allows the worker process to register the time and index of the packet it generates together with
whatever other information it needs to save (the timestamping happens at the moment the savenodestate_update_substate_df
function is called in the worker function).

The worker process of a Transform or a Sink though has no information about the packet that triggered
the worker function, coming from a left connected Node. This information is saved in the Node's com process log file (
described in :doc:`debugging`).
There, Heron saves both the timestamp and the index of the incoming packet from the left Node and the corresponding
timestamp and the index of the packet  the current worker function generates. In this way one can connect which packet
from a left Node has triggered the production of which packet from a right Node.


Examples
_________
To make the above more concrete lets give here two examples of simple pipelines and how one would go about time matching
their outputs.

Within machine (saving a 120fps camera)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
One of Heron's Nodes is a `Spinnaker Camera. <https://github.com/Heron-Repositories/Spinnaker-Camera>`_ Some of the
FLIR cameras this Node can capture can reach large frames per second (FPS) speeds, too high for a standard Save CV2 Video
Node to be fast enough to save appropriately. That is why within the `Vision <https://github.com/Heron-Repositories/Vision>`_
repo there is a Save FFMPEG Video Node.

Let's assume we have a pipeline where a 120fps FLIR camera is captured by a Spinnaker Node and this feeds into a
Save FFMPEG Video Node. Now, there are three possible places where frames can be lost:

#. In the capturing from the camera to the Spinnaker Node.
#. In the transfer from the Spinnaker to the FFMPEG Node.
#. In the save into the video done in the FFMPEG Node.

Let's also assume that our application does not require 100% perfect capture (i.e. zero dropped
frames from camera to video file) but (as in most experimental cases) needs to allow the user to know exactly which
frames have been dropped where and to be able to assign the real time a frame has been captured by the camera sensor to
the corresponding video frame saved in the FFMPEG generated video file. They would also need to be able to assign every
frame in the final video to any other event in the Heron pipline (e.g. a press of a button).
In order to achieve this the user should:

#. Assign a save directory to the "Save the Node State to director:" entry of the secondary window of the Spinnaker Node.
#. Assign a log filename in the "Logfile or Verbosity Level" entry of the secondary window of the Save FFMPEG Video Node.

The Spinnaker Node saves the id and time of each captured frame as specified from the camera's hardware together with
the corresponding index and timestamp of the Node's loop iteration. This way, any frames generated by the camera but are
not captured by the Node become obvious.

The logfile generated by the FFMPEG Node's com process defines the index and timestamp this
process assigns to every packet (frame) that reaches it from the Spinnaker Node. Here, any frame that doesn't make it
across the Nodes will become obvious but more importantly this log file will generate a one to one correspondence between
the indices of the frames that pass through as they are generated by the Spinnaker Node and the indices generated by the
FFMPEG Node. Finally the FFMPEG Node uses an FFMPEG pipeline to write every receiving frame into
an encoded video file. This needs to be optimised (in the pipeline the FFMPEG Node defines) so that the number of frames
in the final video should be equal to the number of indices in the log file of the FFMPEG Node (that means the 3. above
should never happen).

Following the above procedure one can create a one to one connection between any video frame and the time the computer
captured that frame from the camera. They can also use the computer timestamp of any frame (the timestamp that marks
when the frame was captured by the Spinnaker Node) to assign every video frame to any other timestamped event in Heron's
pipeline as long as this happened in a Node running in the same Heron GUI machine.



Across Machines (time matching frames of different cameras)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now let's assume that in the above described Heron pipeline there is also an
`Arducam <https://github.com/Heron-Repositories/Arducam-Quadrascopic-Camera>`_ camera Node running on a Raspberry Pi
connected to the same LAN as the Heron GUI computer. The goal here is to be able to assign every frame from the FLIR
camera saved onto the FFMPEG generated video to every frame captured by the Arducam camera and saved to the Arducam video.
Here a user would need to again provide a save directory to the Arducam Node's secondary window. This way the Arducam Node
would save the timestamp and the index of the captured frame, which, because the saving of the video happens within the
same Arducam Node, would correspond to the equivalent saved frame in the video.

Yet that would not be enough to time match the frames of the two videos to each other, since the timestamps for the
Arducam frames are generated by a different computer to the ones assigned to the FFMPEG Video. The only way around this
would be for the user to have a DAQ that captures hardware triggers for both the FLIR camera and the Arducam one and save
those through a separate Node (e.g. the `NIDAX <https://github.com/Heron-Repositories/NI-DaqMX-Analog-Input>`_ Node).
At the same time the Arducam Node can capture a TTL pulse that comes into the Ras Pi's GPIO and save its index and time
onto the Save State dataframe together with the index of the captured frame. This saved TTL pulse is the same trigger
that triggers the Arducam camera and is also saved in the DAQ. This way the DAQ becomes the central clock onto which the
camera captured frames from both camera systems get registered. Through this common clock and the use of Heron's Save
State dataframes and com process log files, all events on both computers can be time matched to any other event.

