Graphsignal: Inference Monitoring and Profiling
===============================================

|License| |Version| |Status|

Graphsignal is an AI observability platform. It allows ML engineers and
MLOps teams to:

-  Monitor, troubleshoot and optimize AI applications by analyzing
   performance, data and errors.
-  Start measuring and profiling server applications and batch jobs
   automatically by adding a few lines of code.
-  Use Graphsignal in local, remote or cloud environment without
   installing any additional software or opening inbound ports.
-  Keep data private; no code or data is sent to Graphsignal cloud, only
   statistics and metadata.

|Dashboards|

Learn more at `graphsignal.com <https://graphsignal.com>`__.

Install
-------

Install Graphsignal agent by running:

::

   pip install graphsignal

Or clone and install the `GitHub
repository <https://github.com/graphsignal/graphsignal>`__:

::

   git clone https://github.com/graphsignal/graphsignal.git
   python setup.py install

Configure
---------

Configure Graphsignal agent by specifying your API key directly or via
``GRAPHSIGNAL_API_KEY`` environment variable.

.. code:: python

   import graphsignal

   graphsignal.configure(api_key='my-api-key') 

To get an API key, sign up for a free account at
`graphsignal.com <https://graphsignal.com>`__. The key can then be found
in your account’s `Settings / API
Keys <https://app.graphsignal.com/settings/api-keys>`__ page.

Integrate
---------

Use the following examples to integrate Graphsignal agent into your
machine learning application. See integration documentation and `API
reference <https://graphsignal.com/docs/reference/python-api/>`__ for
full reference.

Graphsignal agent is **optimized for production**. All executions
wrapped with ``span`` method will be measured, but only a few will be
traced and profiled to ensure low overhead.

Tracing
~~~~~~~

To measure and trace executions, wrap the code with
```span`` <https://graphsignal.com/docs/reference/python-api/#graphsignaltracerspan>`__
method.

.. code:: python

   with graphsignal.tracer().span(endpoint='predict'):
       # function call or code segment

Other integrations are available as well. See `integration
documentation <https://graphsignal.com/docs/>`__ for more information.

Profiling
~~~~~~~~~

Enable/disable various code profilers depending on the code and model
runtime by passing ``with_profiler`` argument to ``tracer()`` method. By
default ``with_profiler`` is ``True`` and Python profiler is enabled.
Set to ``False`` to disable profiling.

.. code:: python

   with graphsignal.tracer(with_profiler='pytorch').span(endpoint='predict'):
       # function call or code segment

The following values are currently supported: ``True`` (or ``python``),
``tensorflow``, ``pytorch``, ``jax``, ``onnxruntime``. See `integration
documentation <https://graphsignal.com/docs/>`__ for more information on
each profiler.

Exception tracking
~~~~~~~~~~~~~~~~~~

When ``with`` context manager is used with ``span`` method, exceptions
are **automatically recorded**. For other cases, use
```TraceSpan.set_exception`` <https://graphsignal.com/docs/reference/python-api/#graphsignaltracespanset_exception>`__
method.

Data monitoring
~~~~~~~~~~~~~~~

To track data metrics and record data profiles,
```TraceSpan.set_data`` <https://graphsignal.com/docs/reference/python-api/#graphsignaltracespanset_data>`__
method can be used.

.. code:: python

   with graphsignal.tracer().span(endpoint='predict') as span:
       span.set_data('input', input_data)

The following data types are currently supported: ``list``, ``dict``,
``set``, ``tuple``, ``str``, ``bytes``, ``numpy.ndarray``,
``tensorflow.Tensor``, ``torch.Tensor``.

**No raw data is recorded** by the agent, only statistics such as size,
shape or number of missing values.

Observe
-------

After everything is setup, `log in <https://app.graphsignal.com/>`__ to
Graphsignal to monitor and analyze execution performance and monitor for
issues.

Examples
--------

Model serving
~~~~~~~~~~~~~

.. code:: python

   import graphsignal

   graphsignal.configure(api_key='my-api-key')

   ...

   def predict(x):
       with graphsignal.tracer().span(endpoint='predict-prod'):
           return model(x)

Batch job
~~~~~~~~~

.. code:: python

   import graphsignal

   graphsignal.configure(api_key='my-api-key')

   ....

   for x in data:
       with graphsignal.tracer().span(endpoint='predict', tags=dict(job_id='job1')):
           preds = model(x)

More integration examples are available in
```examples`` <https://github.com/graphsignal/examples>`__ repo.

Overhead
--------

Although profiling may add some overhead to applications, Graphsignal
only profiles certain executions, automatically limiting the overhead.

Security and Privacy
--------------------

Graphsignal agent can only open outbound connections to
``agent-api.graphsignal.com`` and send data, no inbound connections or
commands are possible.

No code or data is sent to Graphsignal cloud, only statistics and
metadata.

Troubleshooting
---------------

To enable debug logging, add ``debug_mode=True`` to ``configure()``. If
the debug log doesn’t give you any hints on how to fix a problem, please
report it to our support team via your account.

In case of connection issues, please make sure outgoing connections to
``https://agent-api.graphsignal.com`` are allowed.

For GPU profiling, if ``libcupti`` agent is failing to load, make sure
the `NVIDIA® CUDA® Profiling Tools
Interface <https://developer.nvidia.com/cupti>`__ (CUPTI) is installed
by running:

.. code:: console

   /sbin/ldconfig -p | grep libcupti

.. |License| image:: http://img.shields.io/github/license/graphsignal/graphsignal
   :target: https://github.com/graphsignal/graphsignal/blob/main/LICENSE
.. |Version| image:: https://img.shields.io/github/v/tag/graphsignal/graphsignal?label=version
   :target: https://github.com/graphsignal/graphsignal
.. |Status| image:: https://img.shields.io/uptimerobot/status/m787882560-d6b932eb0068e8e4ade7f40c?label=SaaS%20status
   :target: https://stats.uptimerobot.com/gMBNpCqqqJ
.. |Dashboards| image:: https://graphsignal.com/external/screencast-dashboards.gif
   :target: https://graphsignal.com/
