Graphsignal: Inference Profiling And Monitoring
===============================================

|License| |Version| |Status|

Graphsignal is a machine learning inference profiling and monitoring
platform. It helps data scientists and ML engineers make model inference
faster and more efficient. It is built for real-world use cases and
allows ML practitioners to:

-  Optimize and monitor inference by measuring latency and throughput,
   analyzing bottlenecks and resource utilization.
-  Start profiling and monitoring jobs and server applications
   automatically by adding a few lines of code.
-  Use Graphsignal in local, remote or cloud environment without
   installing any additional software or opening inbound ports.
-  Keep data private; no code or data is sent to Graphsignal cloud, only
   statistics and metadata.

|Dashboards|

Learn more at `graphsignal.com <https://graphsignal.com>`__.

Documentation
-------------

See full documentation at
`graphsignal.com/docs <https://graphsignal.com/docs/>`__.

Getting Started
---------------

1. Installation
~~~~~~~~~~~~~~~

Install Graphsignal agent by running:

::

   pip install graphsignal

Or clone and install the `GitHub
repository <https://github.com/graphsignal/graphsignal>`__:

::

   git clone https://github.com/graphsignal/graphsignal.git
   python setup.py install

2. Configuration
~~~~~~~~~~~~~~~~

Configure Graphsignal agent by specifying your API key directly or via
environment variable.

.. code:: python

   import graphsignal

   graphsignal.configure(api_key='my-api-key')

To get an API key, sign up for a free account at
`graphsignal.com <https://graphsignal.com>`__. The key can then be found
in your account’s `Settings / API
Keys <https://app.graphsignal.com/settings/api-keys>`__ page.

Provide a ``workload_name`` to track runs, deployments or applications
separately.

.. code:: python

   graphsignal.configure(api_key='my-api-key', workload_name='model-serving-prod')

Integration
~~~~~~~~~~~

Use the following examples to integrate Graphsignal agent into your
machine learning application. See integration documentation and `API
reference <https://graphsignal.com/docs/reference/python-api/>`__ for
full reference.

Graphsignal agent is **optimized for production**. All inferences
wrapped with ``inference_span`` will be measured, but only a few will be
profiled to ensure low overhead.

`Python <https://graphsignal.com/docs/integrations/python/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from graphsignal.tracers.python import inference_span

   with inference_span(model_name='my-model'):
       # function call or code segment

`TensorFlow <https://graphsignal.com/docs/integrations/tensorflow/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from graphsignal.tracers.tensorflow import inference_span

   with inference_span(model_name='my-model'):
       # function call or code segment

`Keras <https://graphsignal.com/docs/integrations/keras/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from graphsignal.tracers.keras import GraphsignalCallback

   model.predict(..., callbacks=[GraphsignalCallback(model_name='my-model')])
   # or model.evaluate(..., callbacks=[GraphsignalCallback(model_name='my-model')])

`PyTorch <https://graphsignal.com/docs/integrations/pytorch/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from graphsignal.tracers.pytorch import inference_span

   with inference_span(model_name='my-model'):
       # function call or code segment

`PyTorch Lightning <https://graphsignal.com/docs/integrations/pytorch-lightning/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from graphsignal.tracers.pytorch_lightning import GraphsignalCallback

   trainer = Trainer(..., callbacks=[GraphsignalCallback(model_name='my-model')])
   trainer.predict() # or trainer.validate() or trainer.test()

`Hugging Face <https://graphsignal.com/docs/integrations/hugging-face/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from transformers import pipeline
   from graphsignal.tracers.pytorch import inference_span
   # or from graphsignal.tracers.tensorflow import inference_span

   pipe = pipeline(task="text-generation")

   with inference_span(model_name='my-model'):
       output = pipe('some text')

`JAX <https://graphsignal.com/docs/integrations/jax/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   from graphsignal.tracers.jax import inference_span

   with inference_span(model_name='my-model'):
       # function call or code segment

`ONNX Runtime <https://graphsignal.com/docs/integrations/onnx-runtime/>`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

   import onnxruntime
   from graphsignal.tracers.onnxruntime import initialize_profiler, inference_span

   sess_options = onnxruntime.SessionOptions()
   initialize_profiler(sess_options)

   session = onnxruntime.InferenceSession('my-model-path', sess_options)
   with inference_span(model_name='my-model', onnx_session=session):
       session.run(...)

Measuring Rates
^^^^^^^^^^^^^^^

By using any ``inference_span`` method, multiple metrics are
automatically measured and periodically reported, including inference
performance, CPU, GPU and memory.

To measure additional rates, ``InferenceSpan.set_count(name, value)``
method can be used. For example, by providing the number of processed
items on every inference, item rate per second will be automatically
calculated.

.. code:: python

   with inference_span(model_name='text-classification') as span:
       span.set_count('sentences', 5)
       span.set_count('words', 250)

Reporting Exceptions
^^^^^^^^^^^^^^^^^^^^

When ``with`` context manager is used with ``inference_span`` methods,
exceptions are automatically reported. For other cases, use
``InferenceSpan.add_exception(exc_info)`` method.

.. code:: python

   span = inference_span(model_name='my-model'):
   try:
       preds = model(inputs)
   except:
       span.add_exception(exc_info=True)
   span.stop()

3. Monitoring
~~~~~~~~~~~~~

After everything is setup, `log in <https://app.graphsignal.com/>`__ to
Graphsignal to monitor and analyze inference performance.

Examples
--------

Model serving
~~~~~~~~~~~~~

.. code:: python

   import graphsignal
   from graphsignal.tracers.pytorch import inference_span

   graphsignal.configure(
       api_key='my-api-key', workload_name='my-model-serving')

   ...

   def predict(x):
       with inference_span(model_name='my-model'):
           return model(x)

Batch job
~~~~~~~~~

.. code:: python

   import graphsignal
   from graphsignal.tracers.pytorch import inference_span

   graphsignal.configure(
       api_key='my-api-key', workload_name='job-{0}'.format(datetime.date.today()))

   ....

   for x in data:
       with inference_span(model_name='my-model'):
           preds = model(x)

More integration examples are available in
```examples`` <https://github.com/graphsignal/examples>`__ repo.

Overhead
--------

Although profiling may add some overhead to applications, Graphsignal
only profiles certain inferences, automatically limiting the overhead.

Security and Privacy
--------------------

Graphsignal Profiler can only open outbound connections to
``agent-api.graphsignal.com`` and send data, no inbound connections or
commands are possible.

No code or data is sent to Graphsignal cloud, only statistics and
metadata.

Troubleshooting
---------------

To enable debug logging, add ``debug_mode=True`` to ``configure()``. If
the debug log doesn’t give you any hints on how to fix a problem, please
report it to our support team via your account.

In case of connection issues, please make sure outgoing connections to
``https://agent-api.graphsignal.com`` are allowed.

For GPU profiling, if ``libcupti`` agent is failing to load, make sure
the `NVIDIA® CUDA® Profiling Tools
Interface <https://developer.nvidia.com/cupti>`__ (CUPTI) is installed
by running:

.. code:: console

   /sbin/ldconfig -p | grep libcupti

.. |License| image:: http://img.shields.io/github/license/graphsignal/graphsignal
   :target: https://github.com/graphsignal/graphsignal/blob/main/LICENSE
.. |Version| image:: https://img.shields.io/github/v/tag/graphsignal/graphsignal?label=version
   :target: https://github.com/graphsignal/graphsignal
.. |Status| image:: https://img.shields.io/uptimerobot/status/m787882560-d6b932eb0068e8e4ade7f40c?label=SaaS%20status
   :target: https://stats.uptimerobot.com/gMBNpCqqqJ
.. |Dashboards| image:: https://graphsignal.com/external/screencast-dashboards.gif
   :target: https://graphsignal.com/
