<div align="center">

# AutoOpInspect
</div>

AutoOpInspect is a Python package designed to streamline the inspection and profiling of operators within PyTorch models. It is a helpful tool for developers, researchers, and enthusiasts working in the machine learning field. Whether you are debugging, trying to improve the performance of your PyTorch models, or trying to collect operators info for a project, AutoOpInspect can help you!

<div align="center">
  <img src="./assets/AutoOpInspect_logo.png" width="500"/>
</div>


## Core Features

Here's what AutoOpInspect can offer:

- Operators Data Collection:
  - Automatically collects data of your model's operators such as input and output dimensions, number of parameters, class type, and more.
  - Gain comprehensive insights into the individual operators, aiding in meticulous analysis and optimization of your PyTorch models.
- Operators inference speed evaluation:
  - Automatically and individually measures the inference speed of each operator in your PyTorch models, providing you with detailed performance metrics.
  - This invaluable data assists developers in identifying bottlenecks and optimizing the performance of models for faster inference times on the device of your choice, ensuring peak operational efficiency.
- Automated Dummy Input/Output Data Generation:
  - Effortlessly generate dummy input or output data for any operator within your model.
  - Allows for rapid testing and verification of individual operators, speeding up the debugging process and ensuring the stability of your models.
- Model visualization
  - Offers a clear overview of all the operators in your model along with their detailed information, by printing the model structure in a readable format.
  - Facilitates a deeper understanding of the model's architecture, helping in fine-tuning and making informed adjustments during the development stage.
  - New in v1.0: barplot the inference speed or number of modules in your terminal, for an even better visualization experience!

## Installation

You can install AutoOpInspect using pip, with the following command:

``` bash
pip install auto_op_inspect
```

## Usage

Below are some examples demonstrating how to use the AutoOpInspect package:

### Basic Usage

Create an OpsInfoProvider instance using a PyTorch model and input data:

``` python
from AutoOpInspect import OpsInfoProvider
import torchvision
import torch

model = torchvision.models.vgg11()
input_data = [torch.randn(1, 3, 224, 224)] # make a list of inputs (supports multiple inputs)
ops_info_provider = OpsInfoProvider(model, input_data)
```

You can also specify a target operator to inspect, if you do not need to inspect the whole model:

``` python
target_module = model.features
ops_info_provider = OpsInfoProvider(model, input_data, target=target_module)
```

### Measuring inference speed

You can measure the inference speed of a single module, by specifying an operator.
If `operator` is None (default), then the benchmark will be done through all operators.

``` python
operator = model.features[6]
ops_info_provider.benchmark_speed(operator=operator, device='cpu', iterations=100)
print(ops_info_provider[operator].speed)
```

Before you begin measuring the inference speed, ensure that no other applications are running to limit the potential interference with the benchmark results. This precautionary measure helps in acquiring a more accurate measurement of the inference speed.
You might notice slight variations in the inference speed across different runs; this is normal and can be attributed to a variety of factors including system load, CPU/GPU thermal throttling, etc. If you need more reliable results, it is recommended to run the benchmark several times and consider the average value.
Note that multiple-gpu is not supported.

### Getting Dummy Data

Retrieve dummy input and output data for any operator:

``` python
operator = model.features[6]
dummy_input, dummy_output = ops_info_provider.get_dummy(operator, mode='both')
# available modes are input, output and both
```

### Visualize the model

``` python
print(ops_info_provider)
```

result:

```
Layer (type)                    Input Shape               Output Shape              Param #       Inference (ms)      Other
================================================================================================================================
target_module (VGG)            [[1, 3, 224, 224]]        [[1, 1000]]               132.86M       9.76976
├─ features (Sequential)       [[1, 3, 224, 224]]        [[1, 512, 7, 7]]          9.22M         5.22328
│ ├─ features.0 (Conv2d)       [[1, 3, 224, 224]]        [[1, 64, 224, 224]]       1,792         0.05798         (3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.1 (ReLU)         [[1, 64, 224, 224]]       [[1, 64, 224, 224]]       0             0.02664         (inplace=True)
│ ├─ features.2 (MaxPool2d)    [[1, 64, 224, 224]]       [[1, 64, 112, 112]]       0             0.03027         (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
│ ├─ features.3 (Sequential)   [[1, 64, 112, 112]]       [[1, 128, 112, 112]]      73,856        0.06859
│ │ ├─ features.3.0 (Conv2d)   [[1, 64, 112, 112]]       [[1, 128, 112, 112]]      73,856        0.04601         (64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ │ ├─ features.3.1 (ReLU)     [[1, 128, 112, 112]]      [[1, 128, 112, 112]]      0             0.02824         (inplace=True)
│ ├─ features.4 (ReLU)         [[1, 128, 112, 112]]      [[1, 128, 112, 112]]      0             0.02566         (inplace=True)
│ ├─ features.5 (MaxPool2d)    [[1, 128, 112, 112]]      [[1, 128, 56, 56]]        0             0.03111         (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
│ ├─ features.6 (Conv2d)       [[1, 128, 56, 56]]        [[1, 256, 56, 56]]        295,168       0.04055         (128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.7 (ReLU)         [[1, 256, 56, 56]]        [[1, 256, 56, 56]]        0             0.02655         (inplace=True)
│ ├─ features.8 (Conv2d)       [[1, 256, 56, 56]]        [[1, 256, 56, 56]]        590,080       0.03971         (256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.9 (ReLU)         [[1, 256, 56, 56]]        [[1, 256, 56, 56]]        0             0.03472         (inplace=True)
│ ├─ features.10 (MaxPool2d)   [[1, 256, 56, 56]]        [[1, 256, 28, 28]]        0             0.03484         (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
│ ├─ features.11 (Conv2d)      [[1, 256, 28, 28]]        [[1, 512, 28, 28]]        1.18M         0.04522         (256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.12 (ReLU)        [[1, 512, 28, 28]]        [[1, 512, 28, 28]]        0             0.03280         (inplace=True)
│ ├─ features.13 (Conv2d)      [[1, 512, 28, 28]]        [[1, 512, 28, 28]]        2.36M         0.04639         (512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.14 (ReLU)        [[1, 512, 28, 28]]        [[1, 512, 28, 28]]        0             0.03984         (inplace=True)
│ ├─ features.15 (MaxPool2d)   [[1, 512, 28, 28]]        [[1, 512, 14, 14]]        0             0.04070         (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
│ ├─ features.16 (Conv2d)      [[1, 512, 14, 14]]        [[1, 512, 14, 14]]        2.36M         0.05222         (512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.17 (ReLU)        [[1, 512, 14, 14]]        [[1, 512, 14, 14]]        0             0.04114         (inplace=True)
│ ├─ features.18 (Conv2d)      [[1, 512, 14, 14]]        [[1, 512, 14, 14]]        2.36M         0.06072         (512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
│ ├─ features.19 (ReLU)        [[1, 512, 14, 14]]        [[1, 512, 14, 14]]        0             0.04517         (inplace=True)
│ ├─ features.20 (MaxPool2d)   [[1, 512, 14, 14]]        [[1, 512, 7, 7]]          0             0.04877         (kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
├─ avgpool (AdaptiveAvgPool2d) [[1, 512, 7, 7]]          [[1, 512, 7, 7]]          0             0.04554         (output_size=(7, 7))
├─ classifier (Sequential)     [[1, 25088]]              [[1, 1000]]               123.64M       0.19959
│ ├─ classifier.0 (Linear)     [[1, 25088]]              [[1, 4096]]               102.76M       0.03174         (in_features=25088, out_features=4096, bias=True)
│ ├─ classifier.1 (ReLU)       [[1, 4096]]               [[1, 4096]]               0             0.02045         (inplace=True)
│ ├─ classifier.2 (Dropout)    [[1, 4096]]               [[1, 4096]]               0             0.00173         (p=0.5, inplace=False)
│ ├─ classifier.3 (Linear)     [[1, 4096]]               [[1, 4096]]               16.78M        0.03215         (in_features=4096, out_features=4096, bias=True)
│ ├─ classifier.4 (ReLU)       [[1, 4096]]               [[1, 4096]]               0             0.02042         (inplace=True)
│ ├─ classifier.5 (Dropout)    [[1, 4096]]               [[1, 4096]]               0             0.00164         (p=0.5, inplace=False)
│ ├─ classifier.6 (Linear)     [[1, 4096]]               [[1, 1000]]               4.10M         0.03169         (in_features=4096, out_features=1000, bias=True)
```

From v1.0, you can also visualize the model with a barplot, direcly in you terminal:

``` python
ops_info_provider.barplot_speed(mode = 'sum')
```

result:
``` 
┌─────────────────────────────────────────────────────────────── Operator Speed in ms (sum) ──────────────────────────────────────────────────────────────┐
│            Conv2d : █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████     0.389 │
│              ReLU : ██████████████████████████████████████████████████████████████████████████████████████████████████████████                    0.342 │
│         MaxPool2d : █████████████████████████████████████████████████████████                                                                     0.186 │
│            Linear : █████████████████████████████                                                                                                 0.096 │
│ AdaptiveAvgPool2d : ██████████████                                                                                                                0.046 │
│           Dropout : █                                                                                                                             0.003 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
``` 

``` python
ops_info_provider.barplot_speed(mode = 'mean')
```

result:
``` 
┌────────────────────────────────────────────────────────────── Operator Speed in ms (mean) ──────────────────────────────────────────────────────────────┐
│            Conv2d : █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████     0.049 │
│ AdaptiveAvgPool2d : █████████████████████████████████████████████████████████████████████████████████████████████████████████████████             0.046 │
│         MaxPool2d : ████████████████████████████████████████████████████████████████████████████████████████████                                  0.037 │
│            Linear : ███████████████████████████████████████████████████████████████████████████████                                               0.032 │
│              ReLU : █████████████████████████████████████████████████████████████████████████████                                                 0.031 │
│           Dropout : ████                                                                                                                          0.002 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
``` 

``` python
ops_info_provider.barplot_quantity()
```

result:
``` 
┌─────────────────────────────────────────────────────────────────── Operator Quantity ───────────────────────────────────────────────────────────────────┐
│              ReLU : █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████        11 │
│            Conv2d : ████████████████████████████████████████████████████████████████████████████████████████                                          8 │
│         MaxPool2d : ███████████████████████████████████████████████████████                                                                           5 │
│            Linear : █████████████████████████████████                                                                                                 3 │
│           Dropout : ██████████████████████                                                                                                            2 │
│ AdaptiveAvgPool2d : ███████████                                                                                                                       1 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
``` 

## Contributing

We welcome contributions to the AutoOpInspect project. Whether it's reporting issues, improving documentation, or contributing code, your input is valuable.