Metadata-Version: 2.4
Name: smer-visual
Version: 1.1.0
Summary: This package provides means of explainable classification of images with LLM-generated text descriptions, that are then analyzed using the Self Model Entities Related (SMER) explanation technique.
Author-email: Vojtěch Balek <vojta.balek@gmail.com>, Julija Jeršova <poty01@vse.cz>, Tomáš Kliegr <tomas.kliegr@vse.cz>, Michael Ahmet Ay <aymi00@vse.cz>
Project-URL: Homepage, https://github.com/vojtech-balek/SMERVisual
Project-URL: Bug Tracker, https://github.com/vojtech-balek/SMERVisual/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: opencv-python==4.11.0.86
Requires-Dist: matplotlib==3.9.2
Requires-Dist: numpy==1.26.4
Requires-Dist: pandas==2.2.3
Requires-Dist: Pillow==11.1.0
Requires-Dist: seaborn==0.13.2
Requires-Dist: scikit-learn==1.6.1
Requires-Dist: spacy==3.8.2
Requires-Dist: torch==2.4.1
Requires-Dist: tqdm==4.67.1
Requires-Dist: transformers==4.48.0
Requires-Dist: openai==1.52.2
Requires-Dist: lime==0.2.0.1
Dynamic: license-file

# **SMERVisual**

SMERVisual is a Python package designed for explainable machine learning using the **Self Model Entities Related (SMER) method**. It provides tools for explainable classification of images with **LLM-generated text descriptions**, which are then analyzed using the SMER explanation technique.

The package supports both **OpenAI API** models and **local language models**, offering flexibility in model selection.

---

## **Installation**
Install SMERVisual using pip:
```sh
pip install smer-visual
```

---

## **Features**

### **Image Description**
The `image_description` function generates text descriptions for images using either OpenAI models or local language models. Key features include:
- Support for OpenAI models like `gpt-4o-mini` and local models.
- Customizable prompts for generating concise and informative descriptions.

### **Description Embeddings**
The `get_description_embeddings` function computes embeddings for image descriptions using OpenAI or local models. 

### **Explainable Classification**
The `classify_with_logreg` function performs logistic regression-based classification on image datasets while computing **SMER** values for explainability. Key features include:
- Aggregation of embeddings for classification.
- Computation of feature importance for each word in the description.
- Support for **AOPC (Area Over the Perturbation Curve)** analysis.

### **Visualization**
- **Important Words**: The `plot_important_words` function visualizes the most important words across images.
- **AOPC Curves**: The `plot_aopc` function compares the explainability of SMER and LIME methods by plotting AOPC curves.
- **Bounding Boxes**: The `BoundingBoxGenerator` class overlays bounding boxes on images, highlighting critical words identified in classification.

---

## **Usage Example**

### **Image Description and Embeddings**
```python
from smer_visual.smer import image_description, get_description_embeddings

# Generate image descriptions
descriptions = image_description(
    model="gpt-4o-mini",
    data_folder="path/to/images",
    api_key="your_openai_api_key"
)

# Generate embeddings for descriptions
embeddings_df = get_description_embeddings(
    descriptions=descriptions,
    embedding_model="text-embedding-ada-002",
    api_key="your_openai_api_key"
)
```

### **Explainable Classification**
```python
from smer_visual.smer import classify_with_logreg, aggregate_embeddings
from sklearn.linear_model import LogisticRegression
import numpy as np

# Prepare data
embeddings_df["aggregated_embedding"] = embeddings_df["embedding"].apply(aggregate_embeddings)
X_train = np.stack(embeddings_df["aggregated_embedding"].values)
y_train = embeddings_df["label"]

# Train a logistic regression model
logreg_model = LogisticRegression()
logreg_model.fit(X_train, y_train)

# Perform classification and compute feature importance
aopc_df, updated_dataset = classify_with_logreg(embeddings_df, X_train, logreg_model)
```

### **Visualization**
```python
from smer_visual.smer import plot_important_words, plot_aopc

# Plot important words
plot_important_words(updated_dataset)

# Plot AOPC curves
plot_aopc(aopc_df, logreg_model, max_k=5)
```

### **Bounding Box Generation**
```python
from smer_visual.smer import save_bounding_box_images
results = save_bounding_box_images(
    input_path="data/",
    output_folder="output",
    df_top_words=top_words_df,
    model_id="IDEA-Research/grounding-dino-base",
    box_threshold = 0.5,
    text_threshold = 0.4
)

```

---

## **Why Use SMERVisual?**

- **Explainable AI** – Provides insight into model decision-making.
- **Model-Agnostic** – Compatible with OpenAI APIs and open-source models.
- **Zero-Shot Detection** – No additional training data required.
- **Easy Integration** – Simple API for seamless use with existing machine learning workflows.

---

## **Contributing**
Contributions are welcome. If you’d like to contribute, follow these steps:

1. Fork the repository.
2. Create a new branch (`git checkout -b feature-branch`).
3. Commit changes (`git commit -m "Add new feature"`).
4. Push to the branch (`git push origin feature-branch`).
5. Open a pull request.

---

## **License**
SMERVisual is released under the **MIT License**.

---
