Metadata-Version: 2.1
Name: nevax
Version: 0.1.1
Summary: NeVA: NVIDIA's Visual Question Answering Transformer
Home-page: https://github.com/kyegomez/neva
License: Attribution-NonCommercial 4.0 International
Author: Kye Gomez
Author-email: kye@apac.ai
Requires-Python: >=3.6,<4.0
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: torch
Requires-Dist: transformers
Project-URL: Repository, https://github.com/kyegomez/neva
Description-Content-Type: text/markdown

# NeVA: NVIDIA's Visual Question Answering Transformer

NeVA is a powerful and versatile Visual Question Answering model powered by NVIDIA's expertise. It builds upon the open-source LLaMA model, integrating it with an NVIDIA-trained GPT model to offer state-of-the-art performance.

## Installation

To integrate NeVA into your Python environment, you can install it via pip:

```bash
pip install neva
```


## Description

At a high level, NeVA utilizes a frozen Hugging Face CLIP model to encode images. These encoded images are projected to text embedding dimensions, concatenated with the embeddings of the given prompt, and subsequently passed through the language model. The training process comprises two main stages:

1. **Pretraining**: Only the projection layer is trained with the language model kept frozen. This stage uses image-caption pairs for training.
2. **Finetuning**: Both the language model and the projection layer are trained. This stage utilizes synthetic instruction data generated with GPT4.

## Model Specifications

- **Architecture Type**: Transformer
- **Network Architecture**: GPT + CLIP
- **Model versions**: 8B, 22B, 43B

## Input & Output

- **Input Format**: RGB Image + Text
- **Input Parameters**: temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate
- **Output Format**: Text

## Integration and Compatibility

- **Supported Hardware Platforms**: Hopper, Ampere/Turing
- **Supported Operating Systems**: Linux
- **Runtime**: N/A

## Training & Fine-tuning Data

**Pretraining Dataset**:
- **Link**: [CC-3M](#)
- **Description**: The dataset comprises CC3M images and captions, refined to 595,000 samples.
- **License**: [COCO](#), [CC-3M](#), [BLIP](#)

**Finetuning Dataset**:
- **Link**: Synthetic data produced by GPT4
- **Description**: The dataset, with 158,000 samples, was synthetically generated by GPT4. It encompasses a blend of short question answers, detailed image descriptions, and higher-level reasoning questions.
- **License**: [CC-BY-NC 4.0 License](https://creativecommons.org/licenses/by-nc/4.0/)

## Inference

- **Engine**: Triton
- **Test Hardware**: Other

## References

- [Visual Instruction Tuning paper](#)
- [Blog](#)
- [Codebase](#)
- [Demo](#)

## Licensing

This project is licensed under the [Attribution-NonCommercial 4.0 International](https://creativecommons.org/licenses/by-nc/4.0/) license.


