[![Multi-Modality](agorabanner.png)](https://discord.gg/qUtxnK2NMf)


# ROBOTCAT
![ROBOCAT MODELS](robocat.png)

RoboCAT is a self-improving foundation agent for robotic manipulation developed by DeepMind Robotics. The model architecture of RoboCAT is similar to the RT-1 model. It follows a tokenization approach where robotics images, proprioception, and future actions are tokenized. 


# Usage
There are 2 methods one is by pip `pip install robotcat` and the other is `git clone`


## Method1
* first `python3 -m pip install robocat`

* Then:

```python
import torch
from robocat import PALME, Robocat

model = Robocat(
    palme=PALME(),
    num_actions=11,
    action_bins=256,
    depth=6,
    heads=8,
    dim_head=64,
    token_learner_ff_mult=2,
    token_learner_num_layers=2,
    token_learner_num_output_tokens=8,
    cond_drop_prob=0.2,
    use_attn_conditioner=False,
    conditioner_kwargs=dict()
)

video = torch.rand((1, 3, 224, 224))
texts = ["this is a text"]
output = model(video, texts)
print(output.shape)

torch.save(model.state_dict(), 'rt3_model.pth')

```

* Or train:

```python
from roboCAT import Train

Train()

```


# Method 2

* First `git clone` the repository: `git clone https://github.com/kyegomez/RoboCAT.git`

* Then cd `cd RoboCAT` 

* Then, pip install: `python3 -m pip install -r requirements.txt`

* Then test with 
```python 
import torch
from robocat import PALME, Robocat

model = Robocat(
    palme=PALME(),
    num_actions=11,
    action_bins=256,
    depth=6,
    heads=8,
    dim_head=64,
    token_learner_ff_mult=2,
    token_learner_num_layers=2,
    token_learner_num_output_tokens=8,
    cond_drop_prob=0.2,
    use_attn_conditioner=False,
    conditioner_kwargs=dict()
)

video = torch.rand((1, 3, 224, 224))
texts = ["this is a text"]
output = model(video, texts)
print(output.shape)

torch.save(model.state_dict(), 'rt3_model.pth')

```

* Or run training on C4 `accelerate config`

* Then, `accelerate launch roboCAT/train.py`

## Architecture
The architecture consists of the following key components:

1. Tokenizer: RoboCAT learns a tokenizer for robotics images. It tokenizes proprioception and future actions in a straightforward manner, enabling the prediction of future action tokens.

2. Transformer Model: The tokenized inputs are fed into a Transformer model. The Transformer model predicts future action sequences based on the input tokens. This allows the model to perform a wide range of robotic tasks using a unified interface.

3. Action Spaces: RoboCAT predicts Cartesian 4 or 6 degrees of freedom (DoF) cartesian velocities for the arm and 1 DoF (parallel jaw gripper) or 8 DoF (3-finger) for the hand. This flexible approach enables the model to handle action spaces of different sizes and variable proprioception sizes.

The architecture of RoboCAT allows for the integration of multiple robot embodiments with a unified interface. By predicting the appropriate number of tokens based on the robot's morphology, the model can effectively scale without the need for separate prediction heads for each embodiment.


## Misc Components

### Generalization and Transfer Learning
The RoboCAT paper focuses on studying generalization and transfer learning. It explores how training on one domain benefits testing on another and investigates the effectiveness of transfer learning from simulation to the real world. The authors provide empirical data on cross-task transfer, architecture scaling, and tokenization strategies for perception.

### Evaluation and Automated Testing
RoboCAT emphasizes the importance of rigorous evaluation and presents methodologies for automated evaluation of multi-task policies in real-world settings. The paper provides details on evaluation protocols, data collection, and comparative analysis of different models and approaches.

### Real-World Robotic Tasks
The paper highlights the challenges of real-world robotics tasks and the significance of cross-robot transfer. The authors showcase consistent results across multiple robots and action spaces, demonstrating the value of collecting real-world data for training and evaluation. The effort put into data set detailing and evaluation protocols is commendable.

### Future Directions and Reproducibility
The authors acknowledge the ongoing challenge of reproducibility in robotics research. They emphasize the need for independent replication in different labs and variations in manipulation tasks and hardware. The paper raises questions about the impact of experimental choices and engineering decisions on research outcomes and

 calls for advancements in evaluation methodologies.

## Conclusion

The RoboCAT paper presents a self-improving foundation agent for robotic manipulation that addresses the challenges of generalization and transfer learning in the field of robotics. It offers insights into the model architecture, requirements, and experimental findings. The extensive empirical data, evaluation protocols, and comparisons provide valuable contributions to the research community.

# Roadmap

* Functional prototype

* Integrate VQGAN to generate an image when it has not encountered an known environment

` environment observation -> environment familarity rating [0.0-1.0] -> generate data if lower then [0.5] -> finetune -> action`

* Train on massive datasets

* Finetune as specified on paper

* Release as paid API

* Integrate more modalities like hearing, 3d mapping, nerfs, videos, lidar, locomotion, and the whole lot!


## Citations

```bibtex
@article{Bousmalis2023RoboCat,
    title   = {RoboCat: A Self-Improving Foundation Agent for Robotic Manipulation},
    author  = {Konstantinos Bousmalis*, Giulia Vezzani*, Dushyant Rao*, Coline Devin*, Alex X. Lee*, Maria Bauza*, Todor Davchev*, Yuxiang Zhou*, Agrim Gupta*,1, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Żołna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell†, Francesco Nori† and Nicolas Heess},
    journal = {ArXiv},
    year    = {2023}
}
```



