# OpenMMLA Vision
[![PyPI version](https://img.shields.io/pypi/v/openmmla-vision.svg)](https://pypi.org/project/openmmla-vision/)

Video module of the mBox multimodal learning analytic system. For more details, please refer to [mbox System design](https://github.com/lizaibeim/mbox-uber/blob/main/docs/mbox_system.md).

## Uber Server Setup

Before setting up the video base, you need to set up a server hosting the InfluxDB, Redis, Mosquitto, and Nginx services.
Please refer to [mbox-uber](https://github.com/lizaibeim/mbox-uber/blob/main/README.md) module.

## Video Base & Server Setup

Downloading and Setting up the mbox-video analysis system is accomplished in two steps:  
(1) Clone the repository from GitHub to your local home directory.  
(2) Install the required dependencies.

1. Clone the repository from GitHub
   ```sh
   git clone https://github.com/ucph-ccs/mbox-video.git
   ```

2. Install the required dependencies with [conda environment](https://docs.anaconda.com/free/miniconda/index.html)
   - <details>
     <summary> Conda </summary>
     
       ```sh
       # For Raspberry Pi
       wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
       bash Miniforge3-$(uname)-$(uname -m).sh
       
       # For Mac and Linux
       wget "https://repo.anaconda.com/miniconda/Miniconda3-latest-$(uname)-$(uname -m).sh"
       bash Miniconda3-latest-$(uname)-$(uname -m).sh
       ```
     </details>
    
   - <details>
     <summary> Video Base </summary>
     
       ```sh
       # For video base
       conda create -c conda-forge -n video-base python=3.10.12 -y
       conda activate video-base
       cd mbox-video
       pip install openmmla-vision
       ```
     </details>

## Usage
After setting up the uber server and installation of dependencies, you can run the real-time video analysis by
following the instructions:

1. Stream video from your camera(s) to the server (e.g., Raspberry Pi, MacBook, etc.)  
   If you have multiple cameras and each wired to one machine, you can either run the video streaming on each machine or
   stream all cameras to a centralized RTMP server by following the guidelines in [raspi_rtmp_streaming.md](./docs/raspi_rtmp_streaming.md).

2. Calibrate camera's intrinsic parameters
    1. Go to ***./camera_calib/pattern***, print out the chessboard image and stick it on a flat surface.
    2. Capture the chessboard image with your camera and calibrate the camera by running `./calib_camera.sh` bash script.

3. Synchronize multi-cameras' coordinate systems

   If more than one camera is to be used at the same time, you need to calculate the transformation matrix between the
   main and the alternative camera. The transformation matrix is used to convert the alternative camera's coordinate 
   system to the main camera's coordinate system.

    1. **Centralized mode**  
       If your cameras are streaming to a centralized RTMP server or serially wired to a single machine, then run
       ```sh
       # :param -d the number of cameras needed to sync, default to 2. 
       # :param -s the number of camera sync manager, default to 1.
       # e.g.
       ./sync_camera.sh -d 2 -s 1
       ```
    2. **Distributed Mode**  
       If your cameras are not streamed to a centralized RTMP server and serially wired to a single machine, run camera 
       detector on each your camera hosting machine, and camera sync manager on your synchronizing machine.
       ```sh
       # on camera hosting machine, e.g. Raspberry Pi
       ./sync_camera.sh -d 1 -s 0
       # on synchronizing machine, e.g. MacBook
       ./sync_camera.sh -d 0 -s 1
       ```

4. Run real-time video analysis system
    1. **Centralized mode**  
       If your cameras are streaming to a centralized RTMP server or serially wired to a single machine, then run
       ```sh
       # :param -b the number of video base needed to run, default to 1. 
       # :param -s the number of video base synchronizer needed to run, default to 1. 
       # :param -v the number of visualizer need to run, default to 1. 
       # :param -g whether to display the graphic window, default to true. 
       # :param -r whether to record the video frames in image for clip baking, default to false.
       # :param -v whether to store the real time visualizations with visualizer, default to false.
       # e.g.
       ./run.sh
       ```
    2. **Distributed Mode**  
       If your cameras are not streamed to a centralized RTMP server and serially wired to a single machine, run video
       base on each camera hosting machine and run the synchronizer and visualizer on your synchronizing machine.
       ```sh
       # on camera hosting machine, e.g. Raspberry Pi
       ./run.sh -b 1 -s 0 -v 0 -g false
       # on synchronizing machine, e.g. MacBook
       ./run.sh -b 0 -s 1 -v 1
       ```
       
## [FAQ](https://github.com/lizaibeim/mbox-uber/blob/main/docs/FAQ.md)

## Citation
If you use this code in your research, please cite the following paper:
```
@inproceedings{inproceedings,
author = {Li, Zaibei and Jensen, Martin and Nolte, Alexander and Spikol, Daniel},
year = {2024},
month = {03},
pages = {785-791},
title = {Field report for Platform mBox: Designing an Open MMLA Platform},
doi = {10.1145/3636555.3636872}
}
```

## References
- [apriltags](https://github.com/pupil-labs/apriltags)


## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. 


