
<div align="center">

<h1>tfs-mt<br>
Transformer from scratch for Machine Translation</h1>

[![Release](https://img.shields.io/github/v/release/Giovo17/tfs-mt)](https://github.com/Giovo17/tfs-mt/releases)
[![Build status](https://img.shields.io/github/actions/workflow/status/Giovo17/tfs-mt/main.yml?branch=main)](https://github.com/Giovo17/tfs-mt/actions/workflows/main.yml?query=branch%3Amain)
[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/Giovo17/tfs-mt/blob/main/LICENSE)
[![License: CC BY-SA 4.0](https://img.shields.io/badge/license-CC_BY_SA_4.0-green.svg)](https://creativecommons.org/licenses/by-sa/4.0/)
[![pypi monthly downloads](https://img.shields.io/pypi/dm/tfs-mt)](https://pypi.org/project/tfs-mt/)


[▶️ Getting started](#-getting-started) • [📖 Documentation](https://giovo17.github.io/tfs-mt) • [🤗 Hugging Face](https://huggingface.co/giovo17/tfs-mt) • [🎬 Demo](https://huggingface.co/spaces/giovo17/tfs-mt-demo)

</div>

<hr>

This project implements the Transformer architecture from scratch considering Machine Translation as the usecase. It's mainly intended as an educational resource and a functional implementation of the architecture and the training/inference logic.

## Getting Started

### From PyPI

```bash
pip install tfs-mt
```

### From source

#### Prerequisites

- `uv` [[install](https://docs.astral.sh/uv/#installation)]

#### Steps
```bash
git clone https://github.com/Giovo17/tfs-mt.git
cd tfs-mt

uv sync

cp .env.example .env
# Edit .env file with your configuration
```

## Usage

### Training

To start training the model with the default configuration:

```bash
uv run src/train.py
```

### Inference

To run inference using the trained model from the [HuggingFace repo](https://huggingface.co/giovo17/tfs-mt):

```bash
uv run src/inference.py
```

### Configuration

The whole project parameters can be configured in `src/tfs_mt/configs/config.yml`. Key configurations include:

- **Model Architecture**: Config, dropout, GloVe embedding init, ...
- **Training**: Optimizer, Learning rate scheduler, number of epochs, ...
- **Data**: Dataset, Dataloader, Tokenizer, ...

## Technical walkthrough

The technical breakdown of the architecture and the training/inference workflows can be found here:

- [Architecture](https://giovo17.github.io/tfs-mt/architecture_explain/)

- [Training and Inference](https://giovo17.github.io/tfs-mt/training_inference/introduction/)

### Model Sizes

The project supports various model configurations to suit different computational resources:

| Parameter                | Nano     | Small    | Base     | Original |
| :----------------------- | :------- | :------- | :------- | :------- |
| **Encoder Layers** | 4        | 6        | 8        | 6        |
| **Decoder Layers** | 4        | 6        | 8        | 6        |
| **d_model**        | 50       | 100      | 300      | 512      |
| **Num Heads**      | 4        | 6        | 8        | 8        |
| **d_ff**           | 200      | 400      | 800      | 2048     |
| **Norm Type**      | PostNorm | PostNorm | PostNorm | PostNorm |
| **Dropout**        | 0.1      | 0.1      | 0.1      | 0.1      |
| **GloVe Dim**      | 50d      | 100d     | 300d     | -        |

## Documentation

Full documentation is available at [https://giovo17.github.io/tfs-mt/](https://giovo17.github.io/tfs-mt/).

## License

- **Source code**: licensed under the **MIT License**.
  - Note: This project includes modified code derived from PyTorch Ignite, which is licensed under the **BSD 3-Clause License**. See the `LICENSE` file for the full text of both licenses and original copyright notices.

- **Documentation**: located in the `docs/` directory, licensed under **Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)**. See `docs/LICENSE`.

## Citation

If you use `tfs-mt` in your research or project, please cite:

```bibtex
@software{Spadaro_tfs-mt,
author = {Spadaro, Giovanni},
licenses = {MIT, CC BY-SA 4.0},
title = {{tfs-mt}},
url = {https://github.com/Giovo17/tfs-mt}
}
```
