[![Unit Tests](https://github.com/mpm-tu-berlin/eflips-ingest/actions/workflows/unittests.yml/badge.svg)](https://github.com/mpm-tu-berlin/eflips-ingest/actions/workflows/unittests.yml)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

# eflips-ingest

---

Part of the [eFLIPS/simBA](https://github.com/stars/ludgerheide/lists/ebus2030) list of projects.

---

This repository contains code to import bus schedules from various sources into an [eFLIPS-Model](https://github.com/mpm-tu-berlin/eflips-model) database.

## Installation

1. Set up a [PostgreSQL](https://www.postgresql.org/) database with the [PostGIS](https://postgis.net/) extension and `BTREE_gist` enabled.
   ```bash
   apt install postgresql postgis
   sudo -u postgres psql createdb eflips
   sudo -u postgres psql eflips -c "CREATE EXTENSION postgis;"
   sudo -u postgres psql eflips -c "CREATE EXTENSION btree_gist;"
   ```

2. Clone this git repository (or [download a specific release](https://github.com/mpm-tu-berlin/eflips-depot/releases))
    ```bash
    git clone git@github.com:mpm-tu-berlin/eflips-model.git
    ```
3. Install the packages listed in `poetry.lock` and `pyproject.toml` into your Python environment. Notes:
    - This project depends on [pyproj](https://pyproj4.github.io/pyproj/stable/installation.html), which may require the `proj-bin` package (`apt install proj-bin` on Ubuntu).
    - The supported platforms are macOS and Linux.
    - Using the [poetry](https://python-poetry.org/) package manager is recommended. It can be installed accoring to the
      instructions listed [here](https://python-poetry.org/docs/#installing-with-the-official-installer).
    ```bash
    poetry install
    ```

## Usage

### Command line

The code is organized into various Python files under the `eflips/ingest` folder, each for ingesting a specific data source. These files should be runnable using `python eflips/ingest/x.py` The following data sources are currently supported:

- `bvgxml.py`: XML files emitted by BVG's proprietary software. 
  - Requires at least `GOOGLE_MAPS_API_KEY`, also `OPENELEVATION_URL` is suggested to save money on Google Maps API calls.
  - Known Limitations:
    - The source data is known to contain some partial rotations of the bus routes. These are imported, which may lead to errors when creating a depot for each start and end spot, as some roations start/end at spots that are not depots.
    - Some routes that are going from the same stop to the stop itself are not imported, as they are considered to be invalid.
    - Some route's distance and time is fudged, when it is found to be unrealistically short or long.
### API

---

All API Classes should inherit from `eflip.ingest.base.AbstractIngester`. This class also provides documentation on how to implement the required methods.

---

The `eflips-ingest` package is designed to provide a standardized access method to the different ingesters. It is
designed as a reusable package providing a programmatic API. It provides the `Ingester` class with the following
methods:

- `__init__(self, database_url: str)`: The constructor. It takes the database URL as a parameter.
- `prepare(self, progress_callback: None | Callable[[float], None] = None, **kwargs: Dict[str, str | int | float | bool | Enum | Path],) -> Tuple[bool, UUID | Dict[str, str]]:`: 
  This method is used to validate the data. It
    has multiple keyword arguments, one for each parameter. It returns a tuple containing a boolean indicating whether 
    the data is valid and a UUID if the data is valid or a dictionary containing the error names and messages if the 
    data is invalid. It is suggested that the input form is generated by 
    [introspecting](https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object) 
    this method. If a UUID is returned, this UUID can be used to start the ingestion process, even with another instance
    of this Ingester class (the validated input data is saved to a temporary directory).
- `ingest(self, UUID: UUID, progress_callback: None | Callable[[float], None]) -> None`: This method is used to start the 
    ingestion process. It takes the UUID returned by the `validate` method and a callback function that will be called 
    with the progress of the ingestion process. It is suggested that the progress is displayed in the UI. This method
    may raise an exception if the ingestion process fails. *Note that exceptions should not be raised during the
    ingest process if everything goes well, as we are checking validity in the `validate` method.*
- `prepare_param_names()` This property should return a dict with a short description of each parameter that the `prepare` method takes. This can be used for Naming fields in the UI.
- `prepare_param_descriptions()` This property should return a dict with a long description of each parameter that the `prepare` method takes. This can be used for tooltips in the UI.

In addition, the package provides a `get_ingesters` function that returns a list of `Ingester` classes.

## Testing

---

**NOTE**: Be aware that the tests will clear the database specified in the `DATABASE_URL` environment variable. Make sure that you are not using a database that you want to keep.

---

Testing is done using the `pytest` framework with tests located in the `tests`directory. To run the tests, execute the following command in the root directory of the repository:

```bash
   export PYTHONPATH=tests:. # To make sure that the tests can find the eflips package
   export DATABASE_URL=postgis://postgres:postgres@localhost:5432/postgres # Or whatever your database URL is
   export GOOGLE_MAPS_API_KEY=put_your_api_key_here # Required for some tests
   export OPENELEVATION_URL=put_your_url_here # Optional, required for some tests
   pytest
```



## Development

We utilize the [GitHub Flow](https://docs.github.com/get-started/quickstart/github-flow) branching structure. This means
that the `main` branch is always deployable and that all development happens in feature branches. The feature branches
are merged into `main` via pull requests.


We use [black](https://black.readthedocs.io/en/stable/) for code formatting. You can use 
[pre-commit](https://pre-commit.com/) to ensure the code is formatted correctly before committing. You are also free to
use other methods to format the code, but please ensure that the code is formatted correctly before committing.

Please make sure that your `poetry.lock` and `pyproject.toml` files are consistent before committing. You can use `poetry check` to check this. This is also checked by pre-commit.

## License

This project is licensed under the AGPLv3 license - see the [LICENSE](LICENSE.md) file for details.

## Funding Notice

This code was developed as part of the project [eBus2030+](https://www.eflip.de/) funded by the Federal German Ministry for Digital and Transport (BMDV) under grant number 03EMF0402.