Metadata-Version: 2.4
Name: ctao-qualpipe-webapp
Version: 0.1.2
Summary: CTAO DPPS QualPipe Web Application
Author-email: Matteo Balbo <matteo.balbo@cta-consortium.org>, Mykhailo Dalchenko <mykhailo.dalchenko@cta-consortium.org>
License-Expression: BSD-3-Clause
Project-URL: repository, https://gitlab.cta-observatory.org/cta-computing/dpps/qualpipe/qualpipe-webapp
Project-URL: documentation, http://cta-computing.gitlab-pages.cta-observatory.org/dpps/qualpipe/qualpipe-webapp
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi
Requires-Dist: uvicorn
Requires-Dist: ctao-qualpipe
Requires-Dist: pydantic
Requires-Dist: pyyaml
Requires-Dist: traitlets
Provides-Extra: frontend
Requires-Dist: jinja2; extra == "frontend"
Requires-Dist: libsass; extra == "frontend"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pip-tools>=7.1; extra == "test"
Requires-Dist: ctao-qualpipe-webapp[frontend]; extra == "test"
Requires-Dist: httpx; extra == "test"
Requires-Dist: pytest-requirements; extra == "test"
Provides-Extra: doc
Requires-Dist: ctao-qualpipe-webapp[frontend]; extra == "doc"
Requires-Dist: sphinx; extra == "doc"
Requires-Dist: numpydoc; extra == "doc"
Requires-Dist: ctao-sphinx-theme~=0.1.2; extra == "doc"
Requires-Dist: sphinx-changelog; extra == "doc"
Provides-Extra: webdev
Requires-Dist: watchdog; extra == "webdev"
Provides-Extra: dev
Requires-Dist: setuptools_scm; extra == "dev"
Requires-Dist: towncrier; extra == "dev"
Requires-Dist: ctao-qualpipe_webapp[test,webdev]; extra == "dev"
Provides-Extra: all
Requires-Dist: ctao-qualpipe-webapp[dev,doc,test]; extra == "all"
Dynamic: license-file

# QualPipe-Webapp

<!-- ## 🚀 FastAPI + D3.js QualPipe Dashboard Project -->

The QualPipe-webapp project uses **FastAPI** as backend and frontend, plus **D3.js** for dynamic frontend visualizations, separated into clean services using **Docker Compose**.

## 🚀 Quick Start

### With Docker (Recommended for users)
```bash
make up                    # Start all services
# Open http://localhost/
make down                  # Stop all services
```

### With Pixi (Recommended for developers)
```bash
pixi shell                 # Activate environment (optional)
pixi run dev-setup         # Install dependencies + generate models
pixi run generate-codegen  # Regenerate models (if needed, with schema validation files)
```

---

## 📂 Project Structure

```bash
/qualpipe-webapp
│
├── docker-compose.dev.yml           # Docker compose for developer
├── docker-compose.yml               # Docker compose for user
├── Makefile                         # Makefile to build Backend and Frontend
│
├── docs/                            # Document folder
│
├── src/
│   └── qualpipe_webapp/             # FastAPI backend
│       ├── backend/                 # FastAPI backend
│       │   ├── main.py              # Main FastAPI app
│       │   ├── data/                # JSON data sources
│       │   ├── requirements.txt     # Backend dependencies
│       │   ├── requirements-dev.txt # Backend dependencies for developer
│       │   └── Dockerfile           # Backend container
│       │
│       ├── frontend/                # FastAPI frontend
│       │   ├── /templates/          # Template pages
│       │   ├── /static/             # Static files (css, js, lib)
│       │   ├── main.py              # Main FastAPI app
│       │   ├── requirements.txt     # Frontend dependencies
│       │   ├── requirements-dev.txt # Frontend dependencies for developer
│       │   └── Dockerfile           # Frontend container
│       │
│       ├── nginx/
│       │   ├── nginx.conf
│       │   └── ssl/                 # (optional later)
│
└── .gitignore
```

## Instructions

### Docker Deployment

To run it in the background, from the project folder `qualpipe-webapp` simply execute:

```bash
make up
```

Then open your browser:

- Frontend: http://localhost/

- API: http://localhost/api/data

To stop everything, execute:

```bash
make down
```

### Pixi Development

For development with automatic model generation:

```bash
# Activate pixi environment (optional)
pixi shell

# Setup environment and generate models
pixi run dev-setup

# Manually regenerate models (if criteria change)
pixi run generate-codegen

# Or use the installed console script
qualpipe-generate-models --help
```

If you also want to automatically see the logs you can then run it via docker compose

To run it the first time execute: (later on you can remove the `--build`)

```bash
docker-compose up --build
```

To stop it you can soft-kill with <kbd>CTRL-C</kbd> and shut down all the services executing `make down`.

If instead you do any modification to the `config`  while the containers are running or in case Docker caches wrong stuff, you can restart it running:

```bash
make restart
```

or

```bash
docker-compose down -v --remove-orphans
docker-compose up --build
```

## 🛠️ Makefile Commands

Description of all the `Makefile` functionalities:

| Command            | What it does                                                |
| ------------------ | ----------------------------------------------------------- |
| make build         | Build the Docker images                                     |
| make up            | Build and start services (backend + frontend) in background |
| make down          | Stop all services                                           |
| make logs          | View combined logs (both frontend and backend)              |
| make logs-backend  | View only backend logs                                      |
| make logs-frontend | View only frontend logs                                     |
| make restart       | Restart the containers                                      |
| make prune         | Clean up unused Docker containers/images                    |
| ------------------ | ----------------------------------------------------------- |
| make build-dev     | Build the Docker images                                     |
| make up-dev        | Build and start services (backend + frontend) in background |

## 🔧 Code Generation with Pixi

This project uses **Pixi** for package management and includes automated code generation for Pydantic models based on qualpipe criteria, and automatic schema generation.

### Prerequisites

Install [Pixi](https://pixi.sh/latest/) on your system:

```bash
# On macOS
curl -fsSL https://pixi.sh/install.sh | bash

# On Linux
curl -fsSL https://pixi.sh/install.sh | bash

# Or using conda/mamba
conda install -c conda-forge pixi
```

### Available Pixi Commands

| Command | Description |
|---------|-------------|
| `pixi run dev-setup` | Complete development setup (install + generate models) |
| `pixi run install` | Install the package with dependencies |
| `pixi run install-all` | Install with all optional dependencies |
| `pixi run generate-codegen` | Generate Pydantic models from qualpipe criteria |
| `pixi run generate-frontend-schema` | Generate yaml file for javascript frontend validation |
| `pixi shell` | Activate the pixi environment |

### Code Generation Workflow

The project automatically generates Pydantic models from qualpipe criteria classes:

```bash
# Generate models manually
pixi run generate-codegen

# Or use the console script directly (after installation)
qualpipe-generate-models --module qualpipe.core.criterion \
    --out-generated src/qualpipe_webapp/backend/generated \
    --out-schemas src/qualpipe_webapp/frontend/static
```

### Generated Files

The code generation creates:

- **Python Models**: `src/qualpipe_webapp/backend/generated/qualpipe_criterion_model.py`
  - Pydantic models for each criterion type
  - Validation logic for telescope parameters
  - Type-safe configuration classes

- **JSON/YAML Schemas**: `src/qualpipe_webapp/frontend/static/`
  - `criteria_schema.json` - JSON schema for frontend criteria report validation
  - `criteria_schema.yaml` - YAML schema for configuration

### Frontend schema generation

Such files are then implemented into `metadata_schema.yaml` with:

```bash
pixi run generate-frontend-schema
```

which is used for complete frontend validations, with all the correct references reads from the configuration file `config.js`. The `metadata_schema.yaml` file is auto-generated from a template, so

> [!IMPORTANT]
> Do not edit `metadata_schema.yaml` directly — edit `template_metadata_schema.yaml` instead.

### Development Setup with Pixi

1. **Clone and setup environment**:
   ```bash
   git clone <repository-url>
   cd qualpipe-webapp
   pixi run dev-setup  # Install dependencies then generate models and schema files
   ```

2. **Activate the environment**:
   ```bash
   pixi shell
   ```

3. **Run tests**:
   ```bash
   pixi run test
   ```

4. **Regenerate models and schema files** (if qualpipe criteria change):
   ```bash
   pixi run generate-codegen
   pixi run generate-frontend-schema
   ```

### Integration with CI/CD

The generated models are automatically created during:
- **Local development**: `pixi run dev-setup` runs code generation
- **CI/CD pipelines**: Code generation runs before tests
- **Package installation**: Post-install hooks generate models

> [!IMPORTANT] Important Notes
> ⚠️ **Generated files are git-ignored** - They're created automatically and should not be committed.
>
> ✅ **Tests depend on generated models** - Always run code generation before testing.
>
> 🔄 **Automatic regeneration** - Models update automatically when qualpipe criteria change.

### 🛠️ Troubleshooting

**Import errors with generated models?**
```bash
pixi run generate-codegen  # Regenerate models
```

<a id="tests-failing"></a>
**Tests failing?**
```bash
# Ensure models and schemas are generated before running tests
pixi run generate-codegen
pixi run generate-frontend-schema
pixi run test
```

**Pixi environment issues?**
```bash
pixi clean              # Clean cache
pixi run dev-setup      # Complete reinstall + code generation
```

## Developers

For developers, you can build containers that automatically file changes, avoiding the need to restart the containers every time. To do so, simply execute:

```bash
make build-dev
make up-dev
```

This will automatically output also every log produced.

### 🧪 Running Tests

#### 1. Python Tests with Pytest

**Install dependencies:**
```bash
pip install -r src/qualpipe_webapp/backend/requirements-dev.txt
pip install -r src/qualpipe_webapp/frontend/requirements-dev.txt
```

**Run tests:**
```bash
pytest -v src/qualpipe_webapp/backend/tests/
pytest -v src/qualpipe_webapp/frontend/tests/
```

**Alternative method**

The same result can be achieved in a more compact way using pixi:

```bash
pixi run test
```

---

#### 2. JavaScript Unit Tests with Mocha

Install [NodeJS](https://nodejs.org/en) from the official website.

**Install NodeJS dependencies:**
```bash
npm install
```

**Run all JS unit tests:**
```bash
npx mocha src/qualpipe_webapp/frontend/unit_tests_js/ --reporter spec
```

**Run a specific test file:**
```bash
npx mocha src/qualpipe_webapp/frontend/unit_tests_js/base.test.js --reporter spec
```

The report will be automatically saved in the file `test-results.xml`

> [!NOTE]
> Add new unit tests in the folder: `src/qualpipe_webapp/frontend/unit_tests_js/`

If you also want to see the coverage of your unit-tests simply execute:

```bash
npm run coverage
```

---

#### 3. End-to-End Tests with Playwright

In order to run end-to-end interface test with the real frontend DOM you need first to install [NodeJS](https://nodejs.org/en) and its dependencies (see first step of previous section).

**Install Playwright browsers:**

To install the Playwright library and the browsers (Chromium, Firefox, WebKit) necessary to run the tests, execute in the terminal:
```bash
npx playwright install
```

**Start the FastAPI webapp (in another terminal):**
```bash
make up-dev
```
or
```bash
make up
```

**Run all Playwright E2E tests:**
```bash
npx playwright test
```

**Run a specific Playwright test:**

To run e.g. the `home.test.js` test execute:

```bash
npx playwright test src/qualpipe_webapp/frontend/tests_e2e_js/home.test.js
```

In this way Playwright will open the browser, load the page, and verify that the DOM successfully pass the selected test.

**Generate and view HTML test report:**
```bash
npx playwright test --reporter html
npx playwright show-report
```
It will serve the HTML report at http://localhost:9323. Press <kbd>Ctrl+C</kbd> to quit.

> [!NOTE]
> Add new end-to-end tests in the folder: `src/qualpipe_webapp/frontend/tests_e2e_js/`

#### 🛠️ Troubleshooting

> [!IMPORTANT] Playwright test are failing for a specific browser?
> You can execute Playwright tests on a specific browser (chromium, firefox, webkit)
>
>```bash
>npx playwright test --project=chromium
>```
>
>⚠️ **Note:** WebKit browser generally works only with the latest Mac OS update.

> [!IMPORTANT] Some tests about validation are failing?
> Ensure you have generated models and schema (see the [Tests failing?](#tests-failing) section for detailed instructions.)

> [!IMPORTANT] All test failing with ERR_CONNECTION_REFUSED?
> Verify that the WebApp is running. Open a browser and navigate to http://localhost/home . If you can't display the page, the WebApp is not running. See instruction above on how to run it.

---

## 👩‍💻 Contributing

If you want to contribute in developing the code, be aware that we are using `pre-commit`, `code-spell` and `ruff` tools for automatic adherence to the code style. To enforce running these tools whenever you make a commit, setup the [`pre-commit hook`][pre-commit] executing:

```
pre-commit install
```

The `pre-commit hook` will then execute the tools with the same settings as when a merge request is checked on GitLab, and if any problems are reported the commit will be rejected. You then have to fix the reported issues before tying to commit again.

## 📄 License

This project is licensed under the BSD 3-Clause License. See the LICENSE file for details.

[pre-commit]:https://pre-commit.com/

---

## Useful Links

- [pre-commit documentation](https://pre-commit.com/)
- [Mocha documentation](https://mochajs.org/)
- [Playwright documentation](https://playwright.dev/)
- [Pytest documentation](https://docs.pytest.org/)
