# `pipeline-eds`

`pipeline-eds` is a Python project designed to simplify API access to Emerson Enterprise Data Server (EDS) machines. It facilitates seamless data exchange between Emerson's Ovation local systems and various external parties, including third-party contractors and internal employees. The project is distributed on PyPI under the package name `pipeline-eds`.

<br>
<hr>
<br>

## 🚀 Getting Started

This section provides a quick guide to help you get `pipeline` up and running. Choose the setup method that best suits your needs: CLI-only usage or local development.

### 💻 CLI Installation (Recommended for End-Users)

For a simple command-line interface (CLI) experience, **`pipx`** is the recommended installation method. `pipx` installs and runs Python applications in isolated environments, preventing conflicts with your system's Python packages.

1.  **Install `pipx`**
    If you don't have `pipx` installed, you can get it with `pip`:
    ```bash
    pip install pipx
    python -m pipx ensurepath
    ```
2.  **Install `pipeline-eds` with `pipx`**
    Install the package directly from PyPI. If you need Windows-specific dependencies like `pyodbc` and `matplotlib`, use the `[windows]` extra.
    ```bash
    pipx install pipeline-eds
    # For Windows users:
    pipx install "pipeline-eds[windows]"
    ```
3.  **Run CLI Commands**
    The `pyproject.toml` file defines `pipeline`, `eds`, and `pipeline-eds` as command-line aliases. Once installed, you can use any of these aliases directly from your terminal.
    ```bash
    eds configure
    eds trend M100FI --start June3 --end June17
    ```



### 🛠️ Developer & Contributor Setup

If you plan to contribute to the project or need to work with the source code, follow these steps to set up a full development environment.

1.  **Clone the Repository**
    Start by cloning the project from GitHub and navigating into the directory:
    ```bash
    git clone https://github.com/City-of-Memphis-Wastewater/pipeline.git
    cd pipeline
    ```
2.  **Install `pyenv` and `Poetry`**
    This project uses **`pyenv`** for managing Python versions and **`Poetry`** for dependency management. This combination ensures a clean, reproducible development environment without interfering with your system's Python installation.
      * **Install `pyenv`:** Refer to the official `pyenv` documentation for your operating system ([pyenv-win](https://github.com/pyenv-win/pyenv-win) for Windows, [pyenv](https://github.com/pyenv/pyenv) for Linux/macOS).
      * **Install `Poetry`:** See the [Poetry documentation](https://www.google.com/search?q=https://python-poetry.org/docs/%23installation) for installation instructions.
3.  **Configure the Environment**
    Use `pyenv` to set the Python version for the project and then tell Poetry to use that version:
    ```bash
    pyenv install 3.11.9
    pyenv local 3.11.9
    poetry env use 3.11.9
    ```
4.  **Install Dependencies**
    Poetry will read the `pyproject.toml` file and install all necessary packages into a new virtual environment:
    ```bash
    poetry install
    ```
5.  **Run Development Commands**
    Once installed, you can execute commands using `poetry run`:
    ```bash
    poetry run python -m pipeline.cli
    poetry run eds ping # 
    ```
    This ensures that all commands run within the project's isolated environment.
	You can run `poetry run eds` directly because of the `[tool.poetry.scripts]` section in the `pyproject.toml`, which states that `eds = "pipeline.cli:app"`.

<br>
<hr>
<br>

## 📦 Distribution and Packaging

This project supports multiple packaging formats to make installation flexible across platforms.  
While some formats allow installation on systems without internet access, note that the **application itself requires internet connectivity** to call its API.

### 🔹 Executables (`.exe`, `.elf`)
- **Generated by**: [`build_executable.py`](./build_executable.py)
- **Variants**:
  - **Windows `.exe`**: Tested on Windows 11. Runs standalone without requiring Python.
  - **Linux `.elf`**:
    - Built on **WSL2 Ubuntu** for general Linux systems.
    - Built on **Termux** for Android devices.  
      - Smoothest rollout on Termux: no need to install Python separately.
      - Avoids the `.shortcuts` widget permission error seen with `.pyz`.
- **Internet required for install**: ❌ (binaries can be copied directly)  
- **Internet required for use**: ✅ (API calls)  

### 🔹 Python Zip App (`.pyz` + `.bat`)
- **Generated by**: [`build_shiv.sh`](./build_shiv.sh) on WSL2 Ubuntu
- **Best for**: Systems that already have Python installed.
- **Windows support**: A `.bat` launcher is provided for smoother execution.
- **Termux notes**:
  - Works, but calling `.pyz` from the Termux `.shortcuts` widget can trigger a **permission error**.
  - Requires Python to be installed on Termux.
- **Maintainability**: Easier to update regularly compared to static binaries.
- **Internet required for install**: ❌ (once `.pyz` is copied)  
- **Internet required for use**: ✅  

### 🔹 `pipx` Install
- **Best for**: Staying current with rolling changes.
- **Update shortcut**: On Termux, an update shortcut is available directly from the home screen widget.
- **Requirements**: Python and `pipx` installed.
- **Internet required for install/update**: ✅  
- **Internet required for use**: ✅  

### 🔹 Source Distributions (`.tar.gz`, `.whl`)
- **Generated by**: Poetry
- **Best for**: Developers or environments where building from source is preferred.
- **Internet required for install**: ✅ (to fetch dependencies)  
- **Internet required for use**: ✅  

### 🔹 Docker Containers
- **Generated by**: Dockerfiles in the repository
- **Best for**: Containerized deployments where dependencies and environment isolation are important.
- **Notes**:
  - Provides a reproducible runtime environment.
  - Useful for CI/CD pipelines or server deployments.
  - Currently built manually; orchestration (e.g., automated builds, registry publishing) is a **future goal**.
- **Internet required for install**: ✅ (to pull base images and dependencies)  
- **Internet required for use**: ✅  

### 🌐 Connectivity Summary

| Format          | Install Without Internet | Python Needed | Best Use Case |
|-----------------|--------------------------|---------------|---------------|
| `.exe`          | Yes                      | No            | Windows systems, simple rollout |
| `.elf` (Ubuntu) | Yes                      | No            | Linux servers/desktops |
| `.elf` (Termux) | Yes                      | No            | Android/Termux, smoothest rollout |
| `.pyz` + `.bat` | Yes                      | Yes           | Python‑ready systems, maintainable updates |
| `pipx`          | No                       | Yes           | Always‑updated installs, Termux widget support |
| `.tar.gz`/`.whl`| No                       | Yes           | Developers building from source |
| Docker          | No (manual build)        | No            | Containerized deployments, CI/CD |

---

### ⚠️ Notes and Limitations
- These packages simplify **installation** on disconnected systems, but the application itself requires internet access to function (API calls).
- **Alpine / iSH (iPhone)**: MUSL‑compatible packaging is not yet available. The main barrier is credential storage, as current builds rely on `cryptography` and `keyring`.
- **Docker orchestration**: Currently builds are processed manually. Future goals include automated orchestration and registry publishing.

---

## 🛠️ Build Scripts

### `build_executable.py`
- Automates creation of `.exe` and `.elf` binaries.
- Targets:
  - Windows standalone `.exe`
  - Linux `.elf` (both WSL2 Ubuntu and Termux builds)
- Intended for distributing binaries to systems without requiring Python ins>
- Currently run manually; future goal is to integrate into an automated buil>

### `build_shiv.sh`
- Automates creation of `.pyz` zipapps and corresponding `.bat` launchers.
- Targets:
  - Cross‑platform `.pyz` archives
  - Windows `.bat` wrapper for smooth launching
- Provides a maintainable, portable distribution option for Python‑ready sys>
- Currently run manually; orchestration and CI/CD integration are planned.

---

## ✨ Tips for Optimal Usage & Maintenance
To ensure a smooth and efficient experience with `pipeline-eds`, consider the following best practices:

### Keep `pipeline-eds` Updated: 
Regularly upgrade your `pipeline-eds` installation to benefit from the latest features, bug fixes, and performance improvements.
```bash
pipx upgrade pipeline-eds
```
This command will update pipeline-eds and its dependencies in its isolated pipx environment.

### Maintain Your Python Environment:

- Desktop Users: While `pipx` isolates `pipeline-eds`, it's good practice to keep your underlying Python installation updated.
- Termux Users: Regularly update your Termux environment and packages to ensure compatibility and security:
```bash
pkg update && pkg upgrade
pkg install rust
```

### Understanding eds configure and Credential Management:
The first time you execute a command requiring access to your EDS API (e.g., eds trend), `pipeline-eds` will guide you through a one-time configuration process. Your sensitive API credentials (URL, username, password) are securely stored using your operating system's native keyring service. This is a robust and secure method that avoids storing plaintext passwords in files. If your credentials change, you can re-run eds configure at any time to update them.

### Network Connectivity (VPN Essential):
A critical requirement for `pipeline-eds` to function is proper network connectivity to your Emerson Ovation EDS machine. If your EDS server is located on a private network (e.g., within your organization's internal network), you must be connected to the appropriate Virtual Private Network (VPN). Failure to do so will result in connection errors when `pipeline-eds` attempts to fetch data.

### Leveraging Flexible Date/Time Inputs:
The `eds trend` command offers highly flexible date and time parsing for its --start and --end options, thanks to the `pendulum` package. You can use a wide variety of natural language inputs, such as:

- `--start "2023-09-18"`
- `--start "Sept 18"`
- `--end "now"` 

Experiment with different formats to suit your query needs. Remember to use quotes around values if they contain spaces.
It would be great if terms like "yesterday" worked, but these fail to be recognized by `pendulum`.

<br>
<hr>
<br>

## 🔐 Security & Configuration

`pipeline` uses a two-tiered approach to manage configuration and secrets.

  * **Non-Sensitive Configuration**: Non-sensitive settings like URLs and paths are stored in a local JSON file (`~/.pipeline-eds/config.json`). This file is easy to inspect and manage.
  * **Secrets and Credentials**: For CLI users, API credentials and passwords are **securely stored** using your operating system's native keyring. This is a much safer alternative to storing plaintext passwords in a file. The `pipeline configure` command guides you through this one-time setup process.

**Note for Developers**: While the CLI now uses the keyring, some functionality within the codebase still relies on the `secrets.yaml` file for credential management. This file is not required for general CLI usage but may be necessary for specific development workflows and legacy components.

**Important**: You must be on the same network as your server (e.g., via a VPN) if it is not publicly accessible.

<br>
<hr>
<br>

## ⚙️ Project Implementation & Use Cases

`pipeline` is designed to be deployed as a scheduled task on a Windows server.

  * The project is executed by **Windows Task Scheduler**, which calls a PowerShell script (`main_eds_to_rjn_quiet.ps1`) as the entry point.
  * The iterative timing (e.g., hourly execution) is handled by the `Task Scheduler`, not by Python.
  * For these automated tasks, a standard `venv` is used, as `Task Scheduler` can run under different user accounts.

<br>
<hr>
<br>

## 📱 Running on Android (`Termux`)

The `pipeline` project can be installed and run on Android devices using the **[Termux](https://termux.dev/)** terminal emulator.  
For most users, **CLI installation via `pipx` is the recommended method**, as development is not expected in this environment and `pipx` provides the smoothest way to stay up to date.

### Termux Installation Options

There are several ways to run `pipeline` inside Termux, depending on your needs:

- **`pipx` (Recommended)**  
  - Provides rolling updates and integrates with the Termux home‑screen widget.  
  - As of **v0.3.8**, you can add a **shortcut button** to update `pipeline` directly from the Termux widget without opening the terminal.  
  - Requires Python and `pipx` to be installed.  
  - Best choice if you want to keep up with frequent changes.

- **Native ELF Binary**  
  - A prebuilt `.elf` binary is available specifically for Termux.  
  - Runs without requiring Python to be installed.  
  - Avoids the `.shortcuts` widget permission error that can occur when launching `.pyz` apps.  
  - Smoothest rollout for users who just want the CLI tool without managing Python environments.

- **`.pyz` Zipapp**  
  - Works if Python is already installed in Termux.  
  - More maintainable than static binaries since you can replace the `.pyz` file with newer versions.  
  - ⚠️ Known limitation: launching `.pyz` directly from the Termux `.shortcuts` widget may trigger a **permission error**.

---

### Termux Limitations

- **No `pyenv` or `Poetry`**: Package management must be done with `pip` directly. You can and should use `venv` for isolation.  
- **Limited Library Support**: Some libraries that require native compilation (e.g., `pandas`, `numpy`) or GUI dependencies are not supported on Termux.  
- **HTML Viewer**: You may need to manually configure the default HTML viewer to a full‑featured browser on Android.  

### Comparable Apple iPhone Rollout

- **Alpine (iSH on iOS)**: Not yet supported; credential storage via `cryptography` and `keyring` is the current blocker because they are not MUSL compatible..

---

### ✅ Recommendations

- Use **`pipx`** if you want the latest updates and widget integration.  
- Use the **native ELF binary** if you want the simplest, no‑Python setup.  
- Use **`.pyz`** only if you already have Python installed and are comfortable updating the archive manually.



### 🌐 Termux and Web-Based Visuals (Plotly)
When using `pipeline-eds` in Termux to generate plots (e.g., with `eds trend`), the visuals are displayed as web-based HTML pages using libraries like Plotly. Instead of directly opening a graphical window (which is not typically supported by Termux's command-line environment), `pipeline-eds serves` these HTML files via a local web server (often on localhost).

### Why localhost

- Termux Sandboxing: Termux operates in a sandboxed environment on Android. This security measure restricts direct access to certain system resources, including the ability to automatically launch web browsers or other GUI applications from the command line.
- Local Server Approach: To work around this, `pipeline-eds` acts as a small web server, making the generated HTML plot accessible at a specific localhost URL (e.g., [http://127.0.0.1:8000.](http://127.0.0.1:8000`.)
- Manual Opening: Improvemenrs have been made using `xdg-open` and `termux-url-open`, so manual opening of web graphics is no longer required. Due to the sandboxing, Termux has limited ways to automatically open this URL in your default Android browser, but it is possible and now suceeds. Another approach is the `am` command, or `termux-api` but those are not used in this package. If for any reason auyomatic launching fails, you canmanually copy the provided URL from the Termux output and paste it into your preferred web browser (e.g., Chrome, Firefox) on your Android device. This allows your full-featured browser to render the interactive Plotly graph.
- Security: Localhost-based plotting is also a security measure, ensuring that applications within Termux explicitly serve content, and the user is able to view it in a secure yet less restricted environment (the browser).  Using localhost allows for a minimal Termux installation, like from the Pay Store, without relying on `termux-gui` or `x11`.
  
<br>
<hr>
<br>

## 📝 Final Note on Naming
The project is internally referred to as `pipeline`, but the PyPI package is named `pipeline-eds` to avoid a name conflict with an existing, unrelated package on PyPI. For CLI usage, the pyproject.toml file creates aliases so you can use `pipeline`, `eds`, and `pipeline-eds` interchangeably in your terminal. This allows for a more intuitive command-line experience without the need to use the full PyPI package name.
