# q2netcdf

[![Python Version](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/mousebrains/q2netcdf/actions/workflows/ci.yml/badge.svg)](https://github.com/mousebrains/q2netcdf/actions/workflows/ci.yml)
[![Coverage](https://codecov.io/gh/mousebrains/q2netcdf/branch/main/graph/badge.svg)](https://codecov.io/gh/mousebrains/q2netcdf)
[![PyPI](https://img.shields.io/pypi/v/q2netcdf)](https://pypi.org/project/q2netcdf/)

> **Note**: The package requires Python 3.10+, but the standalone `mergeqfiles` script is compatible with Python 3.7+ for deployment on legacy systems (e.g., MicroRider instruments).

Python tools for manipulating [Rockland Scientific's](https://rocklandscientific.com) Q-files as generated by the [ISDP data logger](https://rocklandscientific.com/news/rockland-data-logger/).

## Features

- **Q-file to NetCDF Conversion**: Convert binary Q-files to CF-1.8 compliant NetCDF format
- **Header Parsing**: Extract metadata, sensor configurations, and timing information
- **Data Extraction**: Read scalar channels and frequency spectra from binary records
- **File Manipulation**: Merge, reduce, and inspect Q-files
- **Configuration Generation**: Create syntactically correct `isdp.cfg` files
- **Format Support**: Handles Q-file versions 1.2 and 1.3
- **Sensor Mapping**: Built-in support for 200+ sensor identifier codes

## Command-Line Tools

- **`q2netcdf`** - Translate Q-files into CF-1.8 compliant NetCDF files
- **`QFile`** - Dump header and data records from Q-files
- **`QHeader`** - Display header record information
- **`QReduce`** - Reduce Q-file size by removing records
- **`mergeqfiles`** - Merge multiple Q-files (designed for MicroRider/Slocum Glider integration)
- **`mkISDPcfg`** - Generate syntactically correct `isdp.cfg` configuration files

## Quick Start

```bash
# Display Q-file header information
QHeader data.q

# Convert Q-file to NetCDF
q2netcdf data.q --nc output.nc

# Dump all records from Q-file
QFile data.q --logLevel INFO
```

## Installation

### From PyPI

```bash
pip install q2netcdf
```

Or using [pipx](https://pipx.pypa.io/stable/installation/) for the command-line tools:

```bash
pipx install q2netcdf
```

### From Source

```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e .
```

### For Development

```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest  # Run tests
```

## Installation on the MicroRider

For deployment on Rockland Scientific MicroRider instruments:

1. Transfer [src/q2netcdf/mergeqfiles.py](src/q2netcdf/mergeqfiles.py) and [src/q2netcdf/mergeqfiles.cfg](src/q2netcdf/mergeqfiles.cfg) to the MicroRider's data directory.
2. Dismount the data directory on your host computer.
3. Connect to the MicroRider via a serial connection using your favorite terminal emulator at 115,200 baud.
4. `ls data` to verify the files are present. This will cause the MicroRider to unmount data from being read-only to the MicroRider and then remount it as read-write. This will typically cause the serial connection to drop then be reestablished. You might have to restart your terminal emulator's connection.
5. `mv data/mergeqfiles.py /usr/local/bin/mergeqfiles` to install the script as mergeqfiles.
6. Test with `mergeqfiles --help` to make sure the installation was successful.
7. You can disconnect the MicroRider's serial connection.
8. You should now run a simulation on the Slocum glider to verify Q-files are being uploaded to the science computer's storage as .mri files.
9. Diagnostic information is written into the data/mergeqfiles.log file on the MicroRider.
10. You can modify data/mergeqfiles.cfg to adjust the information included in the .mri file sent to the glider.

This is designed for integration with [TWR's Slocum Glider uRider proglet](https://www.teledynemarine.com/brands/webb-research/slocum-glider).

## Python API

```python
from q2netcdf import QFile

# Stream records from a Q-file
with QFile("data.q") as qf:
    header = qf.header()
    print(header.channels(), header.spectra())
    for record in qf.data():
        print(record.stime, record.channels)

# Convert a Q-file to an xarray Dataset
from q2netcdf.q2netcdf import loadQfile

ds = loadQfile("data.q")
print(ds)
ds.to_netcdf("output.nc")
```

## Usage Examples

### Converting Q-files to NetCDF

```bash
q2netcdf data.q --nc output.nc
```

### Inspecting Q-file Headers

```bash
# Show full header with channel and spectra names
QHeader data.q

# Show condensed header
QHeader data.q --nothing

# Show only configuration
QHeader data.q --channels --spectra --frequencies
```

### Examining Q-file Data Records

```bash
# Dump all records
QFile data.q

# Limit number of records displayed
QFile data.q --n 10
```

### Merging Q-files

```bash
mergeqfiles -o merged.q file1.q file2.q file3.q
```

### Generating ISDP Configuration Files

```bash
mkISDPcfg --help
```

## File Format

Q-files are binary files containing oceanographic data from Rockland Scientific instruments. The format is documented in Rockland Technical Note TN-054.

### Q-file Structure
- **Header Record** (0x1729): File version, timestamp, channel/spectra identifiers, frequencies, configuration
- **Data Records** (0x1657): Timestamped measurements with scalar channels and frequency spectra

### Supported Versions
- **v1.2**: Original format with full metadata in each data record
- **v1.3**: Optimized format with reduced redundancy

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](docs/CONTRIBUTING.md) for guidelines.

### Development Setup

```bash
git clone git@github.com:mousebrains/q2netcdf.git
cd q2netcdf
pip install -e ".[dev]"
pytest                    # Run tests
ruff check src/ tests/    # Lint
mypy src/                 # Type check
```

To run a module directly: `python3 -m q2netcdf.QHeader --help`

## Additional Documentation

- [CHANGELOG.md](docs/CHANGELOG.md) - Version history and changes
- [CONTRIBUTING.md](docs/CONTRIBUTING.md) - Contributor guidelines and development setup
- [SECURITY.md](docs/SECURITY.md) - Security policy and vulnerability reporting
- [QFILE_FORMAT.md](docs/QFILE_FORMAT.md) - Q-file binary format specification

## License

This project is licensed under the [GNU General Public License v3.0 or later](LICENSE).

## Author

**Pat Welch**
Email: pat@mousebrains.com

## Acknowledgments

Developed for use with [Rockland Scientific](https://rocklandscientific.com) oceanographic instruments and the ISDP data logger.
