# Q-File Format Specification

**Document Version:** 1.0
**Q-File Versions:** 1.2, 1.3
**Reference:** Based on Rockland Scientific Technical Note TN-054

---

## Overview

Q-files are binary files generated by Rockland Scientific's ISDP (Instrument Signal and Data Processor) data logger. They contain oceanographic microstructure measurements including shear probes, thermistors, and other sensors.

### Key Characteristics
- **Binary format** - Little-endian byte order
- **Self-describing** - Header contains metadata and sensor identifiers
- **Time-series data** - Sequential records with timestamps
- **Spectra support** - Both scalar channels and frequency spectra

---

## File Structure

```
┌─────────────────────────────────────┐
│      Header Record (0x1729)         │  Variable size
│  - Version, timestamp                │
│  - Channel identifiers               │
│  - Spectra identifiers               │
│  - Frequencies                       │
│  - Configuration                     │
├─────────────────────────────────────┤
│      Data Record (0x1657)           │  Fixed size
│  - Timestamp, measurements           │
├─────────────────────────────────────┤
│      Data Record (0x1657)           │
├─────────────────────────────────────┤
│             ...                      │
└─────────────────────────────────────┘
```

---

## Record Types

### Record Identifiers (16-bit hex)

| Identifier | Name | Description |
|------------|------|-------------|
| `0x1729` | HEADER | File header with metadata |
| `0x0827` | CONFIG_V12 | Configuration (v1.2 beta only) |
| `0x1657` | DATA | Data record with measurements |

---

## Header Record (0x1729)

### Fixed Header (20 bytes)

| Offset | Size | Type | Description |
|--------|------|------|-------------|
| 0 | 2 | uint16 | Identifier (0x1729) |
| 2 | 4 | float32 | Version (1.2 or 1.3) |
| 6 | 8 | uint64 | Timestamp (milliseconds since year 0) |
| 14 | 2 | uint16 | Nc - Number of channels |
| 16 | 2 | uint16 | Ns - Number of spectra |
| 18 | 2 | uint16 | Nf - Number of frequencies |

### Channel Identifiers (2 × Nc bytes)

Array of 16-bit hex identifiers for scalar channels:
```
[0x610, 0x611, 0x710, ...]  # sh_0, sh_1, dT_0, ...
```

### Spectra Identifiers (2 × Ns bytes)

Array of 16-bit hex identifiers for frequency spectra:
```
[0x920, 0x930, ...]  # shear_raw, shear_gfd, ...
```

### Frequencies (2 × Nf bytes)

Array of float16 frequency values in Hz:
```
[0.5, 1.0, 1.5, ..., 100.0]
```

### Configuration Record

**Version 1.2:**
```
uint16: Config identifier (0x0827 - often incorrect in beta)
uint16: Config size in bytes
bytes:  JSON-like configuration string
```

**Version 1.3:**
```
uint16: Config size in bytes
bytes:  JSON-like configuration string
```

### Configuration Format

Configuration is stored as a simplified JSON format:

```
{
  "fft_length": 4,
  "diss_length": 32,
  "f_aa": 98,
  "hp_cut": 0.125,
  "shear_despiking": [8.0, 0.25, 0.04],
  "algorithm": "glide",
  "instrument": "slocum_glider"
}
```

### Data Record Size (2 bytes)

Final field in header:
```
uint16: Size of each data record in bytes
```

---

## Data Record (0x1657)

### Version 1.2 Format

| Offset | Size | Type | Description |
|--------|------|------|-------------|
| 0 | 2 | uint16 | Identifier (0x1657) |
| 2 | 2 | uint16 | Record number |
| 4 | 8 | uint64 | Error code |
| 12 | 2 | float16 | Start time (seconds from header time) |
| 14 | 2 | float16 | End time (seconds from header time) |
| 16 | 2×Nc | float16[] | Channel values |
| 16+2×Nc | 2×Ns×Nf | float16[] | Spectra values (row-major) |

**Total size:** 16 + 2×Nc + 2×Ns×Nf bytes

### Version 1.3 Format (Optimized)

| Offset | Size | Type | Description |
|--------|------|------|-------------|
| 0 | 2 | uint16 | Identifier (0x1657) |
| 2 | 2 | float16 | Start time (seconds from header time) |
| 4 | 2×Nc | float16[] | Channel values |
| 4+2×Nc | 2×Ns×Nf | float16[] | Spectra values (row-major) |

**Total size:** 4 + 2×Nc + 2×Ns×Nf bytes

**Removed fields:** Record number, error code, end time (reduced redundancy)

---

## Sensor Identifiers

### Identifier Encoding (16-bit)

```
Bits 15-4:  Sensor type (0xFFF0)
Bits 3-0:   Instance number (0x000F)
```

**Example:**
- `0x610` = Shear probe #0 (type 0x610, instance 0)
- `0x611` = Shear probe #1 (type 0x610, instance 1)
- `0x710` = Temperature probe #0 (type 0x710, instance 0)

### Common Identifiers

| ID Range | Name | Description |
|----------|------|-------------|
| 0x010 | dT | Pre-emphasis thermal signal |
| 0x030 | incl_X | Inclinometer X-axis |
| 0x040 | incl_Y | Inclinometer Y-axis |
| 0x100 | W | Vertical velocity |
| 0x110 | U | Horizontal velocity (E-W) |
| 0x120 | V | Horizontal velocity (N-S) |
| 0x130 | P | Pressure |
| 0x140 | P0 | Atmospheric pressure |
| 0x150 | T | Temperature |
| 0x160 | T0 | Initial temperature |
| 0x170 | C | Conductivity |
| 0x180 | C0 | Initial conductivity |
| 0x200 | JAC_T | Temperature coefficient |
| 0x210 | JAC_C | Conductivity coefficient |
| 0x300 | Ax | Acceleration X |
| 0x310 | Ay | Acceleration Y |
| 0x320 | Az | Acceleration Z |
| 0x400 | Mx | Magnetometer X |
| 0x410 | My | Magnetometer Y |
| 0x420 | Mz | Magnetometer Z |
| 0x500 | gradT | Temperature gradient |
| 0x510 | gradC | Conductivity gradient |
| 0x520 | uT | Micro-temperature |
| 0x530 | uCond | Micro-conductivity |
| 0x610 | sh | Shear probe |
| 0x640 | dT | Temperature gradient |
| 0x650 | dC | Conductivity gradient |
| 0x710 | sh_GTD | Goodman shear |
| 0x810 | K_max | Integration limit |
| 0x820 | var_res | Variance resolved |
| 0x910 | freq | Frequency |
| 0x920 | shear_raw | Raw shear spectrum |
| 0x930 | shear_gfd | Goodman shear spectrum |

---

## Data Types

### Standard Types

| Type | Size | Range | Description |
|------|------|-------|-------------|
| uint16 | 2 bytes | 0 to 65,535 | Unsigned integer |
| uint64 | 8 bytes | 0 to 2^64-1 | Unsigned long integer |
| float16 | 2 bytes | ±65,504 | Half-precision float |
| float32 | 4 bytes | ±3.4×10^38 | Single-precision float |

### Byte Order
All multi-byte values use **little-endian** byte order (LSB first).

---

## Time Representation

### Header Timestamp
- **Type:** uint64
- **Units:** Milliseconds since year 0 (January 1, 0000)
- **Example:** `63787142400000` = January 1, 2022, 00:00:00 UTC

### Record Timestamps
- **Type:** float16
- **Units:** Seconds offset from header timestamp
- **Example:** `125.5` = 125.5 seconds after header time

**Absolute time calculation:**
```python
record_time = header_time + timedelta(seconds=record_offset)
```

---

## Configuration Parameters

### Common Parameters

| Parameter | Type | Description |
|-----------|------|-------------|
| `fft_length` | int | FFT length in seconds |
| `diss_length` | int | Dissipation length in seconds |
| `f_aa` | float | Anti-aliasing filter frequency (Hz) |
| `hp_cut` | float | High-pass filter cutoff (Hz) |
| `algorithm` | string | Processing algorithm ("glide", "profiler") |
| `instrument` | string | Instrument type |
| `shear_despiking` | array | [limit, fit_2_sd, fit_length] |
| `ucond_despiking` | array | [limit, fit_2_sd, fit_length] |
| `q` | float | Quality factor |
| `tau` | int | Time constant |
| `overlap` | int | FFT overlap (0 or 1) |
| `fit_order` | int | Polynomial fit order |
| `goodman_spectra` | bool | Include Goodman spectra |

---

## Version Differences

### Version 1.2 vs 1.3

| Feature | Version 1.2 | Version 1.3 |
|---------|-------------|-------------|
| Header identifier | 0x1729 | 0x1729 |
| Config identifier | 0x0827 (buggy) | None |
| Record number | ✓ | ✗ |
| Error code | ✓ | ✗ |
| End time | ✓ | ✗ |
| Data record size | 16 + data | 4 + data |
| Space savings | 0% | ~25% |

**Recommendation:** Use version 1.3 for new files (reduced size, same data).

---

## File Reading Algorithm

### Pseudocode

```python
def read_qfile(filename):
    with open(filename, 'rb') as fp:
        # Check for header
        if check_identifier(fp) == 0x1729:
            header = read_header(fp)
            data_parser = create_data_parser(header)

        # Read data records
        records = []
        while True:
            if check_identifier(fp) == 0x1657:
                record = data_parser.read_record(fp)
                if record is None:
                    break
                records.append(record)
            else:
                break  # Unknown record type or EOF

        return header, records
```

---

## Error Handling

### Common Issues

1. **Truncated files** - EOF while reading record
   - **Detection:** `len(buffer) != expected_size`
   - **Handling:** Stop reading, return partial data

2. **Invalid identifiers** - Wrong magic numbers
   - **Detection:** `identifier != expected`
   - **Handling:** Log warning, skip to next record

3. **Unsupported versions** - Unknown version number
   - **Detection:** `version not in [1.2, 1.3]`
   - **Handling:** Raise NotImplementedError

4. **Missing identifiers** - Sensor ID not in mapping
   - **Detection:** `identifier not in hex_map`
   - **Handling:** Use hex ID as name, log warning

---

## Example: Reading a Q-File

### Python Example

```python
from q2netcdf import QFile

with QFile("data.q") as qf:
    # Read header
    header = qf.header()
    print(f"Version: {header.version}")
    print(f"Channels: {header.Nc}")
    print(f"Time: {header.time}")

    # Read first 10 records
    for i, record in enumerate(qf.data()):
        if i >= 10:
            break
        print(f"Record {i}: time={record.t0}, channels={record.channels}")
```

### C Example (Conceptual)

```c
struct QHeader {
    uint16_t identifier;    // 0x1729
    float version;          // 1.2 or 1.3
    uint64_t timestamp;     // ms since year 0
    uint16_t Nc, Ns, Nf;    // Counts
};

struct QRecord {
    uint16_t identifier;    // 0x1657
    float16_t time;         // Offset in seconds
    float16_t channels[Nc];
    float16_t spectra[Ns][Nf];
};
```

---

## NetCDF Conversion

### Mapping to NetCDF

Q-files are converted to CF-1.8 compliant NetCDF with:

**Dimensions:**
- `time` - Time dimension (unlimited)
- `freq` - Frequency dimension (if spectra present)
- `ftime` - File time (file start time)

**Coordinates:**
- `time(time)` - Datetime64 timestamps
- `freq(freq)` - Frequency array in Hz
- `ftime(ftime)` - File start time

**Variables:**
- Each channel becomes a variable with dimension `(time,)`
- Each spectrum becomes a variable with dimensions `(time, freq)`
- Configuration parameters become variables with dimension `(ftime,)`

**Attributes:**
- `Conventions = "CF-1.8"`
- `long_name` from identifier mapping
- `units` from identifier mapping

---

## References

1. Rockland Scientific Technical Note TN-054: "Q-File Format Specification"
2. Rockland Scientific ISDP User Manual
3. CF Conventions: http://cfconventions.org/
4. IEEE 754-2008 Half-Precision Float

---

## Appendix: Complete Identifier List

See `QHexCodes.py` for the complete mapping of 200+ sensor identifiers.

### Most Common Identifiers

```python
COMMON_IDENTIFIERS = {
    0x130: "P",        # Pressure
    0x150: "T",        # Temperature
    0x170: "C",        # Conductivity
    0x610: "sh",       # Shear probe
    0x640: "dT",       # Temperature gradient
    0x650: "dC",       # Conductivity gradient
    0x920: "shear_raw",     # Raw shear spectrum
    0x930: "shear_gfd",     # Goodman shear spectrum
}
```

---

## Document History

| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2025-10-02 | Initial documentation |

---

**Maintained by:** q2netcdf project
**Contact:** pat@mousebrains.com
**Repository:** https://github.com/mousebrains/q2netcdf
