.. moduleauthor:: Paul Ross <apaulross@gmail.com>
.. sectionauthor:: Paul Ross <apaulross@gmail.com>

.. Description of CPIP command line tools

.. _total_depth.processing_rp66v1_files:


Processing RP66V1 Files
=======================

This describes how you can programatically access the contents of an RP66V1 file with TotalDepth with worked examples.
First, here is how TotalDepth represents the structure of a RP66V1 file.

RP66V1 File Structure
------------------------------

A physical RP66V1 file (typically a file on disk) consists of a :term:`RP66V1.Storage Unit Label` followed of any number of :term:`RP66V1.Explicitly Formatted Logical Record` (EFLRs) and any number of :term:`RP66V1.Indirectly Formatted Logical Record` (IFLRs) in any order.

At this level the file conceptually looks like this:

.. code-block:: console

    Physical File
    |-> Storage Unit Label
    |-> EFLR
    |-> EFLR
    |-> IFLR
    |-> EFLR
    |-> IFLR
    |-> EFLR
        ...


The EFLRs and IFLRs are organised into one or more any number of *Logical Files*.
Specifically a Logical File starts with an ``FILE-HEADER`` EFLR followed by an ``ORIGIN`` or ``WELL-REFERENCE`` EFLR.

A Logical FIle might represent all the data recorded in, say, a "Repeat Section" and Logical Files are independent from each other [#]_.

This conceptual model for the Logical File delimited RP66V1 file is something like this:

.. code-block:: console

    Physical File
    |-> Storage Unit Label
    |-> Logical File[0] "Repeat Section"
        |-> EFLR type FILE-HEADER
        |-> EFLR type ORIGIN
        |-> EFLR
        |-> EFLR
        |-> IFLR
        |-> EFLR
        |-> IFLR
        ...
    |-> Logical File[1] "Main Log"
        |-> EFLR type FILE-HEADER
        |-> EFLR type ORIGIN
        |-> EFLR
        ...

The collection of IFLRs for any Logical File can be further organised into a single *Log Pass*.

Log Pass
^^^^^^^^^^^^^^

A :term:`Log Pass` represents a typical, independent, logging recording. A "Repeat Section" and a "Main Log" are two, distinct, independent Log Passes.
The *Log Pass* is defined by two EFLRs, a ``CHANNEL`` that describes all the available measurements for that Logical File and a ``FRAME`` that describes the sets of these channels that go into a recording, the index and the spacing.
The *Log Pass* represents a single data collection run (example: "Repeat Section") and contains one or more *Frame Arrays*.


Frame Array
^^^^^^^^^^^^^^

A :term:`Frame Array` contains a number of frames of channel data converted to the platforms internal types (usually C doubles) and stored in memory as a table.
In this table each row represents a particular value on the X axis (typically depth or time) and each column is the output of a specific channel. Each row is often referred to as a :term:`Frame`
There can be multiple values for any channel in a frame except for the X axis which has a single value per frame.
A value can be an :term:`Absent Value` indicating that no data was recorded for this channel and frame.


Conceptual Model Presented by TotalDepth
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By example here is a RP66V1 file that contains two Logical Files representing, say, the "Repeat Section" and the "Main Log". Each of these has a number of EFLRs and a Log Pass that contains two Frame Arrays, one is sampled every inch in depth and the other every six inches. The two Frame Arrays might have different channels.

This model of the physical RP66V1 file will be something like:

.. code-block:: console

    Physical File
    |-> Storage Unit Label
    |-> Logical File[0] "Repeat Section"
    |   |-> EFLRs
    |   |-> ...
    |   |-> Log Pass
    |       |-> Frame Array at 1 inch spacing with channels A, B, C
    |       |-> Frame Array at 6 inch spacing with channels B, D, E
    |-> Logical File[1] "Main Log"
        |-> EFLRs
    |   |-> ...
        |-> Log Pass
    |       |-> Frame Array at 1 inch spacing with channels A, B, C, X
    |       |-> Frame Array at 6 inch spacing with channels B, D, E, F


This model is exposed with the class :py:class:`TotalDepth.RP66V1.core.LogicalFile.LogicalIndex`. The LogicalIndex contains a sequence of :py:class:`TotalDepth.RP66V1.core.LogicalFile.LogicalFile` objects that allows random access to all parts of the file.

The following examples show how to iterate through a RP66V1 file with the ``LogicalIndex`` and access the tables and frame data.
The code snippets here are all in :file:`example_data/RP66V1/demo_read.py`

.. warning:: At this version, |version|, these APIs are provisional, not final.


Basic Pattern for Reading RP66V1 Files with TotalDepth
------------------------------------------------------------

All these examples take the following pattern where a ``LogicalIndex`` is created as a context manager.

This can be done with a file path as a string:

.. code-block:: python

    from TotalDepth.RP66V1.core import LogicalFile

    with LogicalFile.LogicalIndex(path) as logical_index:
        # Do something

Or an open, binary, file:

.. code-block:: python

    from TotalDepth.RP66V1.core import LogicalFile

    with open(path, 'rb') as fobj:
        with LogicalFile.LogicalIndex(fobj) as logical_index:
            # Do something

Example Data
^^^^^^^^^^^^^^^^^^^^^^

There are some example RP66V1 files distributed in :file:`example_data/RP66V1/data`, for example:

.. code-block:: python

    from TotalDepth.RP66V1.core import LogicalFile
    
    path = os.path.join('example_data', 'RP66V1', 'data', '206_05a-_3_DWL_DWL_WIRE_258276498.DLIS')
    with LogicalFile.LogicalIndex(path) as logical_index:
        # Do something


There is also some example RP66V1 binary data in the module :py:mod:`tests.unit.RP66V1.core.test_data`, for example:

.. code-block:: python

    from TotalDepth.RP66V1.core import LogicalFile
    from tests.unit.RP66V1.core import test_data

    file_object = io.BytesIO(test_data.BASIC_FILE)
    with LogicalFile.LogicalIndex(file_object) as logical_index:
        # Do something


Inspecting the Logical File
------------------------------

Once a :py:class:`TotalDepth.RP66V1.core.LogicalFile.LogicalIndex` has been created the
:py:class:`TotalDepth.RP66V1.core.LogicalFile.LogicalFile` object can be accessed, for example:

.. code-block:: python

    from TotalDepth.RP66V1.core import LogicalFile
    from tests.unit.RP66V1.core import test_data

    file_object = io.BytesIO(test_data.BASIC_FILE)
    with LogicalFile.LogicalIndex(file_object) as logical_index:
        for l, logical_file in enumerate(logical_index.logical_files):
            print(f'LogicalFile [{l}]: {logical_file}')


Produces the single Logical File in ``BASIC_FILE``:

.. code-block:: console

    LogicalFile [0]: <TotalDepth.RP66V1.core.LogicalFile.LogicalFile object at 0x11ca5ec50>

The Logical File object has at least these properties:

=================================== ======================================================
Property                            Description
=================================== ======================================================
``file_header_logical_record``      The ``FILE-HEADER`` EFLR that defines the Logical
                                    File.
                                    
                                    See: [RP66V1 Section 5.1 File Header Logical Record
                                    (FHLR)].
``origin_logical_record``           The ``ORIGIN`` EFLR that defines the origin of the
                                    Logical Record.
                                    
                                    See [RP66V1 Section 5.2 Origin Logical Record (OLR)].
``defining_origin``                 Returns the Defining Origin of the Logical File.
                                    This is the first row of the ``ORIGIN`` Logical
                                    Record.
                                    
                                    From [RP66V1 Section 5.2.1 Origin Objects]:
                                    *"The first Object in the first ORIGIN Set is the
                                    Defining Origin for the Logical File in which it is
                                    contained, and the corresponding Logical File is
                                    called the Origin's Parent File.
                                    It is intended that no two Logical Files will ever
                                    have Defining Origins with all Attribute Values
                                    identical.""*
``channel``                         The CHANNEL EFLR or None.
``frame``                           The FRAME EFLR or None.
``has_log_pass``                    True if this Logical File has both a CHANNEL and a
                                    FRAME EFLR.
=================================== ======================================================

Notes:

* TotalDepth implements an EFLR as the class:
    :py:class:`TotalDepth.RP66V1.core.LogicalRecord.EFLR.ExplicitlyFormattedLogicalRecord`
* That class has a method, used below, that provides verbose information about the table:
    :py:meth:`TotalDepth.RP66V1.core.LogicalRecord.EFLR.ExplicitlyFormattedLogicalRecord.str_long()` 

Here is an example of accessing all of the above properties for the ``BASIC_FILE``:

.. code-block:: python
    :emphasize-lines: 7-26

    from TotalDepth.RP66V1.core import LogicalFile
    from tests.unit.RP66V1.core import test_data

    file_object = io.BytesIO(test_data.BASIC_FILE)
    with LogicalFile.LogicalIndex(file_object) as logical_index:
        for l, logical_file in enumerate(logical_index.logical_files):
            print(f'***** logical_file.file_header_logical_record.str_long():')
            print(logical_file.file_header_logical_record.str_long())
            print()
            print(f'***** logical_file.origin_logical_record.str_long():')
            print(logical_file.origin_logical_record.str_long())
            print()
            print(f'***** logical_file.defining_origin:')
            print(logical_file.defining_origin)
            print()
            if logical_file.channel is not None:
                print(f'***** logical_file.channel.str_long():')
                print(logical_file.channel.str_long())
                print()
            if logical_file.frame is not None:
                print(f'***** logical_file.frame.str_long():')
                print(logical_file.frame.str_long())
                print()
            print(f'***** logical_file.has_log_pass:')
            print(logical_file.has_log_pass)
            print()

Gives:

.. code-block:: console

    ***** logical_file.file_header_logical_record.str_long():
    <ExplicitlyFormattedLogicalRecord EFLR Set type: b'FILE-HEADER' name: b''>
      Template [2]:
        CD: 001 10100 L: b'SEQUENCE-NUMBER' C: 1 R: 20 (ASCII) U: b'' V: None
        CD: 001 10100 L: b'ID' C: 1 R: 20 (ASCII) U: b'' V: None
      Objects [1]:
        OBNAME: O: 2 C: 0 I: b'1'
          CD: 001 00001 L: b'SEQUENCE-NUMBER' C: 1 R: 20 (ASCII) U: b'' V: [b'0000000001']
          CD: 001 00001 L: b'ID' C: 1 R: 20 (ASCII) U: b'' V: [b'HES INSITE.1                                                     ']

    ***** logical_file.origin_logical_record.str_long():
    <ExplicitlyFormattedLogicalRecord EFLR Set type: b'ORIGIN' name: b''>
      Template [20]:
        CD: 001 11000 L: b'FILE-ID' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'FILE-SET-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'FILE-SET-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'FILE-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'FILE-TYPE' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'PRODUCT' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'VERSION' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'PROGRAMS' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'CREATION-TIME' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'ORDER-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'DESCENT-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'RUN-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'WELL-ID' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'WELL-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'FIELD-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'PRODUCER-CODE' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'PRODUCER-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'COMPANY' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'NAME-SPACE-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11000 L: b'NAME-SPACE-VERSION' C: 0 R: 19 (IDENT) U: b'' V: None
      Objects [1]:
        OBNAME: O: 2 C: 0 I: b'0'
          CD: 001 01101 L: b'FILE-ID' C: 1 R: 20 (ASCII) U: b'' V: [b'HES INSITE.1']
          CD: 001 01101 L: b'FILE-SET-NAME' C: 1 R: 19 (IDENT) U: b'' V: [b'BURU ENERGY LIMITED/VALHALLA NORTH 1']
          CD: 001 01101 L: b'FILE-SET-NUMBER' C: 1 R: 18 (UVARI) U: b'' V: [257346645]
          CD: 001 01101 L: b'FILE-NUMBER' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01101 L: b'FILE-TYPE' C: 1 R: 19 (IDENT) U: b'' V: [b'PLAYBACK']
          CD: 001 01101 L: b'PRODUCT' C: 1 R: 20 (ASCII) U: b'' V: [b'HES INSITE']
          CD: 001 01101 L: b'VERSION' C: 1 R: 20 (ASCII) U: b'' V: [b'R5.1.4']
          CD: 000 00000 L: b'PROGRAMS' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01101 L: b'CREATION-TIME' C: 1 R: 21 (DTIME) U: b'' V: [<<class 'TotalDepth.RP66V1.core.RepCode.DateTime'> 2012-03-07 10:00:49.000 STD>]
          CD: 001 01101 L: b'ORDER-NUMBER' C: 1 R: 20 (ASCII) U: b'' V: [b'9262611']
          CD: 000 00000 L: b'DESCENT-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 000 00000 L: b'RUN-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01101 L: b'WELL-ID' C: 1 R: 20 (ASCII) U: b'' V: [b'N/A']
          CD: 001 01101 L: b'WELL-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'VALHALLA NORTH 1']
          CD: 001 01101 L: b'FIELD-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'VALHALLA']
          CD: 001 01101 L: b'PRODUCER-CODE' C: 1 R: 16 (UNORM) U: b'' V: [280]
          CD: 001 01101 L: b'PRODUCER-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'Halliburton']
          CD: 001 01101 L: b'COMPANY' C: 1 R: 20 (ASCII) U: b'' V: [b'BURU ENERGY LIMITED']
          CD: 000 00000 L: b'NAME-SPACE-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 000 00000 L: b'NAME-SPACE-VERSION' C: 0 R: 19 (IDENT) U: b'' V: None

    ***** logical_file.defining_origin:
    OBNAME: O: 2 C: 0 I: b'0'
      CD: 001 01101 L: b'FILE-ID' C: 1 R: 20 (ASCII) U: b'' V: [b'HES INSITE.1']
      CD: 001 01101 L: b'FILE-SET-NAME' C: 1 R: 19 (IDENT) U: b'' V: [b'BURU ENERGY LIMITED/VALHALLA NORTH 1']
      CD: 001 01101 L: b'FILE-SET-NUMBER' C: 1 R: 18 (UVARI) U: b'' V: [257346645]
      CD: 001 01101 L: b'FILE-NUMBER' C: 1 R: 18 (UVARI) U: b'' V: [1]
      CD: 001 01101 L: b'FILE-TYPE' C: 1 R: 19 (IDENT) U: b'' V: [b'PLAYBACK']
      CD: 001 01101 L: b'PRODUCT' C: 1 R: 20 (ASCII) U: b'' V: [b'HES INSITE']
      CD: 001 01101 L: b'VERSION' C: 1 R: 20 (ASCII) U: b'' V: [b'R5.1.4']
      CD: 000 00000 L: b'PROGRAMS' C: 0 R: 19 (IDENT) U: b'' V: None
      CD: 001 01101 L: b'CREATION-TIME' C: 1 R: 21 (DTIME) U: b'' V: [<<class 'TotalDepth.RP66V1.core.RepCode.DateTime'> 2012-03-07 10:00:49.000 STD>]
      CD: 001 01101 L: b'ORDER-NUMBER' C: 1 R: 20 (ASCII) U: b'' V: [b'9262611']
      CD: 000 00000 L: b'DESCENT-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
      CD: 000 00000 L: b'RUN-NUMBER' C: 0 R: 19 (IDENT) U: b'' V: None
      CD: 001 01101 L: b'WELL-ID' C: 1 R: 20 (ASCII) U: b'' V: [b'N/A']
      CD: 001 01101 L: b'WELL-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'VALHALLA NORTH 1']
      CD: 001 01101 L: b'FIELD-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'VALHALLA']
      CD: 001 01101 L: b'PRODUCER-CODE' C: 1 R: 16 (UNORM) U: b'' V: [280]
      CD: 001 01101 L: b'PRODUCER-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'Halliburton']
      CD: 001 01101 L: b'COMPANY' C: 1 R: 20 (ASCII) U: b'' V: [b'BURU ENERGY LIMITED']
      CD: 000 00000 L: b'NAME-SPACE-NAME' C: 0 R: 19 (IDENT) U: b'' V: None
      CD: 000 00000 L: b'NAME-SPACE-VERSION' C: 0 R: 19 (IDENT) U: b'' V: None

    ***** logical_file.channel.str_long():
    <ExplicitlyFormattedLogicalRecord EFLR Set type: b'CHANNEL' name: b''>
      Template [8]:
        CD: 001 11100 L: b'LONG-NAME' C: 0 R: 20 (ASCII) U: b'' V: None
        CD: 001 11100 L: b'PROPERTIES' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11100 L: b'REPRESENTATION-CODE' C: 0 R: 15 (USHORT) U: b'' V: None
        CD: 001 11100 L: b'DIMENSION' C: 0 R: 18 (UVARI) U: b'' V: None
        CD: 001 11100 L: b'ELEMENT-LIMIT' C: 0 R: 18 (UVARI) U: b'' V: None
        CD: 001 11100 L: b'UNITS' C: 0 R: 27 (UNITS) U: b'' V: None
        CD: 001 11100 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
        CD: 001 11100 L: b'SOURCE' C: 0 R: 24 (OBJREF) U: b'' V: None
      Objects [5]:
        OBNAME: O: 2 C: 0 I: b'DEPT'
          CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'DEPT/Depth']
          CD: 000 00000 L: b'PROPERTIES' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01001 L: b'REPRESENTATION-CODE' C: 1 R: 15 (USHORT) U: b'' V: [7]
          CD: 001 01001 L: b'DIMENSION' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'ELEMENT-LIMIT' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'UNITS' C: 1 R: 27 (UNITS) U: b'' V: [b'm']
          CD: 000 00000 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
          CD: 000 00000 L: b'SOURCE' C: 0 R: 24 (OBJREF) U: b'' V: None
        OBNAME: O: 2 C: 0 I: b'TENS'
          CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'TENS/Tension']
          CD: 000 00000 L: b'PROPERTIES' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01001 L: b'REPRESENTATION-CODE' C: 1 R: 15 (USHORT) U: b'' V: [2]
          CD: 001 01001 L: b'DIMENSION' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'ELEMENT-LIMIT' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'UNITS' C: 1 R: 27 (UNITS) U: b'' V: [b'lbs']
          CD: 000 00000 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
          CD: 001 01001 L: b'SOURCE' C: 1 R: 24 (OBJREF) U: b'' V: [ObjectReference(T=b'TOOL', N=ObjectName(O=2, C=0, I=b'DEP'))]
        OBNAME: O: 2 C: 0 I: b'ETIM'
          CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'ETIM/Elapsed Time']
          CD: 000 00000 L: b'PROPERTIES' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01001 L: b'REPRESENTATION-CODE' C: 1 R: 15 (USHORT) U: b'' V: [7]
          CD: 001 01001 L: b'DIMENSION' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'ELEMENT-LIMIT' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'UNITS' C: 1 R: 27 (UNITS) U: b'' V: [b'min']
          CD: 000 00000 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
          CD: 001 01001 L: b'SOURCE' C: 1 R: 24 (OBJREF) U: b'' V: [ObjectReference(T=b'TOOL', N=ObjectName(O=2, C=0, I=b'DEP'))]
        OBNAME: O: 2 C: 0 I: b'DHTN'
          CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'DHTN/CH Tension']
          CD: 000 00000 L: b'PROPERTIES' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01001 L: b'REPRESENTATION-CODE' C: 1 R: 15 (USHORT) U: b'' V: [2]
          CD: 001 01001 L: b'DIMENSION' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'ELEMENT-LIMIT' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'UNITS' C: 1 R: 27 (UNITS) U: b'' V: [b'lbs']
          CD: 000 00000 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
          CD: 001 01001 L: b'SOURCE' C: 1 R: 24 (OBJREF) U: b'' V: [ObjectReference(T=b'TOOL', N=ObjectName(O=2, C=0, I=b'RWCH'))]
        OBNAME: O: 2 C: 0 I: b'GR'
          CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'GR/Gamma API']
          CD: 000 00000 L: b'PROPERTIES' C: 0 R: 19 (IDENT) U: b'' V: None
          CD: 001 01001 L: b'REPRESENTATION-CODE' C: 1 R: 15 (USHORT) U: b'' V: [2]
          CD: 001 01001 L: b'DIMENSION' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'ELEMENT-LIMIT' C: 1 R: 18 (UVARI) U: b'' V: [1]
          CD: 001 01001 L: b'UNITS' C: 1 R: 27 (UNITS) U: b'' V: [b'api']
          CD: 000 00000 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
          CD: 001 01001 L: b'SOURCE' C: 1 R: 24 (OBJREF) U: b'' V: [ObjectReference(T=b'TOOL', N=ObjectName(O=2, C=0, I=b'D4TG'))]

    ***** logical_file.frame.str_long():
    <ExplicitlyFormattedLogicalRecord EFLR Set type: b'FRAME' name: b''>
      Template [8]:
        CD: 001 11100 L: b'DESCRIPTION' C: 0 R: 20 (ASCII) U: b'' V: None
        CD: 001 11100 L: b'CHANNELS' C: 0 R: 23 (OBNAME) U: b'' V: None
        CD: 001 11100 L: b'INDEX-TYPE' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11100 L: b'DIRECTION' C: 0 R: 19 (IDENT) U: b'' V: None
        CD: 001 11100 L: b'SPACING' C: 0 R: 7 (FDOUBL) U: b'' V: None
        CD: 001 11100 L: b'ENCRYPTED' C: 0 R: 15 (USHORT) U: b'' V: None
        CD: 001 11100 L: b'INDEX-MIN' C: 0 R: 7 (FDOUBL) U: b'' V: None
        CD: 001 11100 L: b'INDEX-MAX' C: 0 R: 7 (FDOUBL) U: b'' V: None
      Objects [1]:
        OBNAME: O: 2 C: 0 I: b'50'
          CD: 000 00000 L: b'DESCRIPTION' C: 0 R: 20 (ASCII) U: b'' V: None
          CD: 001 01001 L: b'CHANNELS' C: 5 R: 23 (OBNAME) U: b'' V: [ObjectName(O=2, C=0, I=b'DEPT'), ObjectName(O=2, C=0, I=b'TENS'), ObjectName(O=2, C=0, I=b'ETIM'), ObjectName(O=2, C=0, I=b'DHTN'), ObjectName(O=2, C=0, I=b'GR')]
          CD: 001 01001 L: b'INDEX-TYPE' C: 1 R: 19 (IDENT) U: b'' V: [b'BOREHOLE-DEPTH']
          CD: 001 01001 L: b'DIRECTION' C: 1 R: 19 (IDENT) U: b'' V: [b'INCREASING']
          CD: 001 01111 L: b'SPACING' C: 1 R: 7 (FDOUBL) U: b'm' V: [0.1]
          CD: 000 00000 L: b'ENCRYPTED' C: 0 R: 15 (USHORT) U: b'' V: None
          CD: 000 00000 L: b'INDEX-MIN' C: 0 R: 7 (FDOUBL) U: b'' V: None
          CD: 000 00000 L: b'INDEX-MAX' C: 0 R: 7 (FDOUBL) U: b'' V: None

    ***** logical_file.has_log_pass:
    True


More about :term:`RP66V1.EFLR` Tables
------------------------------------------

An :term:`RP66V1.Explicitly Formatted Logical Record` is a table of data organised in rows and columns.

.. code-block:: python

    Table
        Row
            Value
            Value
            ...
        Row
            Value
            Value
            ...
        ...
        
This is implemented by TotalDepth as:

* Table: :py:class:`TotalDepth.RP66V1.core.LogicalRecord.EFLR.ExplicitlyFormattedLogicalRecord`
* Row is an ``Object``: :py:class:`TotalDepth.RP66V1.core.LogicalRecord.EFLR.Object`
* Value is an ``Attribute``: :py:class:`TotalDepth.RP66V1.core.LogicalRecord.EFLR.Attribute`


Reading :term:`EFLR` Contents
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Each value in a row/column is known as an :term:`RP66V1.Attribute`

This is implemented by
:py:class:`TotalDepth.RP66V1.core.LogicalRecord.EFLR.Attribute`
which has the following properties:

============ ========= =======================================================================
Property     Type      Description
============ ========= =======================================================================
``label``    ``bytes`` The label identifying the Attribute.
``count``    ``int``   The number of the values the Attribute has.
``rep_code`` ``int``   The Representation Code of the values of the Attribute.
``units``    ``bytes`` The units of the value.
``value``    ``list``  The value itself as a list of instances of the Representation Code.
============ ========= =======================================================================


These Attributes are iterable, for example the following code accesses the contents of every ``PARAMETER`` EFLR:

.. code-block:: python
    :emphasize-lines: 7-16

    from TotalDepth.RP66V1.core import LogicalFile
    from tests.unit.RP66V1.core import test_data

    file_object = io.BytesIO(test_data.BASIC_FILE)
    with LogicalFile.LogicalIndex(file_object) as logical_index:
        for logical_file in logical_index.logical_files:
            for position, eflr in logical_file.eflrs:
                # eflr is a TotalDepth.RP66V1.core.LogicalRecord.EFLR.ExplicitlyFormattedLogicalRecord
                if eflr.set.type == b'PARAMETER':
                    print(eflr)
                    for row in eflr.objects:
                        # row is a TotalDepth.RP66V1.core.LogicalRecord.EFLR.Object
                        print(f'    Row: {row.name.I}')
                        for attr in row.attrs:
                            # attr is a TotalDepth.RP66V1.core.LogicalRecord.EFLR.Attribute
                            print(f'        Attr: {attr.label} = {attr.value} ({attr.units})')

Will produce something like this (output truncated):

.. code-block:: console

    <ExplicitlyFormattedLogicalRecord EFLR Set type: b'PARAMETER' name: b''>
        Row: b'LOC'
            Attr: b'LONG-NAME' = [b'LOCATION'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b"LATITUDE: 18DEG 01' 32.8'' S"] (b'')
        Row: b'SVCO'
            Attr: b'LONG-NAME' = [b'SERVICECONAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'Halliburton'] (b'')
        Row: b'IQVR'
            Attr: b'LONG-NAME' = [b'WLIQ VERSION'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'R3.2.0'] (b'')
        Row: b'STAT'
            Attr: b'LONG-NAME' = [b'STATE NAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'WA'] (b'')
        Row: b'COUN'
            Attr: b'LONG-NAME' = [b'COUNTRY NAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'AUSTRALIA'] (b'')
        Row: b'SON'
            Attr: b'LONG-NAME' = [b'JOB NUMBER'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'9262611'] (b'')
        Row: b'SECT'
            Attr: b'LONG-NAME' = [b'SECTION'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'N/A'] (b'')
        Row: b'TOWN'
            Attr: b'LONG-NAME' = [b'TOWNSHIP'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'N/A'] (b'')
        Row: b'RANG'
            Attr: b'LONG-NAME' = [b'RANGE'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'N/A'] (b'')
        Row: b'APIN'
            Attr: b'LONG-NAME' = [b'API S/N'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'N/A'] (b'')
        Row: b'CN'
            Attr: b'LONG-NAME' = [b'CUSTOMER NAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'BURU ENERGY LIMITED'] (b'')
        Row: b'WN'
            Attr: b'LONG-NAME' = [b'WELL NAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'VALHALLA NORTH 1'] (b'')
        Row: b'FN'
            Attr: b'LONG-NAME' = [b'FIELD NAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'VALHALLA'] (b'')
        Row: b'RIG'
            Attr: b'LONG-NAME' = [b'RIG NAME'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'ENSIGN RIG #32'] (b'')
        Row: b'PDAT'
            Attr: b'LONG-NAME' = [b'PERMANENT DATUM'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'MSL'] (b'')
        Row: b'LMF'
            Attr: b'LONG-NAME' = [b'LOG MEAS FROM'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'RT'] (b'')
        Row: b'DMF'
            Attr: b'LONG-NAME' = [b'DRILL MEAS FROM'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'RT'] (b'')
        Row: b'FL1'
            Attr: b'LONG-NAME' = [b'LOCATIONLINE1'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b"LATITUDE: 18DEG 01' 32.8'' S"] (b'')
        Row: b'FL2'
            Attr: b'LONG-NAME' = [b'LOCATIONLINE2'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b"LONGITUDE: 124DEG 43' 47.1'' E"] (b'')
        Row: b'FL3'
            Attr: b'LONG-NAME' = [b'LOCATIONLINE3'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'EASTING: 683112'] (b'')
        Row: b'FL4'
            Attr: b'LONG-NAME' = [b'LOCATIONLINE4'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'NORTHING: 8006107'] (b'')
        Row: b'FL5'
            Attr: b'LONG-NAME' = [b'LOCATIONLINE5'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'GDA ZONE 51'] (b'')
        Row: b'DATE'
            Attr: b'LONG-NAME' = [b'DATE'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'06-Mar-2012'] (b'')
        Row: b'LCC'
            Attr: b'LONG-NAME' = [b'PRODUCER-CODE'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [b'280'] (b'')
        Row: b'EDF'
            Attr: b'LONG-NAME' = [b'DF ELEV'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [114.9000015258789] (b'm')
        Row: b'EPD'
            Attr: b'LONG-NAME' = [b'ELEVATION'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [0.0] (b'm')
        Row: b'EGL'
            Attr: b'LONG-NAME' = [b'GL ELEV'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [109.0] (b'm')
        Row: b'GVFD'
            Attr: b'LONG-NAME' = [b'GRAVITY FIELD'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [1.0] (b'g')
        Row: b'EKB'
            Attr: b'LONG-NAME' = [b'KB ELEV'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [114.9000015258789] (b'm')
        Row: b'TVDS'
            Attr: b'LONG-NAME' = [b'TVDSS CORRECTN'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [5.90000057220459] (b'm')
        Row: b'APD'
            Attr: b'LONG-NAME' = [b'DEPTH ABOVE PD'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [5.90000057220459] (b'm')
        Row: b'DDEV'
            Attr: b'LONG-NAME' = [b'MAX INC'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [1.8200000524520874] (b'deg')
        Row: b'DDEG'
            Attr: b'LONG-NAME' = [b'MAX INC DEPTH'] (b'')
            Attr: b'DIMENSION' = [1] (b'')
            Attr: b'AXIS' = None (b'')
            Attr: b'ZONES' = None (b'')
            Attr: b'VALUES' = [2225.169921875] (b'm')
        ...

Or the Attributes can be extracted by identity or integer index, for example:

.. code-block:: python
    :emphasize-lines: 4-8

    file_object = io.BytesIO(test_data.BASIC_FILE)
    with LogicalFile.LogicalIndex(file_object) as logical_index:
        for logical_file in logical_index.logical_files:
            for position, eflr in logical_file.eflrs:
                if eflr.set.type == b'PARAMETER':
                    print(eflr[0])
                    print()
                    print(eflr[0][0])

Gives:

.. code-block:: console

    OBNAME: O: 2 C: 0 I: b'LOC'
      CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'LOCATION']
      CD: 001 01001 L: b'DIMENSION' C: 1 R: 18 (UVARI) U: b'' V: [1]
      CD: 000 00000 L: b'AXIS' C: 0 R: 23 (OBNAME) U: b'' V: None
      CD: 000 00000 L: b'ZONES' C: 0 R: 23 (OBNAME) U: b'' V: None
      CD: 001 01101 L: b'VALUES' C: 1 R: 20 (ASCII) U: b'' V: [b"LATITUDE: 18DEG 01' 32.8'' S"]

    CD: 001 01101 L: b'LONG-NAME' C: 1 R: 20 (ASCII) U: b'' V: [b'LOCATION']


.. _total_depth.processing_rp66v1_files.reading_frames_as_numpy:

Reading the Frame Data and Accessing the ``numpy`` Arrays
---------------------------------------------------------------------

Here is an example of accessing the numpy arrays and using ``np.describe()`` to describe each array:

.. code-block:: python
    :emphasize-lines: 7-19

    import numpy as np

    from TotalDepth.RP66V1.core import LogicalFile

    with LogicalFile.LogicalIndex(path_in) as logical_index:
        for logical_file in logical_index.logical_files:
            if logical_file.has_log_pass:
                for frame_array in logical_file.log_pass:
                    print(frame_array)
                    frame_count = logical_file.populate_frame_array(frame_array)
                    print(
                        f'Loaded {frame_count} frames and {len(frame_array)} channels'
                        f' from {frame_array.ident} using {frame_array.sizeof_array} bytes.'
                    )
                    for channel in frame_array.channels:
                        print(channel)
                        # channel.array is a numpy array
                        np.info(channel.array)
                        print()

The output will be:

.. code-block:: console

    Loaded 921 frames and 4 channels from OBNAME: O: 2 C: 0 I: b'2000T' using 14736 bytes.
    
    FrameChannel: OBNAME: O: 2 C: 4 I: b'TIME'            Rc:   2 Co:    1 Un: b'ms'        Di: [1] b'1 second River Time'
    class:  ndarray
    shape:  (921, 1)
    strides:  (4, 4)
    itemsize:  4
    aligned:  True
    contiguous:  True
    fortran:  True
    data pointer: 0x7faa710b6000
    byteorder:  little
    byteswap:  False
    type: float32
    
    FrameChannel: OBNAME: O: 2 C: 4 I: b'TDEP'            Rc:   2 Co:    1 Un: b'0.1 in'    Di: [1] b'1 second River Depth'
    class:  ndarray
    shape:  (921, 1)
    strides:  (4, 4)
    itemsize:  4
    aligned:  True
    contiguous:  True
    fortran:  True
    data pointer: 0x7faa710b7000
    byteorder:  little
    byteswap:  False
    type: float32
    
    FrameChannel: OBNAME: O: 2 C: 0 I: b'TENS_SL'         Rc:   2 Co:    1 Un: b'lbf'       Di: [1] b'Cable Tension'
    class:  ndarray
    shape:  (921, 1)
    strides:  (4, 4)
    itemsize:  4
    aligned:  True
    contiguous:  True
    fortran:  True
    data pointer: 0x7fae6c8c2600
    byteorder:  little
    byteswap:  False
    type: float32
    
    FrameChannel: OBNAME: O: 2 C: 0 I: b'DEPT_SL'         Rc:   2 Co:    1 Un: b'0.1 in'    Di: [1] b'Station logging depth'
    class:  ndarray
    shape:  (921, 1)
    strides:  (4, 4)
    itemsize:  4
    aligned:  True
    contiguous:  True
    fortran:  True
    data pointer: 0x7fae6c8c3600
    byteorder:  little
    byteswap:  False
    type: float32
    ...


Making Calculations on the ``numpy`` Data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Very similar to the above we can make some calculations using standard ``numpy`` calls:

.. code-block:: python
    :emphasize-lines: 7-18

    import numpy as np

    from TotalDepth.RP66V1.core import LogicalFile

    with LogicalFile.LogicalIndex(path_in) as logical_index:
        for logical_file in logical_index.logical_files:
            if logical_file.has_log_pass:
                for frame_array in logical_file.log_pass:
                    print(frame_array)
                    frame_count = logical_file.populate_frame_array(frame_array)
                    print(
                        f'Loaded {frame_count} frames and {len(frame_array)} channels'
                        f' from {frame_array.ident} using {frame_array.sizeof_array} bytes.'
                    )
                    for channel in frame_array.channels:
                        print(channel.ident, channel.long_name, channel.units)
                        # channel.array is a numpy array
                        print(f'Min: {channel.array.min():12.3f} Max: {channel.array.max():12.3f}')


Would give this output:

.. code-block:: console

    FrameArray: ID: OBNAME: O: 2 C: 0 I: b'2000T' b''
      FrameChannel: OBNAME: O: 2 C: 4 I: b'TIME'            Rc:   2 Co:    1 Un: b'ms'        Di: [1] b'1 second River Time'
      FrameChannel: OBNAME: O: 2 C: 4 I: b'TDEP'            Rc:   2 Co:    1 Un: b'0.1 in'    Di: [1] b'1 second River Depth'
      FrameChannel: OBNAME: O: 2 C: 0 I: b'TENS_SL'         Rc:   2 Co:    1 Un: b'lbf'       Di: [1] b'Cable Tension'
      FrameChannel: OBNAME: O: 2 C: 0 I: b'DEPT_SL'         Rc:   2 Co:    1 Un: b'0.1 in'    Di: [1] b'Station logging depth'
    Loaded 921 frames and 4 channels from OBNAME: O: 2 C: 0 I: b'2000T' using 14736 bytes.
    OBNAME: O: 2 C: 4 I: b'TIME' b'1 second River Time' b'ms'
    Min: 16677259.000 Max: 17597260.000
    OBNAME: O: 2 C: 4 I: b'TDEP' b'1 second River Depth' b'0.1 in'
    Min:   852606.000 Max:   893302.000
    OBNAME: O: 2 C: 0 I: b'TENS_SL' b'Cable Tension' b'lbf'
    Min:     1825.000 Max:     2594.000
    OBNAME: O: 2 C: 0 I: b'DEPT_SL' b'Station logging depth' b'0.1 in'
    Min:   852606.000 Max:   893303.000
    
    FrameArray: ID: OBNAME: O: 2 C: 0 I: b'800T' b''
      FrameChannel: OBNAME: O: 2 C: 5 I: b'TIME'            Rc:   2 Co:    1 Un: b'ms'        Di: [1] b'400 milli-second time channel'
      FrameChannel: OBNAME: O: 2 C: 5 I: b'TDEP'            Rc:   2 Co:    1 Un: b'0.1 in'    Di: [1] b'MSCT depth channel'
      FrameChannel: OBNAME: O: 2 C: 1 I: b'ETIM'            Rc:   2 Co:    1 Un: b's'         Di: [1] b'Elapsed Logging Time'
      ... Lots more omitted
      FrameChannel: OBNAME: O: 2 C: 0 I: b'CMLP'            Rc:   2 Co:    1 Un: b'in'        Di: [1] b'Coring Motor Linear Position'
    Loaded 2301 frames and 43 channels from OBNAME: O: 2 C: 0 I: b'800T' using 395772 bytes.
    OBNAME: O: 2 C: 5 I: b'TIME' b'400 milli-second time channel' b'ms'
    Min: 16677259.000 Max: 17597260.000
    OBNAME: O: 2 C: 5 I: b'TDEP' b'MSCT depth channel' b'0.1 in'
    Min:   852606.000 Max:   893304.000
    OBNAME: O: 2 C: 1 I: b'ETIM' b'Elapsed Logging Time' b's'
    Min:        0.000 Max:      920.001
    ... Lots more omitted
    OBNAME: O: 2 C: 0 I: b'CMLP' b'Coring Motor Linear Position' b'in'
    Min:       -0.927 Max:        2.891


Limiting the Amount of Data Read
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The RP66V1 Frame Array can be very large so to make it more manageable the :py:meth:`TotalDepth.RP66V1.core.LogicalFile.LogicalIndex.populate_frame_array` can take the following, optional, arguments:

* `channels`: A sequence of channel identifiers. Only these channels will be populated into the numpy arrays in the Frame Array. The other channels will have a zero length numpy array. Channel 0, the X axis, will always be populated.
* `frame_slice` to reduce the number of frames that are populated. You can use either of these classes:

    * :py:class:`TotalDepth.common.Slice.Slice` which takes optional start, stop, step values that default to ``(0, len(data), 1)``. For example if there are 128 frames available then ``Slice(64, None, 2)`` would populate every other frame from frame 64 to the end.
    * :py:class:`TotalDepth.common.Slice.Split` which takes single integer, this is maximum number of frames to be populated and they will be evenly spaced throughout the Frame Array. For example if there are 128 available frames that ``Split(8)`` would populate each numpy array with every 16th frame producing 8 frames.

For example, adding the two highlighted lines which populates every 64th frame and channels 1 and 2:

.. code-block:: python
    :emphasize-lines: 2,10-11
    
    from TotalDepth.RP66V1.core import LogicalFile
    from TotalDepth.common import Slice

    with LogicalFile.LogicalIndex(path_in) as logical_index:
        for logical_file in logical_index.logical_files:
            if logical_file.has_log_pass:
                for frame_array in logical_file.log_pass:
                    frame_count = logical_file.populate_frame_array(
                        frame_array,
                        frame_slice=Slice.Slice(0, None, 64),
                        channels={frame_array.channels[1].ident, frame_array.channels[2].ident}
                    )
                    print(
                        f'Loaded {frame_count} frames'
                        f' from {frame_array.ident} using {frame_array.sizeof_array} bytes.'
                    )
                    for channel in frame_array.channels:
                        if len(channel.array):
                            print(channel.ident, channel.long_name, channel.units)
                            print(f'Min: {channel.array.min():12.3f} Max: {channel.array.max():12.3f}')
                    print()

Gives:

.. code-block:: console

    Loaded 15 frames from OBNAME: O: 2 C: 0 I: b'2000T' using 180 bytes.
    OBNAME: O: 2 C: 4 I: b'TIME' b'1 second River Time' b'ms'
    Min: 16677259.000 Max: 17573260.000
    OBNAME: O: 2 C: 4 I: b'TDEP' b'1 second River Depth' b'0.1 in'
    Min:   852606.000 Max:   892658.062
    OBNAME: O: 2 C: 0 I: b'TENS_SL' b'Cable Tension' b'lbf'
    Min:     1877.000 Max:     2561.000
    
    Loaded 36 frames from OBNAME: O: 2 C: 0 I: b'800T' using 432 bytes.
    OBNAME: O: 2 C: 5 I: b'TIME' b'400 milli-second time channel' b'ms'
    Min: 16677259.000 Max: 17573260.000
    OBNAME: O: 2 C: 5 I: b'TDEP' b'MSCT depth channel' b'0.1 in'
    Min:   852606.000 Max:   893135.188
    OBNAME: O: 2 C: 1 I: b'ETIM' b'Elapsed Logging Time' b's'
    Min:        0.000 Max:      896.001


.. rubric:: Footnotes

.. [#] RP66V1 provides a method of collecting together physical files by using the :term:`RP66V1.Storage Unit Label` where the fields :term:`RP66V1.Storage Set Identifier` and :term:`RP66V1.Storage Unit Sequence Number` provide a means of linking physical files. In practice this has not been seen but if this is your use case then the class :py:class:`TotalDepth.RP66V1.core.pFile.FileRead` can minimally read the Storage Unit Label and its fields.

