Formatted I/O — cis_interface 0.3.0 documentation

# Formatted I/O

In addition to passing raw strings, the cis_interface framework also has support for formatting/processing messages from/to language native objects.

## Scalars

The format of messages containing scalar variables (strings, integers, and floating point numbers) can be specified using a C-style format string (See C-Style Format Strings for details). In the example below, the format string `"%6s\t%d\t%f\n"` indicates that each message will contain a 6 character string, an integer, and a float. In addition to providing a format string, the C API requires the use of different functions for initializing channels and sending/receiving to/from them.

Model Code:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import CisInput, CisOutput

    # Initialize input/output channels
    in_channel = CisInput('inputA', '%6s\t%d\t%f\n')
    out_channel = CisOutput('outputA', '%6s\t%d\t%f\n')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, msg = in_channel.recv()
        if not flag:
            print("Model A: No more input.")
            break
        name, count, size = msg

        # Print received message
        print('Model A: %s, %d, %f' % (name, count, size))

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(name, count, size)
        if not flag:
            print("Model A: Error sending output.")
            break

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import CisInput, CisOutput

    # Initialize input/output channels
    in_channel = CisInput('inputB', '%6s\t%d\t%f\n')
    out_channel = CisOutput('outputB', '%6s\t%d\t%f\n')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, msg = in_channel.recv()
        if not flag:
            print("Model B: No more input.")
            break
        name, count, size = msg

        # Print received message
        print('Model B: %s, %d, %f' % (name, count, size))

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(name, count, size)
        if not flag:
            print("Model B: Error sending output.")
            break

(Example in other languages)

The same YAML can be used as was used for the example from Getting started with the modification that the files are now read/written line-by-line (`filetype: ascii`) rather than all at once:

Model YAML:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    models:
      - name: python_modelA
        language: python
        args: ./src/formatted_io1_modelA.py
        inputs: inputA
        outputs: outputA

      - name: python_modelB
        language: python
        args: ./src/formatted_io1_modelB.py
        inputs: inputB
        outputs: outputB

    connections:
      - input: outputA  # Connection between model A output & model B input
        output: inputB
      - input: ./Input/input.txt  # Connection between file and model A input
        output: inputA
        filetype: ascii
      - input: outputB  # Connection between model B output and file
        output: ./output.txt
        filetype: ascii

(Example in other languages)

## Tables by Row

Tables can also be passed in a similar manner. For input from a table, the format string does not need to be provided and will be determined by the source of the table. There are different API classes/functions for I/O from/to table channels versus standard channels in each language. (e.g. `CisInput` vs. `CisAsciiTableInput` in Python)

Model Code:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisAsciiTableInput, CisAsciiTableOutput)

    # Initialize input/output channels
    in_channel = CisAsciiTableInput('inputA')
    out_channel = CisAsciiTableOutput('outputA', '%6s\t%d\t%f\n')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, msg = in_channel.recv()
        if not flag:
            print("Model A: No more input.")
            break
        name, count, size = msg

        # Print received message
        print('Model A: %s, %d, %f' % (name, count, size))

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(name, count, size)
        if not flag:
            print("Model A: Error sending output.")
            break

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisAsciiTableInput, CisAsciiTableOutput)

    # Initialize input/output channels
    in_channel = CisAsciiTableInput('inputB')
    out_channel = CisAsciiTableOutput('outputB', '%6s\t%d\t%f\n')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, msg = in_channel.recv()
        if not flag:
            print("Model B: No more input.")
            break
        name, count, size = msg

        # Print received message
        print('Model B: %s, %d, %f' % (name, count, size))

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(name, count, size)
        if not flag:
            print("Model B: Error sending output.")
            break

(Example in other languages)

The `filetype: table` options in the YAML tell the cis_interface framework that the file should be read/written as a table row-by-row including verification that each row conforms with the table.

Model YAML:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    models:
      - name: python_modelA
        language: python
        args: ./src/formatted_io2_modelA.py
        inputs: inputA
        outputs: outputA

      - name: python_modelB
        language: python
        args: ./src/formatted_io2_modelB.py
        inputs: inputB
        outputs: outputB

    connections:
      - input: outputA  # Connection between model A output & model B input
        output: inputB
      - input: ./Input/input.txt  # Connection between file and model A input
        output: inputA
        filetype: table
      - input: outputB  # Connection between model B output and file
        output: ./output.txt
        filetype: table

(Example in other languages)

## Tables as Arrays

Tables can also be passed as arrays. In Python and Matlab, this is done using separate classes/functions. In C and C++, this is done by passing 1 to the `as_array` arguments of the table API classes/functions.

Model Code:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisAsciiArrayInput, CisAsciiArrayOutput)

    # Initialize input/output channels
    in_channel = CisAsciiArrayInput('inputA')
    out_channel = CisAsciiArrayOutput('outputA', '%6s\t%d\t%f\n')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, arr = in_channel.recv()
        if not flag:
            print("Model A: No more input.")
            break

        # Print received message
        print('Model A: (%d rows)' % len(arr))
        for i in range(len(arr)):
            print('   %s, %d, %f' % tuple(arr[i]))

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(arr)
        if not flag:
            print("Model A: Error sending output.")
            break

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisAsciiArrayInput, CisAsciiArrayOutput)

    # Initialize input/output channels
    in_channel = CisAsciiArrayInput('inputB')
    out_channel = CisAsciiArrayOutput('outputB', '%6s\t%d\t%f\n')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, arr = in_channel.recv()
        if not flag:
            print("Model B: No more input.")
            break

        # Print received message
        print('Model B: (%d rows)' % len(arr))
        for i in range(len(arr)):
            print('   %s, %d, %f' % tuple(arr[i]))

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(arr)
        if not flag:
            print("Model B: Error sending output.")
            break

(Example in other languages)

The YAML only differs in that `as_array: True` for the connections to the files to indicate that the files should be interpreted as tables and that the tables should be read/written in their entirety as arrays.

Model YAML:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    models:
      - name: python_modelA
        language: python
        args: ./src/formatted_io3_modelA.py
        inputs: inputA
        outputs:
          name: outputA
          field_names: name,count,size

      - name: python_modelB
        language: python
        args: ./src/formatted_io3_modelB.py
        inputs: inputB
        outputs:
          name: outputB
          field_names: name,count,size

    connections:
      - input: outputA  # Connection between model A output & model B input
        output: inputB
      - input: ./Input/input.txt  # Connection between file and model A input
        output: inputA
        as_array: True
        filetype: table
      - input: outputB  # Connection between model B output and file
        output: ./output.txt
        as_array: True
        filetype: table

(Example in other languages)

## Tables as Pandas Data Frames

In Python, tables can also be passed as Pandas data frames.

Model Code:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisPandasInput, CisPandasOutput)

    # Initialize input/output channels
    in_channel = CisPandasInput('inputA')
    out_channel = CisPandasOutput('outputA')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, frame = in_channel.recv()
        if not flag:
            print("Model A: No more input.")
            break

        # Print received message
        nrows = len(frame.index)
        print('Model A: (%d rows)' % len(frame.index))
        print(frame)

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(frame)
        if not flag:
            print("Model A: Error sending output.")
            break

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisPandasInput, CisPandasOutput)

    # Initialize input/output channels
    in_channel = CisPandasInput('inputB')
    out_channel = CisPandasOutput('outputB')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, frame = in_channel.recv()
        if not flag:
            print("Model B: No more input.")
            break

        # Print received message
        nrows = len(frame.index)
        print('Model B: (%d rows)' % nrows)
        print(frame)

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(frame)
        if not flag:
            print("Model B: Error sending output.")
            break

(Example in other languages)

The YAML specifies `filetype: pandas` for the connections to files to indicate that the files should be interpreted as CSV tables using Pandas.

Model YAML:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    models:
      - name: python_modelA
        language: python
        args: ./src/formatted_io4_modelA.py
        inputs: inputA
        outputs:
          name: outputA
          field_names: name,count,size

      - name: python_modelB
        language: python
        args: ./src/formatted_io4_modelB.py
        inputs: inputB
        outputs:
          name: outputB
          field_names: name,count,size

    connections:
      - input: outputA  # Connection between model A output & model B input
        output: inputB
      - input: ./Input/input.txt  # Connection between file and model A input
        output: inputA
        filetype: pandas
      - input: outputB  # Connection between model B output and file
        output: ./output.txt
        filetype: pandas

(Example in other languages)

As Pandas data frames are a Python specific construction, they cannot be used within models written in other languages. However, the files can still be read using Pandas. The data format returned to models on the receiving end of sent Pandas data frames will receive arrays in the proper native data type. In addition, a model written in Python can receive any array sent by another model (whether or not it is Python) as a Pandas data frame.

## 3D Structures as Ply/Obj

3D structures can be passed around in Ply or Obj formats.

Model Code:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    import pprint
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisPlyInput, CisPlyOutput)

    # Initialize input/output channels
    in_channel = CisPlyInput('inputA')
    out_channel = CisPlyOutput('outputA')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, ply = in_channel.recv()
        if not flag:
            print("Model A: No more input.")
            break

        # Print received message
        print('Model A: (%d verts, %d faces)' % (ply.nvert, ply.nface))
        pprint.pprint(ply)

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(ply)
        if not flag:
            print("Model A: Error sending output.")
            break

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    import pprint
    # Import classes for input/output channels
    from cis_interface.interface.CisInterface import (
        CisPlyInput, CisPlyOutput)

    # Initialize input/output channels
    in_channel = CisPlyInput('inputB')
    out_channel = CisPlyOutput('outputB')

    # Loop until there is no longer input or the queues are closed
    while True:

        # Receive input from input channel
        # If there is an error, the flag will be False
        flag, ply = in_channel.recv()
        if not flag:
            print("Model B: No more input.")
            break

        # Print received message
        print('Model B: (%d verts, %d faces)' % (ply.nvert, ply.nface))
        pprint.pprint(ply)

        # Send output to output channel
        # If there is an error, the flag will be False
        flag = out_channel.send(ply)
        if not flag:
            print("Model B: Error sending output.")
            break

(Example in other languages)

The YAML specifies `filetype: ply` (`filetype: obj` for Obj) for the connections to files to indicate that the files should be interpreted as Ply/Obj file formats.

Model YAML:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    models:
      - name: python_modelA
        language: python
        args: ./src/formatted_io5_modelA.py
        inputs: inputA
        outputs: outputA

      - name: python_modelB
        language: python
        args: ./src/formatted_io5_modelB.py
        inputs: inputB
        outputs: outputB

    connections:
      - input: outputA  # Connection between model A output & model B input
        output: inputB
      - input: ./Input/input.ply  # Connection between file and model A input
        output: inputA
        filetype: ply
      - input: outputB  # Connection between model B output and file
        output: ./output.ply
        filetype: ply

(Example in other languages)

In Python the data is returned as a dictionary subclass (`cis_interface.serialize.PlySerialize.PlyDict` or `cis_interface.serialize.ObjSerialize.ObjDict`) while in C/C++ it is returned as a structure (`ply_t` or `obj_t`).