# OpenZGY library

[[_TOC_]]

## <span style="color:blue">What's in the box?</span>

The ZGY file format was developed by Schlumberger for storing 3d data
for seismic interpretation.

The OpenZGY libraries allow reading and writing this format. There are
both C++ and Python versions. For Python you can choose between a thin
wrapper around the C++ implementation and a pure Python reference
implementation. In the reference implementation, zfp compression and
cloud access are separate binary packages but the core is only Python. 

## <span style="color:blue">Documentation</span>

See the doc folder for human readable documentation.
The file format itself is explained in [doc/openzgy](./doc/openzgy.html).
Other files in that folder deal with more specific issues.

Documentation for the public API and for internal code is generated by
Doxygen in the Linux builds. You need to do a "make doxygen" at the top
level before the documentation becomes available in

    build/deploy/{native,pure}/{apidoc,intdoc}.{tgz,pdf}

The .tgz files contain html pages. Unpack the tarball and open
html/index.html. If you have
[LaTeX](https://www.latex-project.org/)
installed when building there will also be a single-document pdf
versions of those two next to the .tgz files.

## <span style="color:blue">Tools</span>

The repository contains a tool for copying a zgy file, possbly with
compression or uncompression and possibly with uploading to or
downloading from the cloud.

There are also various tools for inspecting a ZGY file.

A [separate project](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion)
provides conversion from SEG-Y to ZGY.

## <span style="color:blue">Performance</span>

The pure Python reference implementation is single threaded.

OpenZGY/C++ will in most cases make use of multiple threads.

Applications are allowed to issue multiple read requests in parallel
to the same open file. Even if they don't, parallelization done inside
OpenZGY ensures that the performance is still reasonable

Read requests to cloud storage will automatically consolidate neighbor
bricks. This makes cloud I/O faster for typical access patterns, since
reading a typical brick (1MB) at a time from cloud storage is very
inefficient. Note that the brick size isn't increased, it just looks
that way. So when you need smaller bricks because you have scattered
access then this will work as well.

Individual read requests will be parallelized inside OpenZGY for
copy-out, int8/int16 to float conversion, compression, linux local
file read, and cloud read. As applicable.

Write requests will be serialized with respect to float to int8/int16
conversion, compression, and upload to cloud where applicable.
Also the algorithms that decimate for low resolution data.

The last two bullets are implemented for local file access but some
are still work in progress for cloud I/O.

## <span style="color:blue">Building and testing the core parts</span>

### Building Linux core

The code should build pretty much out of the box as long as Seismic
Store access is not enabled. On Linux there is a top level Makefile.
"make" builds and tests everything, while "make build" only builds.
In both cases both C++ and Python versions are built / packaged.

### Building Windows core

Prerequisites are Visual Studio 2019 with Platform Toolset v142.
If building with cloud support you might also need NuGet,
or some other mechanism to consume the Seismic Store libraries.

On Windows there is a Visual Studio solution for building the C++ code
and you can run "cd python; python setup.py sdist bdist_wheel" to
package the pure Python version.

The Visual Studio solution includes a project for running the C++ tests.

#### Example Python manual tests

These are for windows.
You can run them on Linux also by adjusting the paths.

```sh
cd python
virtualenv vvv
vvv\Scripts\activate
pip install dist\OpenZGY-0.2.unknown-py3-none-any.whl
zgycopy --snr 30 original.zgy compressed.zgy
zgycopy compressed.zgy roundtrip.zgy
zgygui
```

#### Example C++ manual tests

```sh
set OPENZGY_TESTDATA=build\testdata
build\deploy\native\x64\Debug\OpenZGY.Test.exe
```

```sh
build\deploy\native\x64\Debug\OpenZGY.Tools.ZgyCopy.exe -i original.zgy -o compressed.zgy --sqnr 30 --float
build\deploy\native\x64\Debug\OpenZGY.Tools.ZgyCopy.exe -i compressed.zgy -o roundtrip.zgy
```

## <span style="color:blue">Building and testing with cloud access enabled</span>

### Source code for SDAPI

There are two git repositories where the SDAPI source code can be downloaded.
The repository where active development is taking place is only accessible inside Schlumberger at
[https://dev.azure.com/.../os-seismic-store-cpp-lib](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-seismic-store-cpp-lib).
The repository that is publicly accessible but may lag behind the internal version is at
[https://community.opengroup.org/.../seismic-store-cpp-lib](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib.git)
So when the instructions below mention downloading the cloud library you may download SDAPI from
[Azure](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-seismic-store-cpp-lib) or [OSDU](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib.git).

### Building Linux cloud

If you want to use Seismic Store to access the cloud then there are
some additional steps needed. You will need to download and build the
Seismic Store SDK a.k.a. SDAPI. And then package the binaries so they
can be picked up by OpenZGY. The Linux build expects it to be
available as a tarball in the source tree. So if you want cloud
support this will need some manual tweaking.

See [Building the Seismic Store SDK on Linux](building-the-seismic-store-sdk-on-linux)
for more information.

In the C++ code the cloud support and the compression support will
automatically be included if the SDK is available in the source tree.
And they will be quietly support if the respective tar files are missing.

Alternatively you can explicitly add HAVE_SD=yes and HAVE_ZFP=yes to
make sure those get built and tested, with error messages if they are
not found. Or set to an empty string (not "no"!) if you don't want to
build them even if present.

In the Python code the cloud support gets compiled into a C++ Python
extension which needs to be installed (using pip) next to OpenZGY
itself.

### Building Windows cloud

By default, integration with Seismic Store is disabled for windows
build. To enable it you need to explicitly define HAVE_SD=yes and edit
the project OpenZGY.vcxproj to link with the Seismic Store SDK (a.k.a.
SDAPI) that can be downloaded from
[Azure](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-seismic-store-cpp-lib) or [OSDU](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib.git).

On Windows the Visual Studio solution expects the SDAPI headers and
binaries to be available as a NuGet package. You will need to download
and build the Seismic Store SDK and push the binaries to NuGet.

To enable reading and writing seismic store files using the pure
Python implementation you will need to build a small binary Python
extension module that wraps the C++ SDAPI. This produces a wheel that
you can subsequently install. Currently this module only builds on
Linux.

The C++ Python extension that wraps the entire OpenZGY/C++
implementation also does not build yet. So for cloud access you are
currently limited to C++.

```sh
@rem ... Download https://dist.nuget.org/win-x86-commandline/latest/nuget.exe
@rem ... Download Visual Studio 2019 and Platform Toolset v142 from Microsoft
@rem ... Open a Developer Command prompt for VS 2019
git clone https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy.git
nuget.exe restore openzgy\native\windows\packages.config -PackagesDirectory openzgy\native\windows\packages
msbuild.exe openzgy\native\windows\OpenZGY.Test.vcxproj /p:Configuration=Debug /p:Platform="x64"
```

### Testing on the cloud

For running tests against the cloud you will also need to set
OPENZGY_SDAPIKEY, OPENZGY_SDURL, and OPENZGY_TOKEN to appropriate
values. Production code using OpenZGY will probably use a different
mechanism to configure cloud access. Also set OPENZGY_SDTESTDATA and
OPENZGY_SDTESTSINK to two locations in the cloud. The first one a
read-only folder of test data and the second one an empty folder for
tests to write to.

## <span style="color:blue">Feature matrix</span>

|Package                |linux|windows|read|write|update|seisstore|zfp compress|old compress|
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|OpenZGY/C++            |y|y|y|y|-|y|y|N/A|
|OpenZGY/C++ Python wrapper |y|-|y|y|-|y|y|N/A|
|OpenZGY/Python         |y|y|y|y|N/A?|linux|y|N/A|
|ZGY-Public, ZGY-Cloud  |y|y|y|y|y|y|N/A|y|
|Old Python wrapper     |y|y|y|y|y|y|N/A|y|

## <span style="color:blue">Output folders</span>

On Linux, running "make build" or "make" at the top level produces the deliverables listed below. On Windows the output folder structure is similar.

| <span style="color:#FF6622; font-size: 14pt;">native</span> |
| ---------- |
| This is a C++ implementation of the OpenZGY library. |

    build/deploy/native/include/openzgy/...
    build/deploy/native/$arch/*.so* (linux), *.dll/*.lib (windows)
    build/deploy/native/{apidoc,intdoc}.{tgz,pdf}

| <span style="color:#FF6622; font-size: 14pt;">wrapper</span> |
| ---------- |
| This is a Python wrapper around the native implementation. It is supposed to be functionally identical to the legacy python implementation. |
  
    build/deploy/wrapper/$arch/OpenZgyBindings-*.whl

| <span style="color:#FF6622; font-size: 14pt;">python</span> |
| ---------- |
| This is a pure python implementation of the core OpenZGY library. Binary packages are needed for optional compression and seismic store access. |

    build/deploy/pure/OpenZGY-*/OpenZGY-*.whl
    build/deploy/pure/OpenZGY-*/OpenZGY-*.tar.gz
    build/deploy/sdglue/$arch/SdGlue*.whl
    build/deploy/pure/{apidoc,intdoc}.{tgz,pdf}

| <span style="color:#FF6622; font-size: 14pt;">legacy C++<br>legacy ZGY-Public</span>|
| ---------- |
| This is the binary SDK for the closed source ZGY-Public library and ZGY-Cloud plug-in with a Python wrapper. This software is deprecated. |

    zgy/Slb.Salmon.ZgyPublic/doc/...
    zgy/Slb.Salmon.ZgyPublic/include/...
    zgy/Slb.Salmon.ZgyPublic/Lin64_gcc485/...
    zgy/Slb.Salmon.ZgyPublic/x64_vs140/...

## <span style="color:blue">Building the Seismic Store SDK on Linux</span>

The Seismic Store SDK (a.k.a. SDAPI) source code can be downloaded from
[Azure](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-seismic-store-cpp-lib) or [OSDU](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib.git).
The simplest approach is to clone this into the top level of the OpenZGY folder.
Important: You need to use **git clone --recursive**.
Also create an empty folder named seismic-service-bin to hold the binaries
built from os-seismic-store-cpp-lib. So your directory structure should be
something like:
```
openzgy/
  +-- native/...
  +-- python/...
  +-- (etc)
  +-- os-seismic-store-cpp-lib/...
  +-- seismic-service-bin/...
```

Build os-seismic-store-cpp-lib according to the instructions in the
README.md file in
[Azure](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-seismic-store-cpp-lib) or [OSDU](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib.git).
Or use the scripts in sd-env.
If cloning from the OSDU repo you might still want to clone into
os-seismic-store-cpp-lib instead of seismic-store-cpp-lib as it is
called there, so the instructions in this file match exactly.

The software requires gcc 4.9.2 or later. This means that if building
on CentOS/RedHat 7 the compilers need to be upgraded. This can cause
some annoying ripple effects.

Bundle the required headers and the compiled binaries into a gzipped
tar file, like so:

```sh
mkdir -p tmp/include tmp/lib/linux64
cp -a -t tmp/include ${SRC}/src/core/*.h ${SRC}/src/lib/accessors/*.h
cp -a -t tmp/lib/linux64 libjsoncpp* libsdapi*
tar zcf sdapi_linux64_osdu.tar.gz -C tmp .
```

Move the tar file to
&lt;OpenZGY-source&gt;/seismic-service-bin/Lin64_gccNNN/sdapi_linux64_osdu.tar.gz,
replacing gccNNN with the version of the compiler being used. So gcc
8.3.1 (currently the default in CentOS/RedHat 8) would be Lin64_gcc831.

Installing the dependencies needed to build SDAPI can be a bit
tedious. Especially if you want to include Azure support. And
especially if you are worried about polluting your regular Linux
installation with a lot of extra packages. There exists a set of
docker files in sd-env/* that might help. Or they might confuse the
issues further. The docker setup allows building code for multiple
Linux distros on the same server. It also tries to do more work on
version handling. This complicates the setup. See sd-env/Makefile for
details.

Note: The Lin64_gccNNN versioning scheme is a holdover from some very
old system and doesn't make much sense. But you shouldn't have any
problems with it unless you are building multiple targets from the
same source folder. The problem is that (a) only the compiler, not the
Linux distribution is included in the name and (b) there really is no
need to include the compiler's patch number in the folder name. If two
Linux distributions happen to use the exact same compiler version then
there is a name clash. <!-- TODO-Low: Fix versioning scheme -->

Note: The top level Makefile accepts a SDAPI_INTERNAL=yes argument. All
this does is to switch the name of the SDAPI tar file from
sdapi_linux64_osdu.tar.gz to sdapi_linux64_local.tar.gz. This is only
useful when building multiple versions of SDAPI.

### Complete example - Linux

Here I am running on Ubuntu Xenial and I want to build the SDAPI
library inside a docker container and then build OpenZGY itself
outside docker.

```sh
git clone git@ssh.dev.azure.com:v3/slb-swt/Colors/openzgy oz
cd oz
git clone --recursive git@ssh.dev.azure.com:v3/slb-des-ext-collaboration/open-data-ecosystem/os-seismic-store-cpp-lib
cd sd-env
make clobber
make build-xenial
make run-xenial
cd ..
docker start -ai sd-xenial
  ./build.sh
  exit
mkdir -p seismic-service-bin/Lin64_gcc540
docker cp 47dfde70a3d5:/home/me/sdapi_linux64.tar.gz seismic-service-bin/Lin64_gcc540/sdapi_linux64_osdu.tar.gz
make clobber
make build
env OPENZGY_SDURL=... OPENZGY_SDAPIKEY=... OPENZGY_TOKEN=... make
```

## <span style="color:blue">Building the Seismic Store SDK on Windows</span>

Build os-seismic-store-cpp-lib according to the instructions in the
README.md file in
[Azure](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-seismic-store-cpp-lib) or [OSDU](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-cpp-lib.git).

TODO-Doc: Add more details. Especially in how to package the result
(e.g. in NuGet) for consumption by the OpenZGY build.

### Complete example - Windows

TODO-Doc: ADD EXAMPLE.

## <span style="color:blue">Docker and Azure DevOps</span>

Most of the files in scripts/, and azure/ folders are used for setting
up automated builds of multiple versions of OpenZGY. You don't need
these and you don't need docker if you just want to build a single
version of the code that will run on your current architecture.

Similarly, the files in sd-env/ might help building the Seismic Store
library but they are entirely optional. See "Integration with Seismic
Store". The README.md file in Seismic Store explains how to build the
software by hand. If you decide to use sd-env/ you may need to
customize those files.

That being said, the docker files might help setting up the
prerequisites correctly. And using docker avoids polluting your
current environment with a lot of extra packages. The downside is that
if you are not already familiar with docker then this approach might
involve more effort. And the scripts here need to do more and are thus
somewhat more complex than building by hand.

Running "make docker" or "make docker-build" in the top level will try
to use scripts/Dockerfile* to build OpenZGY inside a docker instance.

## <span style="color:blue">Runtime dependencies</span>

If you build OpenZGY for Linux and then want to install the built packages
on a different machine then you may need to install a couple of dependencies
first. The list is much shorter than the environment you need for building.

Please refer to scripts/Dockerfile-DISTRO-test with DISTRO replaced with
the Linux flavor you are using. These dockerfiles assume there is a built
deploy.tgz file in the current directory and will produce a Docker image
with both the two Python versions and the C++ library installed.

The C++ version is not yet installed in /usr/local... TODO fix.

Even if you aren't using Docker the files still show you what you need.
