# Ion Python C Extension

1. [Overall](#overall)
2. [Motivation](#motivation)
3. [Performance Improvement](#performance-improvement)
4. [Setup](#setup)
5. [Development](#development)
6. [Technical Details](#technical-details)<br>
   6.1 &nbsp;[Common Binary Encoding Differences between C Extension and Original Ion Python](#1-common-binary-encoding-differences-between-c-extension-and-original-ion-python)<br>
   6.2 &nbsp;[Known Issues](#2-known-issues)<br>
7. [TODO](#todo)
8. [Deploy](#deploy)<br>
   8.1 &nbsp;[Distribution](#1-distribution)<br>

## Overall

Ion Python C extension utilizes Ion C to access files that close the performance gap between the Ion Python simpleion module and other Ion implementations.

The simpleion module C extension supports limited options for now and will add more incrementally. Refer to [TODO](#todo) for details.  

## Motivation

Python is not fast which causes Ion Python to be slower than other Ion implementations. Ion Python is also slower than other similar python data serialization libraries such as simplejson which is a JSON encoder and decoder. The main reason for the difference in performance between Simplejson and Ion Python simpleion module is because Simplejson binds to a C extension while Ion Python is implemented purely in python. 

There are couple technologies we can choose for binding C extension and C binaries (Ion C): CFFI, Cython and CPython APIs.

CFFI and Ctypes are slower than CPython and Cython for most of our use case, Cython is a little bit faster than CPython but it's a compiler for a new programming language that requires more development time. One of the most challenging issues no matter which tool we use is that how we distribute Ion C binaries as it's `.dylib` on Mac, `.so` on Linux and `.lib` on Windows. Also, CPython C extension code for simpleion was almost completed 2 years ago so we decided to choose this option. 

If the performance becomes our biggest concern in the future, we should reevaluate the performance implications of the C extension to make sure we're keeping up with the innovations in the Python C extension ecosystem.




## Performance Improvement

The performance improvement depends on a multitude of variables (e.g., how the files are structured, what APIs are called the most). Experiment results show **around** 6000% improvement for text writer/reader and 1400% improvement for binary writer/reader.

We use `timeit` module to measure the execution time.
```.py
setup = "from amazon.ion import simpleion"
code = '''
with open("file_name", "br") as fp:
    simpleion.dumps(simpleion.load(fp, single_value=False))
'''
print(timeit.timeit(setup=setup, stmt=code, number=1))
```

#### Experiment Result
`test-driver-report.ion(10n)` are reports generated by [ion-test-driver](https://github.com/amazon-ion/ion-test-driver) which consists of Ion structs and strings.<br/>
`log.ion(10n)` are logs that contain a variety of scalar types, annotations, and nested containers.<br/>

|Files|C extension|Ion Python|Improvement|
|---|---|---|---|
|test-driver-report.ion (42MB)|3.8s|217s|5611%|
|test-driver-report.10n (13.7MB)|3.6s|55s|1428%|
|log.ion (84MB)|14.8s|987s|6569%|
|log.10n (14MB)|15s|221s|1373%|


## Setup

Ensure that cmake is installed. The setup for Ion Python C extension is the same as the original [Ion Python Setup](https://github.com/amazon-ion/ion-python#development). If it runs into any issue during initialization, it will fall back to regular Ion Python. **No extra action needed.**

C extension is built under `ion-python/amazon/ion` and named according to the following format (may be slightly different depending on your platform) `ionc.cpython-$py_version-$platform.$suffix` (e.g., ionc.cpython-39-darwin.so)

#### Getting Started with C Extension:
```
>>> import amazon.ion.simpleion as ion
>>> obj = ion.loads('{abc: 123}')
>>> obj['abc']
123
>>> ion.dumps(obj, binary=True)
b'\xe0\x01\x00\xea\xe9\x81\x83\xd6\x87\xb4\x83abc\xd3\x8a!{'
```


## Development

Architecture of Ion Python C extension:
```
                                   ioncmodule.c
                                        |
                                        | 
                                        ↓ 
Ion C -------> Ion C binaries -----> setup.py ------> C extension -------------------> Ion Python simpleion module
      compile                                  setup               import ionc module   
```
After setup, C extension will be built and imported to simpleion module. If there are changes in `ioncmodule.c`, build the latest C extension by running `python setup.py build_ext --inplace`.


## Technical Details

### 1. Common Binary Encoding Differences between C Extension and Original Ion Python
Note that both binary encodings are **equivalent**; one encoding is not more "correct" than the other.<br/>

#### 1.1 Different ways to represent a struct's length. Refer to [Amazon Ion Binary Encoding](https://amazon-ion.github.io/ion-docs/docs/binary.html#13-struct) for details.<br/>
For Ion struct `{a:2}`:
```text
Text       IVM               ion_symbol_table::{         symbols:[”a”]}  {         “a”: 2     }   
Ion C      \xe0\x01\x00\xea  \xe7\x81\x83      \xd4      \x87\xb2\x81a   \xd3      \x8a 21\x02
Ion Python \xe0\x01\x00\xea  \xe8\x81\x83      \xde\x84  \x87\xb2\x81a   \xde\x83  \x8a 21\x02
```

#### 1.2 Different order of symbols within a symbol table.<br/>
For symbol `abc` with two annotations `annot1` and `annot2`, `annot1::annot2::abc`:
```text
Ion C text        ion_symbol_table::{         symbols:[    "abc", "annot1", "annot2"]}          annot1($11)::annot2($12)::abc($10)
Ion C binary      \xee\x99\x81\x83  \xde\x95  \x87\xbe\x92 \x83abc\x86annot1\x86annot2 \xe5\x82 \x8b         \x8c         \x71\x0a
Ion Python binary ion_symbol_table::{         symbols:[    "annot1", "annot2", "abc",]}         annot1($10)::annot2($11)::abc($12)
ion Python        \xee\x99\x81\x83  \xde\x95  \x87\xbe\x92 \x86annot1\x86annot2\x83abc \xe5\x82 \x8a         \x8b         \x71\x0c
```

### 2. Known Issues

1. C extension only supports at most 9 for timestamp precision. Refer to [amazon-ion/ion-python#160](https://github.com/amazon-ion/ion-python/issues/160) for details.
2. C extension only supports at most 34 decimal digits. Refer to [amazon-ion/ion-python#159](https://github.com/amazon-ion/ion-python/issues/159) for details.
3. C extension has a limitation to read large Clob data. Refer to [amazon-ion/ion-python#207](https://github.com/amazon-ion/ion-python/issues/207) for details.
4. For any memory leak issue, please comment on [amazon-ion/ion-python#155](https://github.com/amazon-ion/ion-python/issues/155).

## TODO

1. More bug fixing.
2. More performance improvement.
3. Support more simpleion options such as `imports`, `catalog`, `omit_version_marker`. (Ion Python uses pure python implementation to handle unsupported options currently)
4. Support pretty print.

## Deploy

### 1. Distribution
PYPI supports two ways of distribution: [Source Code Distribution](https://packaging.python.org/guides/distributing-packages-using-setuptools/#source-distributions) and [Wheel Distribution](https://packaging.python.org/guides/distributing-packages-using-setuptools/#wheels). We support both of them. <br/>

Note that the benefits of wheel distribution are:
1. Pre-compiling Ion C library avoids potential build/compile issues and does not require a C compiler to be present on the user's machine.
2. Installation of wheels is faster and more efficient.

