This is a Cython module with bindings to the [crcany](https://github.com/madler/crcany) library. It supports calculating CRC hashes of arbitary sizes as well as updating a crc hash over time. It takes advantage of crcany's ability to efficiently combine multiple CRCs to parallelize the CRC's calculation.

## Installation

`pip install anycrc`

## Usage

Use an existing model:

```python
>>> import anycrc
>>> crc32 = anycrc.Model('CRC32')
>>> crc32.calc(b'Hello World!')
472456355
```

Read the data in chunks:

```python
>>> crc32.update(b'Hello ')
3928882368
>>> crc32.update(b'World!')
472456355
```

The `update` method changes the internally stored CRC value, while `calc` doesn't. You can use `get` to retrieve the CRC value stored within the object:

```python
>>> crc32.get()
472456355
```

To specify the starting CRC value:

```python
>>> crc32.set(3928882368)
>>> crc32.calc('World!')
472456355
```

To go back to the initial value, use:

```python
>>> crc32.reset()
```

Specify your own CRC parameters:

```python
>>> # width, poly, init, refin, refout, xorout
>>> my_crc = anycrc.CRC(10, 0b0101010101, 0x3ff, True, False, 0)
>>> my_crc.calc('Hello World!')
35
```

The CRC will be computed in parallel when the input is large enough. If you want to disable parallelism, use:

```python
>>> anycrc.set_parallel(False)
```

For a list of pre-built models, check [models.py](https://github.com/marzooqy/anycrc/blob/main/src/anycrc/models.py). To get a list of the models at any time, use the following command:

`python -m anycrc models`

The algorithm used will depend on the length of the input and the CRC's width:

| Data Length | Width ≤ 64 | Width > 64 |
|:-:|:-:|:-:|
| Length < 16 | byte-by-byte | byte-by-byte |
| 16 ≤ Length < 20,000 | slice-by-16 | byte-by-byte |
| Length ≥ 20,000 | parallel slice-by-16 | byte-by-byte |

The maximum possible CRC width is 128 bits.

## Benchmarks

Calculating the CRC32 for lorem ipsum 1 million times:

| Module | Time Elapsed (s) | Speed (GiB/s) | Relative |
|---|:-:|:-:|:-:|
| anycrc | 0.19 | 2.14 | 1.00 |
| zlib | 0.28 | 1.49 | 1.44 |
| fastcrc | 0.80 | 0.52 | 4.12 |
| crcmod-plus | 0.91 | 0.46 | 4.71 |

Calculating the CRC32 for the text of lorem ipsum repeated 1 million times in a single pass:

| Module | Time Elapsed (s) | Speed (GiB/s) | Relative |
|---|:-:|:-:|:-:|
| anycrc (parallel) | 0.02 | 24.51 | 1.00 |
| anycrc (serial) | 0.13 | 3.21 | 7.64 |
| zlib | 0.22 | 1.93 | 12.72 |
| fastcrc | 0.67 | 0.62 | 39.75 |
| crcmod-plus | 0.67 | 0.62 | 39.73 |

Tested on a 10th generation Intel i7 processor. Parallel performance will depend on your system.

#### Notes

Parallelism is disabled when the length of the input data is under 20k, as the serial method is faster in that case.
