Metadata-Version: 2.1
Name: rocksq
Version: 0.1.4
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Summary: Rust persistent queue based on RocksDB with Python bindings
Keywords: database,queue,persistent,rocksdb,python
Home-Page: https://github.com/insight-platform/RocksQ
Author: Ivan Kudriavtsev <ivan.a.kudryavtsev@gmail.com>
Author-email: Ivan Kudriavtsev <ivan.a.kudryavtsev@gmail.com>
License: Apache-2.0
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Source Code, https://github.com/insight-platform/RocksQ

# RocksQ

An inproc RocksDB-based queue with Python bindings. It is implemented in Rust.

Features:

- max capacity limit in number of elements;
- size calculation based on filesystem space usage;
- length calculation based on number of elements;
- supports only bytes-like objects;
- can operate in a multithreaded environment efficiently (push and pop methods can release GIL if necessary);
- keeps the state between restarts;
- pop is non-blocking (returns None if the queue is empty);

What is not supported:

- pub/sub is not supported intentionally (implement it on top of RocksQ if necessary);
- TTL is not supported intentionally (implement it on top of RocksQ if necessary).

## Implementation details

It works on RocksDB and uses a single column family. The keys are 128-bit integers, the values are byte arrays. The keys are generated by incrementing a counter. The read and write counters are stored in a separate key-value pairs.

It is implemented in Rust and PyO3, thus allows to release GIL when necessary. The library does not require external dependencies installed in the environment.

## Supported Platforms and Python Versions

The library is automatically built for X86_64 and AARCH64 architectures with the Manylinux runtime.

Python versions: 3.7-3.12. CI does not build for PyPy, but it should work if you build it manually.

## Installation

```
pip install rocksq
```

## Usage

```python
import time
from rocksq import PersistentQueueWithCapacity

# Create a queue that will persist to disk

# try:
#     PersistentQueue.remove_db('/tmp/queue')
# except:
#     pass

q = PersistentQueueWithCapacity('/tmp/queue')

buf = bytes(256 * 1024)

OPS = 8 * 30
RELEASE_GIL = True

start = time.time()
print("begin writing")
for i in range(OPS):
    q.push(buf, no_gil=RELEASE_GIL)

print("begin reading")
for i in range(OPS):
    v = q.pop(no_gil=RELEASE_GIL)


end = time.time()

print("Time taken: %f" % (end - start))
```

## Performance

The performance is mostly limited by the throughput of the underlying filesystem. The queue is able to saturate the throughput of the filesystem.


