Metadata-Version: 2.1
Name: influxio
Version: 0.2.0
Summary: Import and export data into/from InfluxDB
Author-email: Andreas Motl <andreas.motl@panodata.org>, Richard Pobering <richard.pobering@panodata.org>
License: MIT
Project-URL: changelog, https://github.com/daq-tools/influxio/blob/main/CHANGES.rst
Project-URL: documentation, https://github.com/daq-tools/influxio
Project-URL: homepage, https://github.com/daq-tools/influxio
Project-URL: repository, https://github.com/daq-tools/influxio
Keywords: export,ilp,import,influx,influxdb,line-protocol,lineprotocol,rdbms,sql,sqlalchemy,timeseries,timeseries-data,transfer
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Customer Service
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Manufacturing
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Telecommunications Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Communications
Classifier: Topic :: Database
Classifier: Topic :: Documentation
Classifier: Topic :: Education
Classifier: Topic :: Office/Business
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Version Control
Classifier: Topic :: System :: Archiving
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Text Processing
Classifier: Topic :: Utilities
Requires-Python: <3.13,>=3.8
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: click<9
Requires-Dist: colorama<1
Requires-Dist: crate[sqlalchemy]
Requires-Dist: cratedb-toolkit
Requires-Dist: dask[dataframe]>=2020
Requires-Dist: importlib-metadata; python_version <= "3.9"
Requires-Dist: influx-line==1.0.0
Requires-Dist: influxdb-client[ciso]<2
Requires-Dist: line-protocol-parser<2
Requires-Dist: pandas<2.3
Requires-Dist: psycopg2-binary<3
Requires-Dist: pueblo>=0.0.7
Requires-Dist: SQLAlchemy-Utils<0.42
Requires-Dist: yarl<2
Provides-Extra: develop
Requires-Dist: black<25; extra == "develop"
Requires-Dist: mypy==1.9.0; extra == "develop"
Requires-Dist: poethepoet<0.26; extra == "develop"
Requires-Dist: pyproject-fmt<1.8; extra == "develop"
Requires-Dist: ruff==0.3.5; extra == "develop"
Requires-Dist: validate-pyproject<0.17; extra == "develop"
Provides-Extra: release
Requires-Dist: build<2; extra == "release"
Requires-Dist: twine<6; extra == "release"
Provides-Extra: test
Requires-Dist: pytest<9; extra == "test"
Requires-Dist: pytest-cov<6; extra == "test"

########
influxio
########

.. start-badges

|ci-tests| |ci-coverage| |license| |pypi-downloads|

|python-versions| |status| |pypi-version|

.. |ci-tests| image:: https://github.com/daq-tools/influxio/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/daq-tools/influxio/actions/workflows/tests.yml
    :alt: Build status

.. |ci-coverage| image:: https://codecov.io/gh/daq-tools/influxio/branch/main/graph/badge.svg
    :target: https://app.codecov.io/gh/daq-tools/influxio
    :alt: Coverage

.. |pypi-version| image:: https://img.shields.io/pypi/v/influxio.svg
    :target: https://pypi.org/project/influxio/
    :alt: PyPI Version

.. |python-versions| image:: https://img.shields.io/pypi/pyversions/influxio.svg
    :target: https://pypi.org/project/influxio/
    :alt: Python Version

.. |pypi-downloads| image:: https://static.pepy.tech/badge/influxio/month
    :target: https://www.pepy.tech/projects/influxio
    :alt: PyPI Downloads per month

.. |status| image:: https://img.shields.io/pypi/status/influxio.svg
    :target: https://pypi.org/project/influxio/
    :alt: Status

.. |license| image:: https://img.shields.io/pypi/l/influxio.svg
    :target: https://pypi.org/project/influxio/
    :alt: License

.. end-badges


*****
About
*****

You can use ``influxio`` to import and export data into/from InfluxDB.
It can be used both as a standalone program, and as a library.

``influxio`` is, amongst others, based on the excellent `dask`_, `fsspec`_,
`influxdb-client`_, `influx-line`_, `line-protocol-parser`_, `pandas`_,
and `SQLAlchemy`_ packages.

Please note that ``influxio`` is alpha-quality software, and a work in progress.
Contributions of all kinds are very welcome, in order to make it more solid.
Breaking changes should be expected until a 1.0 release, so version pinning
is recommended, especially when you use it as a library.


********
Synopsis
********

.. code-block:: shell

    # Export from API to database.
    influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "sqlite://export.sqlite?table=demo"

    # Export from data directory to line protocol format.
    influxio copy \
        "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
        "file://export.lp"


**********
Quickstart
**********

If you are in a hurry, and want to run ``influxio`` without any installation,
just use the OCI image on Podman or Docker.

.. code-block:: shell

    docker run --rm --network=host ghcr.io/daq-tools/influxio \
        influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "crate://crate@localhost:4200/testdrive/demo"


*****
Setup
*****

Install ``influxio`` from PyPI.

.. code-block:: shell

    pip install influxio


*****
Usage
*****

This section outlines some example invocations of ``influxio``, both on the
command line, and per library use. Other than the resources available from
the web, testing data can be acquired from the repository's `testdata`_ folder.

Prerequisites
=============

For properly running some of the example invocations outlined below, you will
need an InfluxDB and a CrateDB server. The easiest way to spin up those
instances is to use Podman or Docker.

Please visit the ``docs/development.rst`` documentation to learn about how to
spin up corresponding sandbox instances on your workstation.

Command line use
================

Help
----

.. code-block:: shell

    influxio --help
    influxio info
    influxio copy --help

Import
------

.. code-block:: shell

    # From test data to API.
    # Choose one of dummy, mixed, dateindex, wide.
    influxio copy \
        "testdata://dateindex/" \
        "http://example:token@localhost:8086/testdrive/demo"

    # With selected amount of rows.
    influxio copy \
        "testdata://dateindex/?rows=42" \
        "http://example:token@localhost:8086/testdrive/demo"

    # With selected amount of rows and columns (only supported by certain test data sources).
    influxio copy \
        "testdata://wide/?rows=42&columns=42" \
        "http://example:token@localhost:8086/testdrive/demo"

    # From line protocol file to InfluxDB API.
    influxio copy \
        "file://tests/testdata/basic.lp" \
        "http://example:token@localhost:8086/testdrive/demo"

    # From line protocol file to InfluxDB API.
    influxio copy \
        "https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp" \
        "http://example:token@localhost:8086/testdrive/demo"

    # From line protocol file to any database supported by SQLAlchemy.
    influxio copy \
        "file://export.lp" \
        "sqlite://export.sqlite?table=export"


Export from API
---------------

.. code-block:: shell

    # From API to database file.
    influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "sqlite:///export.sqlite?table=demo"

    # From API to database server.
    influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "crate://crate@localhost:4200/testdrive?table=demo"

    # From API to line protocol file.
    influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "file://export.lp"

    # From API to line protocol on stdout.
    influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "file://-?format=lp"


Export from data directory
--------------------------

.. code-block:: shell

    # From InfluxDB data directory to line protocol file.
    influxio copy \
        "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
        "file://export.lp"

    # From InfluxDB data directory to line protocol file, compressed with gzip.
    influxio copy \
        "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
        "file://export.lp.gz"

    # From InfluxDB data directory to line protocol on stdout.
    influxio copy \
        "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
        ""file://-?format=lp"


OCI
---

OCI images are available on the GitHub Container Registry (GHCR). In order to
run them on Podman or Docker, invoke:

.. code-block:: shell

    docker run --rm --network=host ghcr.io/daq-tools/influxio \
        influxio copy \
        "http://example:token@localhost:8086/testdrive/demo" \
        "stdout://export.lp"

If you want to work with files on your filesystem, you will need to either
mount the working directory into the container using the ``--volume`` option,
or use the ``--interactive`` option to consume STDIN, like:

.. code-block:: shell

    docker run --rm --volume=$(pwd):/data ghcr.io/daq-tools/influxio \
        influxio copy "file:///data/export.lp" "sqlite:///data/export.sqlite?table=export"

    cat export.lp | \
    docker run --rm --interactive --network=host ghcr.io/daq-tools/influxio \
        influxio copy "stdin://?format=lp" "crate://crate@localhost:4200/testdrive/export"

In order to always run the latest ``nightly`` development version, and to use a
shortcut for that, this section outlines how to use an alias for ``influxio``,
and a variable for storing the input URL. It may be useful to save a few
keystrokes on subsequent invocations.

.. code-block:: shell

    docker pull ghcr.io/daq-tools/influxio:nightly
    alias influxio="docker run --rm --interactive ghcr.io/daq-tools/influxio:nightly influxio"
    SOURCE=https://github.com/daq-tools/influxio/raw/main/tests/testdata/basic.lp
    TARGET=crate://crate@localhost:4200/testdrive/basic

    influxio copy "${SOURCE}" "${TARGET}"


*******************
Project information
*******************

Development
===========
For installing the project from source, please follow the `development`_
documentation.

Prior art
=========
There are a few other projects which are aiming at similar goals.

- `InfluxDB Fetcher`_
- `influxdb-write-to-postgresql`_ (IW2PG)
- `Outflux`_


.. _dask: https://www.dask.org/
.. _development: doc/development.rst
.. _fsspec: https://pypi.org/project/fsspec/
.. _influx: https://docs.influxdata.com/influxdb/latest/reference/cli/influx/
.. _influx-line: https://github.com/functionoffunction/influx-line
.. _influxd: https://docs.influxdata.com/influxdb/latest/reference/cli/influxd/
.. _InfluxDB Fetcher: https://github.com/hgomez/influxdb
.. _InfluxDB line protocol: https://docs.influxdata.com/influxdb/latest/reference/syntax/line-protocol/
.. _influxdb-client: https://github.com/influxdata/influxdb-client-python
.. _influxdb-write-to-postgresql: https://github.com/eras/influxdb-write-to-postgresql
.. _line-protocol-parser: https://github.com/Penlect/line-protocol-parser
.. _list of other projects: doc/prior-art.rst
.. _Outflux: https://github.com/timescale/outflux
.. _pandas: https://pandas.pydata.org/
.. _SQLAlchemy: https://pypi.org/project/SQLAlchemy/
.. _testdata: https://github.com/daq-tools/influxio/tree/main/tests/testdata
