Metadata-Version: 2.1
Name: pgsync
Version: 1.1.7
Summary: Postgres to elasticsearch sync
Home-page: https://github.com/toluaina/pg-sync
Author: Tolu Aina
Author-email: tolu@pgsync.com
Maintainer: Tolu Aina
Maintainer-email: tolu@pgsync.com
License: UNKNOWN
Project-URL: Bug Reports, https://github.com/toluaina/pgsync/issues
Project-URL: Funding, https://patreon.com/toluaina
Project-URL: Source, https://github.com/toluaina/pg-sync
Project-URL: Web, https://pgsync.com
Keywords: pgsync,elasticsearch,postgres,change data capture
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
Requires-Python: >=3.6.0
Description-Content-Type: text/markdown
Requires-Dist: certifi (==2020.6.20)
Requires-Dist: click (==7.1.2)
Requires-Dist: elasticsearch-dsl (==7.2.1)
Requires-Dist: elasticsearch (==7.8.1)
Requires-Dist: environs (==8.0.0)
Requires-Dist: faker (==4.1.1)
Requires-Dist: marshmallow (==3.7.1)
Requires-Dist: newrelic (==5.16.0.145)
Requires-Dist: psycopg2-binary (==2.8.5)
Requires-Dist: python-dateutil (==2.8.1)
Requires-Dist: python-dotenv (==0.14.0)
Requires-Dist: redis (==3.5.3)
Requires-Dist: six (==1.15.0)
Requires-Dist: sqlalchemy (==1.3.18)
Requires-Dist: sqlparse (==0.3.1)
Requires-Dist: text-unidecode (==1.3)
Requires-Dist: urllib3 (==1.25.10)

# PostgreSQL to Elasticsearch sync


`PGSync <https://pgsync.com>`_ is a middleware for syncing data from `Postgres <https://www.postgresql.org>`_ to `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_.  
It allows you to keep `Postgres <https://www.postgresql.org>`_ as your source of truth data source and
expose structured denormalized documents in `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_.


### Requirements

- `Python <https://www.python.org>`_ 3.6+
- `Postgres <https://www.postgresql.org>`_ 9.4+
- `Redis <https://redis.io>`_
- `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_ 6.3.1+

### Postgres setup

  Enable `logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>`_ in your 
  Postgres setting.

  - you would also need to set up two parameters in your Postgres config postgresql.conf

    ```wal_level = logical```

    ```max_replication_slots = 1```

### Installation

You can install PGSync from `PyPI <https://pypi.org>`_:

    $ pip install pgsync

### Config

Create a schema for the application named e.g **schema.json**

`Example schema <https://github.com/toluaina/pg-sync/blob/master/examples/airbnb/schema.json>`_

Example spec

.. code-block::

    [
        {
            "database": "[database name]",
            "index": "[elasticsearch index]",
            "nodes": [
                {
                    "table": "[table A]",
                    "schema": "[table A schema]",
                    "columns": [
                        "column 1 from table A",
                        "column 2 from table A",
                        ... additional columns
                    ],
                    "children": [
                        {
                            "table": "[table B with relationship to table A]",
                            "schema": "[table B schema]",
                            "columns": [
                              "column 1 from table B",
                              "column 2 from table B",
                              ... additional columns
                            ],
                            "relationship": {
                                "variant": "object",
                                "type": "one_to_many"
                            },
                            ...
                        },
                        {
                            ... any other additional children
                        }
                    ]
                }
            ]
        }
    ]

### Environment variables 

Setup required environment variables for the application

    SCHEMA='/path/to/schema.json'

    ELASTICSEARCH_HOST=localhost
    ELASTICSEARCH_PORT=9200

    PG_HOST=localhost
    PG_USER=i-am-root # this must be a postgres superuser
    PG_PORT=5432
    PG_PASSWORD=*****

    REDIS_HOST=redis
    REDIS_PORT=6379
    REDIS_DB=0
    REDIS_AUTH=*****


## Running

bootstrap the database (one time only)
  $ bootstrap --config schema.json
run pgsync as a daemon
  $ pgsync --config schema.json --daemon


=======
History
=======

1.0.1 (2020-15-01)
------------------

* First release on PyPI.


1.0.1 (2020-01-01)
------------------

* RC1 release


1.1.0 (2020-04-13)
------------------

* Postgres multi schema support for multi-tennant applications
* Show resulting Query with verbose mode
* this release required you to re-bootstrap your database with 

  * bootstrap -t
  * bootstrap


1.1.1 (2020-05-18)
------------------

* Fixed authentication with Redis
* Fixed Docker build


1.1.2 (2020-06-11)
------------------

* Sync multiple indices in the same schema
* Test for replication or superuser
* Fix PG_NOTIFY error when payload exceeds 8000 bytes limit


1.1.3 (2020-06-14)
------------------

* Bug fix when syncing multiple indices in the same schema


1.1.4 (2020-06-15)
------------------

* Only create triggers for tables present in schema


1.1.5 (2020-06-16)
------------------

* Bug fix when creating multiple triggers in same schema


1.1.6 (2020-07-31)
------------------

* Bug fix when tearing down secondary schema


1.1.7 (2020-08-16)
------------------

* Fix issue #29: SQLAlchemy err: Neither 'BooleanClauseList' object nor 'Comparator' object has an attribute '_orig'




