# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['elasticmagic_qf_attrs']

package_data = \
{'': ['*']}

install_requires = \
['elasticmagic>=0.1.0-beta.1,<0.2.0']

setup_kwargs = {
    'name': 'elasticmagic-qf-attrs',
    'version': '0.1.4',
    'description': '',
    'long_description': '[![Test and build](https://github.com/anti-social/elasticmagic-qf-attrs/actions/workflows/python.yaml/badge.svg)](https://github.com/anti-social/elasticmagic-qf-attrs/actions/workflows/python.yaml)\n\nElasticmagic query filters for attributes\n=========================================\n\nLibrary to store, filter and build facets for custom attributes\n\nThe problem\n-----------\n\nEach attribute pair can be stored in the index as a nested document.\nWe can use following mapping for that:\n\n```yaml\nattrs:\n  type: nested\n  properties:\n    attr_id:\n      type: integer\n    # usually only one of the next fields should be populated\n    value_int:\n      type: integer\n    value_bool:\n      type: boolean\n    value_float:\n      type: float\n```\n\nThis makes it possible to filter documents by an attribute id and its value\n(for example we want to find all the documents with `attr_id = 1234` and `value = 5678`):\n\n```yaml\nquery:\n  bool:\n    filter:\n    - nested:\n        path: attrs\n        query:\n          bool:\n            must:\n            - term:\n                attrs.attr_id: 1234\n            - term:\n                attrs.value_int: 5678\n```\n\nIt is also possible to build a facets for all attributes at once:\n\n```yaml\naggs:\n  attrs_nested:\n    nested:\n      path: attrs\n    aggs:\n      attrs:\n        terms:\n          field: attrs.attr_id\n        aggs:\n          values:\n            field: attrs.value_int\n```\n\nor for a single attribute:\n\n```yaml\naggs:\n  attrs_nested:\n    nested:\n      path: attrs\n    aggs:\n      attr_1234:\n        filter:\n          term:\n            attrs.attr_id: 1234\n        aggs:\n          values:\n            field: attrs.value_int\n```\n\nBut nested documents have some drawbacks. Every nested document is stored\nin the index as different document. For instance, next document will be stored\nas 5 lucene documents:\n\n```yaml\nname: "I\'m a document with nested attributes"\nattrs:\n- attr_id: 1\n  value_int: 42\n- attr_id: 2\n  value_int: 43\n- attr_id: 3\n  value_bool: true\n- attr_id: 4\n  value_float: 99.9\n```\n\nNested queries are slow by itself:\n   \n> In particular, joins should be avoided. nested can make queries several times\nslower and parent-child relations can make queries hundreds of times slower.\n> - https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-search-speed.html#_document_modeling\n\nBut what is worse regular queries are also slower when there are nested documents\nin the index. It is because of all the fields of main documents becomes sparse.\nThis in turn degrades performance of all filters and accesses to doc_values.\n\nThe solution\n------------\n\nThe idea is to encode pair of an attribute id and a corresponding value into\na single value. If our attribute ids are 32-bit integers and all value types\nalso fit into 32 bits we can store them as a single 64-bit value.\n\nSo our mapping can be:\n\n```yaml\nattrs:\n  type: object\n  properties:\n    int:\n      type: long\n    bool:\n      type: long\n    float:\n      type: long\n```\n\nDocument with encoded attributes:\n\n```yaml\nname: "I\'m a document with packed attributes"\nattrs:\n# (1 << 32) | 42\n- int: 0x1_0000002a\n# (2 << 32) | 43\n- int: 0x2_0000002b\n# (3 << 1) | 1\n- bool: 0x7\n# (4 << 32) | {integer representation of 99.9}\n# (4 << 32) | struct.unpack(\'=I\', struct.pack(\'=f\', 99.9))[0]\n- float: 0x4_42c7cccd\n```\n\nNow with a bit of bit magic we can emulate nested queries.\n\nFiltering by attribute id `1234` with value `5678`:\n\n```yaml\nquery:\n  bool:\n    filter:\n    - term:\n        attrs.int: 0x4d2_0000162e\n```\n\nBuilding facet for all attribute values:\n\n```yaml\naggs:\n  attrs_int:\n    terms:\n      field: attrs.int\n      # specify big enough aggregation size\n      # so all flat attrite values should fit\n      size: 10000\n```\n\nOne more step that client should do is to decode and group values by\nattribute id.\n\nHow to use it in python\n-----------------------\n\n```python\nfrom elasticsearch import Elasticsearch\n\nfrom elasticmagic import Cluster, Document, Field\nfrom elasticmagic.types import List, Long\nfrom elasticmagic.ext.queryfilter import QueryFilter\n\nfrom elasticmagic_qf_attrs import AttrBoolFacetFilter\nfrom elasticmagic_qf_attrs import AttrIntFacetFilter\nfrom elasticmagic_qf_attrs import AttrRangeFacetFilter\nfrom elasticmagic_qf_attrs.util import merge_attr_value_bool\nfrom elasticmagic_qf_attrs.util import merge_attr_value_float\nfrom elasticmagic_qf_attrs.util import merge_attr_value_int\n\n# Specify document\nclass AttrsDocument(Document):\n    __doc_type__ = \'attrs\'\n\n    ints = Field(List(Long))\n    bools = Field(List(Long))\n    floats = Field(List(Long))\n\n# Create an index\nindex_name = \'test-attrs\'\nclient = Elasticsearch()\nclient.indices.create(index=index_name)\ncluster = Cluster(client)\nindex = cluster.get_index(index_name)\nindex.put_mapping(AttrsDocument)\n\n# Index example document\nindex.add([\n    AttrsDocument(\n        ints=[\n            merge_attr_value_int(1, 42),\n            merge_attr_value_int(2, 43),\n        ],\n        bools=[merge_attr_value_bool(3, True)],\n        floats=[merge_attr_value_float(4, 99.9)],\n    ),\n], refresh=True)\n\n# Define a query filter\nclass AttrsQueryFilter(QueryFilter):\n    ints = AttrIntFacetFilter(AttrsDocument.ints, alias=\'a\')\n    bools = AttrBoolFacetFilter(AttrsDocument.bools, alias=\'a\')\n    ranges = AttrRangeFacetFilter(AttrsDocument.floats, alias=\'a\')\n\n# Now we can build facets\nqf = AttrsQueryFilter()\nsq = index.search_query()\nsq = qf.apply(sq, {})\nres = sq.get_result()\nassert res.total == 1\nqf_res = qf.process_result(res)\n\n# And finally lets print results\nfor attr_id, facet in qf_res.ints.facets.items():\n    print(f\'> {attr_id}:\')\n    for facet_value in facet.all_values:\n        print(f\'  {facet_value.value}: ({facet_value.count_text})\')\n\nfor attr_id, facet in qf_res.bools.facets.items():\n    print(f\'> {attr_id}:\')\n    for facet_value in facet.all_values:\n        print(f\'  {facet_value.value}: ({facet_value.count_text})\')\n\nfor attr_id, facet in qf_res.ranges.facets.items():\n    print(f\'> {attr_id}: ({facet.count})\')\n\n# Also we can filter documents:\nqf = AttrsQueryFilter()\nsq = index.search_query()\nsq = qf.apply(\n    sq,\n    {\n        \'a1\': \'42\',\n        \'a3\': \'true\',\n        \'a4__lte\': \'100\',\n    }\n)\nres = sq.get_result()\nassert res.total == 1\n\nqf = AttrsQueryFilter()\nsq = index.search_query()\nsq = qf.apply(\n    sq,\n    {\n        \'a4__gte\': \'100\',\n    }\n)\nres = sq.get_result()\nassert res.total == 0\n```\n\nThis script should print:\n\n```\n> 1:\n  42: (1)\n> 2:\n  43: (1)\n> 3:\n  True: (1)\n> 4: (1)\n```\n',
    'author': 'Alexander Koval',
    'author_email': 'kovalidis@gmail.com',
    'maintainer': 'Alexander Koval',
    'maintainer_email': 'kovalidis@gmail.com',
    'url': 'https://github.com/anti-social/elasticmagic-qf-attrs',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'python_requires': '>=3.7,<4.0',
}


setup(**setup_kwargs)
