# -*- coding: utf-8 -*-
from setuptools import setup

packages = \
['jsonsam']

package_data = \
{'': ['*']}

install_requires = \
['Faker>=8.1,<9.0', 'deepdiff>=5.5,<6.0']

entry_points = \
{'console_scripts': ['jsonsam = jsonsam.jsonsam:main',
                     'randdict = jsonsam.randdict:main']}

setup_kwargs = {
    'name': 'jsonsam',
    'version': '0.1.1',
    'description': 'Tool to facilitate editing, splitting and merging of nested JSON and Python dictionaries',
    'long_description': '# JSONSAM Overview\n\n## Quickstart\n\nThe following is a minimal example of the JSONSAM CLI workflow:\n\n```console\n$ jsonsam my.json\nMake edits to my-denorm.json and then rerun with modified file to generate updated JSON output\n$ vim my-denorm.json \t\t# Add/delete/edit line-based paths\n$ jsonsam my-denorm.json \t# Convert edits back to standard JSON\nUpdated JSON file written to my-denorm-norm.json\n```\n\nThe finished product (the standard JSON file with edits applied) is\n`my-denorm-norm.json`. See the [Command Line Examples](#command-line-examples)\nsection for extensive additional examples.\n\n## Introduction\n\nJSONSAM captures some of the techniques I\'ve used over the years to slice and\ndice large chunks (> 1MB) of highly nested JSON.  vim is still my preferred\neditor and I like to use line-based shell primitives like sed and grep when\npossible.  However, since conventional JSON is not line-framed and\nJSON-streaming has multipath semantics, it is generally not amenable to direct\nmodification using conventional Unix utilities.\n\nJSONSAM addresses this by converting JSON into an intermediate denormalized\nformat (not to be confused with JSON-streaming or new-line delimited JSON).\nThe denormalized format captures each path in a JSON list of line-delimited\npaths.  These paths can then be edited with standard tools to add, delete,\nmodify, shift-up/down, reorder nested lists, etc.  Once desired edits are made,\nthe denormalized file is renormalized back into a standard JSON file.  JSONSAM\ncan also work on entire directories of files at once, allowing them to be\neasily merged, split and modified.  Denormalized form might also be useful for\nstreaming if one is willing to trade data expansion for framing.\n\nJSONSAM can also perform set arithmetic (union, intersection and difference)\nagainst a pair of JSON input files.  This is useful when a template or\nreference file exists and it\'s desired to extract, remove or augment another\nJSON file based on this template.\n\nLastly, a random realistic dictionary generator is included as part of the test\ninfrastructure. It creates nested dictionaries and lists of random breadth and\nwidth.\n\nJSONSAM comprises the following components:\n\n- jsonsam: Command line utility for processing JSON\n- randdict: Command line utility for generating random dictionaries\n- JsonSam: Denormalize, edit, normalize and perform set arithmetic on JSON files\n- DictSam: Denormalize, edit, normalize and perform set arithmetic on nested dictionaries/lists\n- DictGen: Generate random deeply nested dictionaries and lists with realistic data\n\n## Installation\n\n### Install from PyPi:\n\n`pip install jsonsam`\n\n### Importing\n\n```python\n>>> from jsonsam import DictSam # Dictionary denorm, norm, and set arithmetic\n>>> from jsonsam import DictGen # Random realistic nested dictionary/list generator\n```\n\n# Command Line Examples\n\n```console\n# online help\n$ jsonsam -h\n```\n\nConsider the following file `simple.json`:\n```json\n$ cat simple.json\n{\n  "Bad": true,\n  "beer": 3.14,\n  "rots": "Our",\n  "young": {\n    "guts": "but",\n    "Vodka": [\n      27,\n      [\n        "goes",\n        2.718,\n        {\n          "Well": "water",\n          "max": "plank"\n        },\n        "old"\n      ]\n    ],\n    "bark": null,\n    "last": 66\n  }\n}\n```\nThis file is in normalized form.  First, we denormalize the file:\n\n```console\n$ jsonsam simple.json\nMake edits to simple-denorm.json and then rerun with modified file to generate updated JSON output\n```\n\nThe denormalized file, `simple-denorm.json`, now contains a list of paths.\nEach line represents a path from root to leaf through the original JSON file\nwith non-terminal integers representing list indices:\n\n```json\n$ cat simple-denorm.json\n["Bad", true]\n["beer", 3.14]\n["rots", "Our"]\n["young", "Vodka", 0, 27]\n["young", "Vodka", 1, 0, "goes"]\n["young", "Vodka", 1, 1, 2.718]\n["young", "Vodka", 1, 2, "Well", "water"]\n["young", "Vodka", 1, 2, "max", "plank"]\n["young", "Vodka", 1, 3, "old"]\n["young", "bark", null]\n["young", "guts", "but"]\n["young", "last", 66]\n```\n\nAll other rows can be arbitrarily mutated, added, removed and reordered\n(non-terminal integers denote nested list indices).  When the desired result is\nachieved, rerun `jsonsam` with the edited `simple-denorm.json` as input to\ngenerate a renormalized `simple-denorm-norm.json` output file.  This represents\nthe original JSON file with the desired additions, deletions and mutations\napplied.  Note that JSONSAM appends `-norm` and `-denorm` strings to the file\nstem for convenience to avoid overwriting original files and track lineage.\nThis behavior can be overridden with the `-o <outputfile>` option.\n\n**Unless otherwise noted all examples that follow will start from `simple.json`\nand `simple-denorm.json`.**\n\n## Relocate dictionary slice from nested lists\nSuppose we want to shift-up the pair `"Well": "water"` out of the dictionary\nnested in the list of lists.  This can be achieved by applying the following\nsingle-line diff to `simple-denorm.json`:\n\n```diff\n ["young", "Vodka", 0, 27]\n ["young", "Vodka", 1, 0, "goes"]\n ["young", "Vodka", 1, 1, 2.718]\n-["young", "Vodka", 1, 2, "Well", "water"]\n+["young", "Vodka", 2, "Well", "water"]\n ["young", "Vodka", 1, 2, "max", "plank"]\n ["young", "Vodka", 1, 3, "old"]\n ["young", "bark", null]\n```\n\nTo apply the changes, we normalize the path list by rerunning JSONSAM:\n\n```diff\n$ jsonsam simple-denorm.json \nUpdated JSON file written to simple-denorm-norm.json\n$ diff --unified=20 <(cat simple.json |jq --sort-keys \'.\') <(cat simple-denorm-norm.json |jq --sort-keys \'.\')\n {\n   "Bad": true,\n   "beer": 3.14,\n   "rots": "Our",\n   "young": {\n     "Vodka": [\n       27,\n       [\n         "goes",\n         2.718,\n         {\n-          "Well": "water",\n           "max": "plank"\n         },\n         "old"\n-      ]\n+      ],\n+      {\n+        "Well": "water"\n+      }\n     ],\n     "bark": null,\n     "guts": "but",\n     "last": 66\n   }\n }\n```\n\nWe see in the above JSON that indeed the `"Well": "water"` pair has been\nplucked from the nested dictionary, and a new dictionary containing the pair\nhas been appended to the outer list.  `simple-denorm-norm.json` now represents\nthe finished product.\n\n## Click to expand additional examples:\n\n<details><summary>Remove an item, and convert a leaf string to a dictionary</summary>\n\n## Remove an item, and convert a leaf string to a dictionary\n\nIn this case we do two things:\n\n1. Remove the `young.bark.null` path\n1. Convert `young.guts.but` path from a terminal string to a dictionary of\n`young.guts.{but:dog, route:cat}`\n\nThis can be achieved by applying the following diff to `simple-denorm.json`:\n\n```json\n ["young", "Vodka", 1, 2, "Well", "water"]\n ["young", "Vodka", 1, 2, "max", "plank"]\n ["young", "Vodka", 1, 3, "old"]\n-["young", "bark", null]\n-["young", "guts", "but"]\n+["young", "guts", "but", "dog"]\n+["young", "guts", "route", "cat"]\n ["young", "last", 66]\n```\n\nLooking at `simple-denorm-norm.json` we see `young.bark.null` has been removed\nand we\'ve converted `young.guts.but` from a terminal string into a dictionary\nof `{but:dog, route:cat}`:\n\n```json\n$ jsonsam simple-denorm.json\nUpdated JSON file written to simple-denorm-norm.json\n$ cat simple-denorm-norm.json\n{\n  "Bad": true,\n  "beer": 3.14,\n  "rots": "Our",\n  "young": {\n    "Vodka": [\n      27,\n      [\n        "goes",\n        2.718,\n        {\n          "Well": "water",\n          "max": "plank"\n        },\n        "old"\n      ]\n    ],\n    "guts": {\n      "but": "dog",\n      "route": "cat"\n    },\n    "last": 66\n  }\n}\n```\n</details>\n\n<details><summary>Convert a terminal string to a list of dictionaries</summary>\n\n## Convert a terminal string to a list of dictionaries\n\nHere we convert the `young.guts` path to a list of dictionaries.\nThis can be achieved by applying the following diff to `simple-denorm.json`:\n\n```json\n ["young", "Vodka", 1, 2, "max", "plank"]\n ["young", "Vodka", 1, 3, "old"]\n ["young", "bark", null]\n-["young", "guts", "but"]\n+["young", "guts", 0, "but", "dog"]\n+["young", "guts", 1, "route", "dog"]\n ["young", "last", 66]\n```\n\nAnd the resulting normalized JSON:\n\n```json\n$ jsonsam simple-denorm.json\nUpdated JSON file written to simple-denorm-norm.json\n$ cat simple-denorm-norm.json\n{\n  "Bad": true,\n  "beer": 3.14,\n  "rots": "Our",\n  "young": {\n    "Vodka": [\n      27,\n      [\n        "goes",\n        2.718,\n        {\n          "Well": "water",\n          "max": "plank"\n        },\n        "old"\n      ]\n    ],\n    "bark": null,\n    "guts": [\n      {\n        "but": "dog"\n      },\n      {\n        "route": "dog"\n      }\n    ],\n    "last": 66\n  }\n}\n```\n\n</details>\n\n\n<details><summary>Add path and reorder a list</summary>\n\n### Add path and reorder a list\n\nThree tasks here:\n1. Add a new `jack.beanstock.happy.941` path\n1. Move list item `young.Vodka.1.2` to `young.Vodka.2`\n1. Add the string "append" to `young.Vodka` list\n\n\nThis can be achieved by applying the following diff to `simple-denorm.json`:\n\n```json\n ["Bad", true]\n ["beer", 3.14]\n ["rots", "Our"]\n+["jack", "beanstock", "happy", 941]\n ["young", "Vodka", 0, 27]\n-["young", "Vodka", 1, 0, "goes"]\n-["young", "Vodka", 1, 1, 2.718]\n-["young", "Vodka", 1, 2, "Well", "water"]\n-["young", "Vodka", 1, 2, "max", "plank"]\n+["young", "Vodka", 1, 1, "goes"]\n+["young", "Vodka", 1, 0, 2.718]\n+["young", "Vodka", 2, "Well", "water"]\n+["young", "Vodka", 2, "max", "plank"]\n+["young", "Vodka", 3, "appended"]\n ["young", "Vodka", 1, 3, "old"]\n ["young", "bark", null]\n ["young", "guts", "but"]\n```\n\nWhich results in:\n\n```json\n$ jsonsam simple-denorm.json\nUpdated JSON file written to simple-denorm-norm.json\n$ cat simple-denorm-norm.json\n{\n  "Bad": true,\n  "beer": 3.14,\n  "jack": {\n    "beanstock": {\n      "happy": 941\n    }\n  },\n  "rots": "Our",\n  "young": {\n    "Vodka": [\n      27,\n      [\n        2.718,\n        "goes",\n        null,\n        "old"\n      ],\n      {\n        "Well": "water",\n        "max": "plank"\n      },\n      "appended"\n    ],\n    "bark": null,\n    "guts": "but",\n    "last": 66\n  }\n}\n```\n\nRemoved list elements become `null` to preserve the ordinal of retained\nelements.\n</details>\n\n<details><summary>Unix pipeline processing</summary>\n\n## Unix pipeline processing\n\nIf no input files are provided, jsonsam defaults to reading from stdin and\nwriting to stdout.  This allows use of standard Unix pipelines for editing JSON\nfiles with line-based shell utilities like sed, grep, awk, etc.\n\nIn the below example, we use grep to remove all `Vodka` keys and sed to convert `beer` to `Whiskey`:\n\n```json\n$ cat simple.json |jsonsam |grep -v Vodka |sed \'s/beer/Whiskey/g\' |jsonsam\n{\n  "Bad": true,\n  "Whiskey": 3.14,\n  "rots": "Our",\n  "young": {\n    "bark": null,\n    "guts": "but",\n    "last": 66\n  }\n}\n```\n\n</details>\n\n<details><summary>Transmogrify 7 JSON files into 11 JSON files, round-robin paths</summary>\n\n### Transmogrify 7 JSON files into 11 JSON files, round-robin paths\n\nIn this example, we will generate 7 random JSON files.  Next we will merge them\nand then split the merge into 11 files, round-robinning the paths across the 11\nnew JSON files:\n\n```console\n$ randdict -B5 -D5 -c 7 -o data.json\n$ ls\ndata000.json  data001.json  data002.json  data003.json  data004.json  data005.json  data006.json\n$ jsonsam *.json - |split -xn r/11\n$ ls\ndata000.json  data001.json  data002.json  data003.json  data004.json  data005.json  data006.json  x00  x01  x02  x03  x04  x05  x06  x07  x08  x09  x0a\n$ ls x* |xargs -IF jsonsam F\nUpdated JSON file written to x00-norm.json\nUpdated JSON file written to x01-norm.json\nUpdated JSON file written to x02-norm.json\nUpdated JSON file written to x03-norm.json\nUpdated JSON file written to x04-norm.json\nUpdated JSON file written to x05-norm.json\nUpdated JSON file written to x06-norm.json\nUpdated JSON file written to x07-norm.json\nUpdated JSON file written to x08-norm.json\nUpdated JSON file written to x09-norm.json\nUpdated JSON file written to x0a-norm.json\n```\n\nThe 11 `x<num>-norm.json` files now contain all the data in the original 7\nfiles, but split round-robin across 11 files.  Substituting `split -xn l/11`\nfor the split command would instead do a conventional partition rather than\nround-robin.  Of course any intermediate mutations, additions and deletions\ncould also be made in the same pipeline.\n\n</details>\n\n## Comand Line Set Operations Examples\n\nSet operations union `|`, intersection `&` and difference `-` can be performed\non JSON files specified with positional arguments (lhs) and `-F` (rhs) options.\nThe following examples will use the original `simple.json` file as the\nleft-hand operand and `simple_aux.json` as the right-hand operand:\n\n```json\n$ cat simple_aux.json\n{\n  "Bad": true,\n  "beer": 3.1415,\n  "NEW": "PATH",\n  "young": {\n    "Vodka": [\n      27\n    ],\n    "bark": null,\n    "guts": "but",\n    "last": 66\n  }\n}\n```\n\n## Difference/except `simple.json - simple_aux.json`\n\nIn the following example we perform the set difference of `simple.json -\nsimple_aux.json`.  Paths common to both `simple.json` and `simple_aux.json` are\nremoved from `simple.json`.\n\nNote the `-I` option can also be added for all operations to ignore leaf values\nin comparison.  This is useful when one wishes to use an existing JSON file as\na template for paths to remove regardless of value.\n\nAlso note removed list elements become `null` to preserve the ordinal of\nretained elements.\n\n```diff\n$ jsonsam simple.json -F simple_aux.json -e\nUpdated JSON file written to simple-norm.json\n$ diff --unified=20 <(cat simple.json |jq --sort-keys \'.\') <(cat simple-norm.json |jq --sort-keys \'.\')\n {\n-  "Bad": true,\n   "beer": 3.14,\n   "rots": "Our",\n   "young": {\n     "Vodka": [\n-      27,\n+      null,\n       [\n         "goes",\n         2.718,\n         {\n           "Well": "water",\n           "max": "plank"\n         },\n         "old"\n       ]\n-    ],\n-    "bark": null,\n-    "guts": "but",\n-    "last": 66\n+    ]\n   }\n }\n```\n\n## Click to expand additional set examples:\n\n<details><summary>Intersection (simple.json & simple_aux.json)</summary>\n\n## Intersection `(simple.json & simple_aux.json)`\n\nIn this example we perform the intersection of the two JSON files.  Note that\n`-I` is passed, so the leaf values are ignored allowing us to also intersect\nwith a slightly less precise value of pi.\n\n```json\n$ jsonsam simple.json -F simple_aux.json -iI\nUpdated JSON file written to simple-norm.json\n$ cat simple-norm.json\n{\n  "Bad": true,\n  "beer": 3.14,\n  "young": {\n    "Vodka": [\n      27\n    ],\n    "bark": null,\n    "guts": "but",\n    "last": 66\n  }\n}\n```\n\n</details>\n\n<details><summary> Union/merge (simple.json | simple_aux.json)</summary>\n\n## Union/merge `(simple.json | simple_aux.json)`\n\nIn this example we perform the union of the two JSON files.  Note the\nconflicting `beer` key is overwritten by the right-hand side\'s value and the\n`NEW` item has been added.\n\n```json\n$ jsonsam simple.json -F simple_aux.json -u\nUpdated JSON file written to simple-norm.json\n$ cat simple-norm.json\n{\n  "Bad": true,\n  "beer": 3.1415,\n  "rots": "Our",\n  "young": {\n    "Vodka": [\n      27,\n      [\n        "goes",\n        2.718,\n        {\n          "Well": "water",\n          "max": "plank"\n        },\n        "old"\n      ]\n    ],\n    "bark": null,\n    "guts": "but",\n    "last": 66\n  },\n  "NEW": "PATH"\n}\n```\n\n</details>\n\n## Random nested dictionary generator examples\n\nA random dictionary generator is included in the package.  It generates deeply\nnested dictionaries of realistic random data.  Random nested lists are also\nsupported.  The full range of JSON types are supported.  Each step in breadth\nand depth is uniquely randomized.\n\nTo generate a random dictionary of max breadth 3, max depth 4 consisting of 40%\nnested lists (compact form for brevity):\n\n```json\n$ randdict -B 3 -D 4 -l 40 |jq -c \'.\'\n{"game":{"yourself":{"difficult":"0181 Aguilar Parkways","always":{"before":530982.9961689024,"leader":{"kid":0.41913904357146525,"available":523048.62318369857},"effect":-609888.9269447384}},"believe":[[{"ok":0.24842658485754932,"kind":"91390 Williams Forges Apt. 824","charge":0.7294452894392176},"482 Bonnie Route",{"cup":null,"hotel":-292649,"bill":0.6952953662736593}],[0.47409833741964447,"093 Becker Meadow"]]},"standard":[{"former":null,"make":270989.99712404597,"pay":245327},{"couple":"Ward-Wright","say":{"state":["Miller LLC","Simpson, Cooper and Cole","Baird, Wilson and Barnes"],"though":"76018 Peterson Keys Suite 971","better":0.28193072232673766},"you":{"expect":null,"matter":{"hear":"49746 Johnson Mountain","artist":"Carr Group","interest":"2366 Miller Mission"}}},{"often":{"reason":"68465 Rosario Drive Apt. 223","early":{"actually":null,"memory":null,"theory":null},"nor":"White, Campbell and Thomas"},"hope":{"poor":2548096.1722068526,"others":372384,"from":["Vaughn PLC","Pena, Singh and Bryant",322924]}}],"truth":[{"state":{"two":{"response":0.08053812548862638,"success":null,"protect":"Long, Ross and Garcia"},"institution":null,"assume":1888192},"cut":[null,{"audience":null,"grow":"Alexander Inc","blood":"Williams, Smith and Hernandez"},1358976.5609615073],"himself":["39624 Guzman Mountains",-2605707.399841271]},["Wilson Group",0.2127797923458118,{"man":{"media":null,"member":725008.2758262581},"forget":1022493.1525128945,"guess":"Thomas Ltd"}]]}\n```\n\n# API Example\n\nThe following example shows how to use the API to generate a random test\ndictionary, pick two overlapping subsets of the dictionary, then take the\ndifference, union and intersection of these dictionaries:\n\n```python\n>>> from jsonsam import DictSam\n>>> from jsonsam import DictGen\n>>> test_dict = DictGen().gen_fake_dict(breadth_rng=(2, 4), depth_rng=(3, 5), list_dist=(3, 1))\n>>> dict_sam_orig = DictSam(test_dict)\n>>> dict_sam_a = dict_sam_orig.random_dict_pick(60)\n>>> dict_sam_b = dict_sam_orig.random_dict_pick(80)\n>>> diff = dict_sam_b - dict_sam_a\n>>> union = dict_sam_b | dict_sam_a\n>>> intersect = dict_sam_b & dict_sam_a\n```\n\n**NOTE: At the moment only JSON-serializable structures are supported by DictSam**\n\nFull API readthedocs coming soon...\n\n# Authors\n\nSee [AUTHORS](AUTHORS.md)\n',
    'author': 'Eric D. Cohen',
    'author_email': 'os@epieye.com',
    'maintainer': None,
    'maintainer_email': None,
    'url': 'https://github.com/project705/jsonsam',
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
    'entry_points': entry_points,
    'python_requires': '>=3.8,<4.0',
}


setup(**setup_kwargs)
