Metadata-Version: 1.1
Name: mergic
Version: 0.0.5
Summary: workflow support for reproducible deduplication and merging
Home-page: https://github.com/ajschumacher/mergic
Author: Aaron Schumacher
Author-email: ajschumacher@gmail.com
License: MIT
Download-URL: https://github.com/ajschumacher/mergic/tarball/0.0.5
Description: ======
        mergic
        ======
        -----------------------------------------------------------
        workflow support for reproducible deduplication and merging
        -----------------------------------------------------------
        
        Say you have a bunch of strings, and some of them are different but refer to the same thing. Maybe it's just some long list, maybe it's the contents of two key columns from data sets that you'd like to merge.
        
        ::
        
            David Copperfield
            Lance Burton
            Dave Copperfield
            Levar Burton
        
        Here's what you can do with ``mergic``:
        
        Give ``mergic`` all your identifiers, one per line. If they are in a file called ``originals.txt``:
        
        .. code:: bash
        
           mergic calc originals.txt
        
        You will see output about the possible groupings ``mergic`` can produce based on its default distance function. (It's easy to use a custom distance function—see below.)
        
        ::
        
            num groups, max group, num pairs, cutoff
            ----------------------------------------
                     4,         1,         0, -0.909090909091
                     3,         2,         1, 0.0909090909091
                     2,         2,         2, 0.25
                     1,         4,         6, 0.714285714286
        
        There are 4 groupings that ``mergic`` can make with the given distance function, ranging from a group for every name to a single group for all four names.
        
        The ``num groups`` is the number of groups that ``mergic`` can make with the given ``cutoff``.
        
        The ``max group`` is the number of items in the largest group for that grouping.
        
        The ``num pairs`` column indicates the number of within-group links that correspond to the grouping. In some sense this represents how much work it is to check all the linkages that ``mergic`` is making. Thinking in groups is much better than thinking in individual pairwise comparisons, but the metric is useful for summarizing how much linking is happening.
        
        The ``cutoff`` determines which pairs of items are put in the same group. If the distance between two items is equal to or less than the ``cutoff``, those items will be grouped together.
        
        Select a cutoff to produce the grouping you would like to see. If you would like to use the cutoff 0.3 and put the results in a file called ``grouping.json``:
        
        .. code:: bash
        
           mergic make originals.txt 0.3 > grouping.json
        
        Now ``grouping.json`` contains a grouping of your original data, specified in `JSON <http://www.json.org/>`__. It's designed to be human-readable, and human-editable, so you can check it and improve it as needed. The largest groups will be at the top, and similar items will be near one another.
        
        ::
        
            {
                "Lance Burton": [
                    "Lance Burton",
                    "Levar Burton"
                ],
                "David Copperfield": [
                    "Dave Copperfield",
                    "David Copperfield"
                ]
            }
        
        There are two proposed groups, with proposed names "Lance Burton" and "David Copperfield". You probably want to copy the produced file and edit the copy.
        
        .. code:: bash
        
           cp grouping.json grouping_fixed.json
           # edit `grouping_fixed.json`
        
        You can edit the proposed group names—the keys of the object—if you like. The original values from the data are in the array values of the object, and you can move them around to correct the grouping. In this case you could also re-run ``mergic`` with a cutoff of 0.1 to get a correct grouping:
        
        ::
        
            {
                "David Copperfield": [
                    "Dave Copperfield",
                    "David Copperfield"
                ],
                "Lance Burton": [
                    "Lance Burton"
                ],
                "Levar Burton": [
                    "Levar Burton"
                ]
            }
        
        Now that ``grouping_fixed.json`` is as perfect as it can be, you can move forward.
        
        You can also compare your two JSON grouping files and see what you changed:
        
        .. code:: bash
        
           mergic diff grouping.json grouping_fixed.json > diff.json
        
        Now the file ``diff.json`` contains just what's different between ``grouping.json`` and ``grouping_fixed.json``. The ``mergic diff`` command is analogous to regular ``diff`` for text files, but it is aware of the JSON partition format so it can capture changes intelligently.
        
        You can apply changes to a ``mergic``-produced file to recover your edited version.
        
        .. code:: bash
           mergic apply grouping.json diff.json > grouping_new.json
        
        Now ``grouping_new.json`` is equivalent to ``grouping_fixed.json``, as you can verify:
        
        .. code:: bash
           mergic diff grouping_fixed.json grouping_new.json
           # {}  // (no changes)
        
        In this way you have a complete and verifiable record of your work, at the level of whole files and also at the level of changes made by hand.
        
        The JSON grouping format is very convenient for humans, but for tabular data a merge table is more useful. A merge table has one column with the original values from your data and one column with the new keys. These are named ``original`` and ``mergic`` in the output:
        
        .. code:: bash
        
           mergic table grouping_fixed.json > merge_table.csv
        
        The file ``merge_table.csv`` looks like this:
        
        ::
        
            original,mergic
            Lance Burton,Lance Burton
            Levar Burton,Levar Burton
            Dave Copperfield,David Copperfield
            David Copperfield,David Copperfield
        
        This merge table can now be used with any tabular data system. For merges, first merge it on to both tables and then merge by the ``mergic`` key. For deduplication, merge it on to the table(s) of interest and then use the ``mergic`` column as you would have used the original data.
        
        
        Installation
        ============
        
        .. code:: bash
        
           pip install mergic
        
        
        Using a Custom Distance Function
        ================================
        
        The ``mergic`` package provides a command-line script called ``mergic`` that uses Python's built-in ``difflib.SequenceMatcher.ratio()`` for calculating string distances, but a major strength of ``mergic`` is that it enables easy customization of the distance function via the ``mergic.Blender`` class. Making a custom ``mergic`` script is as easy as:
        
        .. code:: python
        
           #!/usr/bin/env python
           # custom_mergic.py
           import mergic
        
           # Any custom distance you want to try! e.g.,
           def my_distance(a, b):
               return abs(len(a) - len(b))
        
           mergic.Blender(my_distance).script()
        
        Now ``custom_mergic.py`` can be used just like the standard ``mergic`` script!
        
        There is also a function that determines how to make a key for a group. It takes a list and returns a string. By default ``mergic.Blender`` will use the longest of a group's values (the first longest, in stable sorted order). You can also supply a custom function for generating keys.
        
        
        Distances You Might Like
        ------------------------
        
        `Levenshtein string edit distance <http://en.wikipedia.org/wiki/Levenshtein_distance>`__: The classic! It has many implementations; one of them is `python-Levenshtein <http://www.coli.uni-saarland.de/courses/LT1/2011/slides/Python-Levenshtein.html>`__.
        
        .. code:: python
        
            # pip install python-Levenshtein
            import Levenshtein
            Levenshtein.distance("fuzzy", "wuzzy")
            # 1
        
        SeatGeek's `fuzzywuzzy <https://github.com/seatgeek/fuzzywuzzy>`__: As described in their `blog post <http://chairnerd.seatgeek.com/fuzzywuzzy-fuzzy-string-matching-in-python/>`__, some people have found these variants to work well in practice. Responses from ``fuzzywuzzy`` are phrased as integer percent similarities; one way to make a distance is to subtract from 100.
        
        .. code:: python
        
            # pip install fuzzywuzzy
            from fuzzywuzzy import fuzz
            100 - fuzz.ratio("Levensthein", "Leviathan")
            # 50
        
        There are a ton of distances, even just within the two packages mentioned!
        
        You can also make your own!
        
Platform: UNKNOWN
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.7
Classifier: License :: OSI Approved :: MIT License
Classifier: Development Status :: 2 - Pre-Alpha
