Metadata-Version: 1.0
Name: z3c.vcsync
Version: 0.16
Summary: Synchronize object data with a version control system
Home-page: UNKNOWN
Author: Martijn Faassen
Author-email: faassen@startifact.com
License: ZPL 2.1
Description: Version Control Synchronization
        ===============================
        
        This package contains code that helps with handling synchronization of
        persistent content with a version control system.
        
        This can be useful in software that needs to be able to work
        offline. The web application runs on a user's laptop that may be away
        from an internet connection. When connected again, the user syncs with
        a version control server, receiving updates that may have been made by
        others, and committing their own changes.
        
        Another advantage is that the version control system always contains a
        history of how content developed over time. The version-control based
        content can also be used for other purposes independent of the
        application.
        
        While this package has been written with other version control systems
        in mind, it has only been developed to work with SVN so far. Examples
        below are all given with SVN.
        
        The synchronization sequence is as follows:
        
        1) save persistent state (IState) to svn checkout (ICheckout) on the
        same machine as the Zope application.
        
        2) ``svn up``. Subversion merges in changed made by others users that
        were checked into the svn server.
        
        3) Any svn conflicts are automatically resolved.
        
        4) reload changes in svn checkout into persistent Python objects
        
        5) ``svn commit``.
        
        This is all happening in a single step. It can happen over and over
        again in a reasonably safe manner, as after the synchronization has
        concluded, the state of the persistent objects and that of the local
        SVN checkout will always be in sync.
        
        During synchronisation, the system tries to take care only to
        synchronize those objects and files that have changed. That is, in
        step 1) only applies those objects that have been modified, added or
        removed will have an effect on the checkout. In step 4) only those
        files that have been changed, added or removed on the filesystem due
        to the ``up`` action will change the persistent object state.
        
        State
        -----
        
        Content to synchronize is represented by an object that provides
        ``IState``. A state represents a container object, which should
        contain a ``data`` object (a container that contains the actual data
        to be synchronized) and a ``found`` object (a container that contains
        objects that would otherwise be lost during conflict resolution).
        
        The following methods need to be implemented:
        
        * ``get_revision_nr()``: return the last revision number that the
        application was synchronized with. The state typically stores this
        the application object.
        
        * ``set_revision_nr(nr)``: store the last revision number that the
        application was synchronized with.
        
        * ``objects(revision_nr)``: any object that has been modified (or
        added) since the synchronization for ``revision_nr``. Returning 'too
        many' objects (objects that weren't modified) is safe, though less
        efficient as they will then be re-exported.
        
        Typically in your application this would be implemented by doing
        a catalog search, so that they can be looked up quickly.
        
        * ``removed(revision_nr)``: any path that has had an object removed
        from it since revision_nr.  It is safe to return paths that have
        been removed and have since been replaced by a different object with
        the same name. It is also safe to return 'too many' paths, though
        less efficient as the objects in these paths may be re-exported
        unnecessarily.
        
        Typically in your application you would maintain a list of removed
        objects by hooking into ``IObjectMovedEvent`` and
        ``IObjectRemovedEvent`` and recording the paths of all objects that
        were moved or removed. After an export it is safe to purge this
        list.
        
        In this example, we will use a simpler, less efficient, implementation
        that goes through a content to find changes. It tracks the
        revision number as a special attribute of the root object::
        
        >>> from z3c.vcsync.tests import TestState
        
        The content
        -----------
        
        Now that we have something that can synchronize a tree of content in
        containers, let's actually build ourselves a tree of content.
        
        An item contains some payload data, and maintains the SVN revision
        after which it was changed. In a real application you would typically
        maintain the revision number of objects by using an annotation and
        listening to ``IObjectModifiedEvent``, but we will use a property
        here::
        
        >>> from z3c.vcsync.tests import Item
        
        This code needs a ``get_revision_nr`` method available to get access
        to the revision number of last synchronization. For now we'll just define
        this to return 0, but we will change this later::
        
        >>> def get_revision_nr(self):
        ...    return 0
        >>> Item.get_revision_nr = get_revision_nr
        
        Besides the ``Item`` class, we also have a ``Container`` class::
        
        >>> from z3c.vcsync.tests import Container
        
        It is a class that implements enough of the dictionary API and
        implements the ``IContainer`` interface. A normal Zope 3 folder or
        Grok container will also work.
        
        Let's create a container now::
        
        >>> root = Container()
        >>> root.__name__ = 'root'
        
        The container has two subcontainers (``data`` and ``found``).
        
        >>> root['data'] = data = Container()
        >>> root['found'] = Container()
        >>> data['foo'] = Item(payload=1)
        >>> data['bar'] = Item(payload=2)
        >>> data['sub'] = Container()
        >>> data['sub']['qux'] = Item(payload=3)
        
        As part of the synchronization procedure we need the ability to export
        persistent python objects to the version control checkout directory in
        the form of files and directories.
        
        Now that we have an implementation of ``IState`` that works for our
        state, let's create our ``state`` object::
        
        >>> state = TestState(root)
        
        Reading from and writing to the filesystem
        ------------------------------------------
        
        To integrate with the synchronization machinery, we need a way to dump
        a Python object to the filesystem (to an SVN working copy), and to
        parse it back to an object again.
        
        Let's grok this package first, as it provides some of the required
        infrastructure::
        
        >>> import grok.testing
        >>> grok.testing.grok('z3c.vcsync')
        
        We need to provide a serializer for the Item class that takes an item
        and writes it to the filesystem to a file with a particular extension
        (``.test``)::
        
        >>> from z3c.vcsync.tests import ItemSerializer
        
        We also need to provide a parser to load an object from the filesystem
        back into Python, overwriting the previously existing object::
        
        >>> from z3c.vcsync.tests import ItemParser
        
        Sometimes there is no previously existing object in the Python tree,
        and we need to add it. To do this we implement a factory (where we use
        the parser for the real work)::
        
        >>> from z3c.vcsync.tests import ItemFactory
        
        Both parser and factory are registered per extension, in this case
        ``.test``. This is the name of the utility.
        
        We register these components::
        
        >>> grok.testing.grok_component('ItemSerializer', ItemSerializer)
        True
        >>> grok.testing.grok_component('ItemParser', ItemParser)
        True
        >>> grok.testing.grok_component('ItemFactory', ItemFactory)
        True
        
        We also need a parser and factory for containers, registered for the
        empty extension (thus no special utility name). These can be very
        simple::
        
        >>> from z3c.vcsync.tests import ContainerParser, ContainerFactory
        >>> grok.testing.grok_component('ContainerParser', ContainerParser)
        True
        >>> grok.testing.grok_component('ContainerFactory', ContainerFactory)
        True
        
        Setting up the SVN repository
        -----------------------------
        
        Now we need an SVN repository to synchronize with. We create a test
        SVN repository now and create a svn path to a checkout::
        
        >>> from z3c.vcsync.tests import svn_repo_wc
        >>> repo, wc = svn_repo_wc()
        
        We can now initialize the ``SvnCheckout`` object with the SVN path to
        the checkout we just created::
        
        >>> from z3c.vcsync.svn import SvnCheckout
        >>> checkout = SvnCheckout(wc)
        
        The root directory of the working copy will be synchronized with the
        root container of the state. The checkout will therefore contain
        ``data`` and a ``found`` sub-directories.
        
        Constructing the synchronizer
        -----------------------------
        
        Now that we have the checkout and the state, we can set up a synchronizer::
        
        >>> from z3c.vcsync import Synchronizer
        >>> s = Synchronizer(checkout, state)
        
        Let's make ``s`` the current synchronizer as well. We need this in
        this example to get back to the last revision number::
        
        >>> current_synchronizer = s
        
        It's now time to set up our ``get_revision_nr`` method a bit better,
        making use of the information in the current synchronizer. In actual
        applications we'd probably get the revision number directly from the
        content, and there would be no need to get back to the synchronizer
        (it doesn't need to be persistent but can be constructed on demand)::
        
        >>> def get_revision_nr(self):
        ...    return current_synchronizer.state.get_revision_nr()
        >>> Item.get_revision_nr = get_revision_nr
        
        Synchronization
        ---------------
        
        We'll synchronize for the first time now::
        
        >>> info = s.sync("synchronize")
        
        We will now examine the SVN checkout to see whether the
        synchronization was successful.
        
        To do this we'll introduce some helper functions that help us present
        the paths in a more readable form, relative to the base of the
        checkout::
        
        >>> def pretty_path(path):
        ...     return path.relto(wc)
        >>> def pretty_paths(paths):
        ...     return sorted([pretty_path(path) for path in paths])
        
        We see that the Python object structure of containers and items has
        been translated to the same structure of directories and ``.test``
        files on the filesystem::
        
        >>> pretty_paths(wc.listdir())
        ['data']
        >>> pretty_paths(wc.join('data').listdir())
        ['data/bar.test', 'data/foo.test', 'data/sub']
        >>> pretty_paths(wc.join('data').join('sub').listdir())
        ['data/sub/qux.test']
        
        The ``.test`` files have the payload data we expect::
        
        >>> print wc.join('data').join('foo.test').read()
        1
        >>> print wc.join('data').join('bar.test').read()
        2
        >>> print wc.join('data').join('sub').join('qux.test').read()
        3
        
        Synchronization back into objects
        ---------------------------------
        
        Let's now try the reverse: we will change the SVN content from another
        checkout, and synchronize the changes back into the object tree.
        
        We have a second, empty tree that we will load objects into::
        
        >>> root2 = Container()
        >>> root2.__name__ = 'root'
        >>> state2 = TestState(root2)
        
        We make another checkout of the repository::
        
        >>> import py
        >>> wc2 = py.test.ensuretemp('wc2')
        >>> wc2 = py.path.svnwc(wc2)
        >>> wc2.checkout(repo)
        >>> checkout2 = SvnCheckout(wc2)
        
        Let's make a synchronizer for this new checkout and state::
        
        >>> s2 = Synchronizer(checkout2, state2)
        
        This is now the current synchronizer (so that our ``get_revision_nr``
        works properly)::
        
        >>> current_synchronizer = s2
        
        Now we'll synchronize::
        
        >>> info = s2.sync("synchronize")
        
        The state of objects in the tree must now mirror that of the original state::
        
        >>> sorted(root2.keys())
        ['data']
        
        >>> sorted(root2['data'].keys())
        ['bar', 'foo', 'sub']
        
        Now we will change some of these objects, and synchronize again::
        
        >>> root2['data']['bar'].payload = 20
        >>> root2['data']['sub']['qux'].payload = 30
        >>> info2 = s2.sync("synchronize")
        
        We can now synchronize the original tree again::
        
        >>> current_synchronizer = s
        >>> info = s.sync("synchronize")
        
        We should see the changes reflected into the original tree::
        
        >>> root2['data']['bar'].payload
        20
        >>> root2['data']['sub']['qux'].payload
        30
        
        More information
        ----------------
        
        To learn more about the APIs you can use and need to implement, see
        ``interfaces.py``.
        
        To learn about using ``z3c.vcsync`` to import and export content, see
        ``importexport.txt``.
        
        More low-level information may be gleaned from ``conflicts.txt`` and
        ``internal.txt``.
        
        
        z3c.vcsync changes
        ==================
        
        0.16 (2009-06-02)
        -----------------
        
        * Change the method by which zip files are created for export to a
        less filesystem-intensive method; files are directly added to the
        zip file. This hopefully brings performance benefits on platforms
        where accessing many small files is slow.
        
        0.15 (2008-08-19)
        -----------------
        
        * Fix a bug where the SVN "R" status was not recognized by the py lib.
        Monkey-patch py for now, though a fix should be released with py
        0.9.2 eventually. Strictly depend on py 0.9.1 for now to make sure
        monkey-patch applies cleanly.
        
        * A bit of refactoring of duplicated code in retrieving the objects
        modified and objects removed lists. Still not happy that this gets
        called twice per synchronization, but it was already doing this so
        doesn't get worse either.
        
        0.14 (2008-07-04)
        -----------------
        
        * Fixed a bug where too many path fragments were returned in case of
        conflicts. Now only paths that have in fact changed should be
        returned. Added some tests for this.
        
        * There was a case where two users would add a file with the same name
        to their own states independently. This used to result in an SVN
        error, but now also generates a conflict.  This conflict is a bit
        different in its behavior unfortunately, as it prefers the version
        already in SVN as opposed to the one last added.
        
        0.13 (2008-06-02)
        -----------------
        
        * The root directory of the checkout is now truly equivalent to the
        root object of the state. This means that if the SVN checkout
        content is to remain the same, the state root to synchronize should
        be one level higher (the parent of the current state root).
        
        * Conflict resolution has been cleaned up. When a conflict occurs, the
        other half of the conflict (the one not resolved) is moved into the
        ``found`` directory, which is created in the checkout root. This is
        also represented in the state container object and is synchronized
        like any other content.
        
        0.12 (2008-05-16)
        -----------------
        
        Features added
        ~~~~~~~~~~~~~~
        
        * The API has been cleaned up and revised. This will break code that
        uses this library, but so far I don't think that is many people
        yet. :)
        
        * A major refactoring of the tests, including real SVN tests. This
        requires SVN to be installed on the system where the tests are being
        run, including the ``svn-admin`` command.
        
        * ``IState`` objects now need to implement methods to access and
        maintain the revision number of the last revision.
        
        * The developer must now also implement ``IParser`` utilities for
        files that can be synchronized (besides ``IFactory``, which used to
        be called ``IVcFactory``. The ``IParser`` utility overwrites the
        existing object instead of creating a new one. This allows
        synchronization to be a bit nicer and not remove and recreate
        objects unnecessarily, which makes it harder to implement things
        like references between objects.
        
        * Add a facility to pass a special function along that is called for
        all objects created or modified during the synchronization (or
        during import). This function is called at the end when all objects
        that are going to exist already exist, so can be used in situations
        where the state of an object relies on the existence of another one.
        
        0.11 (2008-03-10)
        -----------------
        
        Bugs fixed
        ~~~~~~~~~~
        
        * Do not try to remove non-existent files during synchronization. A
        file might have been removed in SVN and there is no more need to
        re-remove it if it was also removed locally.
        
        * There was an off-by one error during the "up" phase of
        synchronization with SVN, and as a result a log entry that was
        already processed could be re-processed during this next
        synchronisation. This could in some cases revive folders as unknown
        directories on the filesystem, leading to errors and
        inconsistencies.
        
        0.10 (2008-01-08)
        -----------------
        
        Features added
        ~~~~~~~~~~~~~~
        
        * The ``.sync()`` method now does not return the revision number, but
        an ``ISynchronizationInfo`` object. This has a ``revision_nr``
        attribute and also contains some information on what happened during
        the synchronization process.
        
        Bugs fixed
        ~~~~~~~~~~
        
        * revision number after synchronization was not always updated
        properly to the latest number of the repository. Now retrieve this
        number from ``commit()`` where possible.
        
        0.9.1 (2007-11-29)
        ------------------
        
        Bugs fixed
        ~~~~~~~~~~
        
        * When resolving objects in the ZODB, a path was generated that has
        separators that are actually dependent on the operating system in
        use (``/`` for Unices, but ``\`` for windows). This caused
        synchronization to fail on Windows, completely flattening
        hierarchies. Now use os.path.sep to be platform-independent.
        
        0.9 (2007-11-25)
        ----------------
        
        Features added
        ~~~~~~~~~~~~~~
        
        * The importing logic now allows the user to import new content over
        existing content. In this case any existing content is left alone,
        but new objects are added. Any attempt to overwrite existing content
        is ignored.
        
        Bugs fixed
        ~~~~~~~~~~
        
        * In some cases a containing directory is referenced which does not
        exist anymore when removing files. In this case we do not need to
        remove the file anymore, as the directory itself is gone.
        
        * SVN doesn't actually remove directories, just mark them for
        removal. This could confuse the system during synchronization:
        removed directories might reappear again as they were still on the
        filesystem during loading. Make sure now that any directories marked
        for removal are also properly removed in the filesystem before load
        starts, but after up (as rm-ing a directory marked for removal
        before svn up will actually re-add this directory!).
        
        Restructuring
        ~~~~~~~~~~~~~
        
        * Previously the datetime of last synchronization was used to
        determine what to synchronize both in the ZODB as well as in the
        checkout. This has a significant drawback if the datetime setting of
        the computer the synchronization code is running on is ahead of the
        datetime setting of the version control server: updates could be
        lost.
        
        Changed the code to use a revision_nr instead. This is a number that
        increments with each synchronization, and the number can be used to
        determine both what changes have been made since last
        synchronization in the ZODB as well as in the version control
        system. This is a more robust approach.
        
        0.8.1 (2007-11-07)
        ------------------
        
        Bugs fixed
        ~~~~~~~~~~
        
        * Fix a bug in conversion of SVN timestamps to datetimes. Previous
        code worked in DST, but not during winter time. The new code might
        of course break under DST - the mysterious of datetime conversion
        are legion.
        
        * A cleaner way to cache the files listing from SVN.
        
        * Work around a bug in the Py library. The Py library doesn't support
        the R status code from SVN and raises a NotImplementedError when it
        encounters it. Evilly catch these NotImplementedErrors for now. The
        bug has been reported upstream and should be fixed in the next
        release of Py.
        
        
Keywords: zope3 grok web svn synchronization
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Zope Public License
Classifier: Programming Language :: Python
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Framework :: Zope3
