##############################################################################
#
# Copyright (c) 2005 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################

Packing support for blob data
=============================

Set up:

    >>> from ZODB.MappingStorage import MappingStorage
    >>> from ZODB.serialize import referencesf
    >>> from ZODB.blob import Blob, BlobStorage
    >>> from ZODB import utils
    >>> from ZODB.DB import DB
    >>> import transaction
    >>> storagefile = 'Data.fs'
    >>> blob_dir = 'blobs'

A helper method to assure a unique timestamp across multiple platforms:

    >>> from ZODB.tests.testblob import new_time

UNDOING
=======

See blob_packing.txt.

NON-UNDOING
===========

We need an database with a NON-undoing blob supporting storage:

    >>> base_storage = MappingStorage('storage')
    >>> blob_storage = BlobStorage(blob_dir, base_storage)
    >>> database = DB(blob_storage)

Create our root object:

    >>> connection1 = database.open()
    >>> root = connection1.root()

Put some revisions of a blob object in our database and on the filesystem:

    >>> import time, os
    >>> tids = []
    >>> times = []
    >>> nothing = transaction.begin()
    >>> times.append(new_time())
    >>> blob = Blob()
    >>> with blob.open('w') as file:
    ...     _ = file.write(b'this is blob data 0')
    >>> root['blob'] = blob
    >>> transaction.commit()
    >>> tids.append(blob_storage._tid)

    >>> nothing = transaction.begin()
    >>> times.append(new_time())
    >>> with root['blob'].open('w') as file:
    ...     _ = file.write(b'this is blob data 1')
    >>> transaction.commit()
    >>> tids.append(blob_storage._tid)

    >>> nothing = transaction.begin()
    >>> times.append(new_time())
    >>> with root['blob'].open('w') as file:
    ...     _ = file.write(b'this is blob data 2')
    >>> transaction.commit()
    >>> tids.append(blob_storage._tid)

    >>> nothing = transaction.begin()
    >>> times.append(new_time())
    >>> with root['blob'].open('w') as file:
    ...     _ = file.write(b'this is blob data 3')
    >>> transaction.commit()
    >>> tids.append(blob_storage._tid)

    >>> nothing = transaction.begin()
    >>> times.append(new_time())
    >>> with root['blob'].open('w') as file:
    ...     _ = file.write(b'this is blob data 4')
    >>> transaction.commit()
    >>> tids.append(blob_storage._tid)

    >>> oid = root['blob']._p_oid
    >>> fns = [ blob_storage.fshelper.getBlobFilename(oid, x) for x in tids ]
    >>> [ os.path.exists(x) for x in fns ]
    [True, True, True, True, True]

Get our blob filenames for this oid.

    >>> fns = [ blob_storage.fshelper.getBlobFilename(oid, x) for x in tids ]

Do a pack to the slightly before the first revision was written:

    >>> packtime = times[0]
    >>> blob_storage.pack(packtime, referencesf)
    >>> [ os.path.exists(x) for x in fns ]
    [False, False, False, False, True]

Do a pack to now:

    >>> packtime = new_time()
    >>> blob_storage.pack(packtime, referencesf)
    >>> [ os.path.exists(x) for x in fns ]
    [False, False, False, False, True]

Delete the object and do a pack, it should get rid of the most current
revision as well as the entire directory:

    >>> nothing = transaction.begin()
    >>> del root['blob']
    >>> transaction.commit()
    >>> packtime = new_time()
    >>> blob_storage.pack(packtime, referencesf)
    >>> [ os.path.exists(x) for x in fns ]
    [False, False, False, False, False]
    >>> os.path.exists(os.path.split(fns[0])[0])
    False

Avoiding parallel packs
=======================

Blob packing (similar to FileStorage) can only be run once at a time. For
this, a flag (_blobs_pack_is_in_progress) is set. If the pack method is called
while this flag is set, it will refuse to perform another pack, until the flag
is reset:

    >>> blob_storage._blobs_pack_is_in_progress
    False
    >>> blob_storage._blobs_pack_is_in_progress = True
    >>> blob_storage.pack(packtime, referencesf)
    Traceback (most recent call last):
    BlobStorageError: Already packing
    >>> blob_storage._blobs_pack_is_in_progress = False
    >>> blob_storage.pack(packtime, referencesf)

We can also see, that the flag is set during the pack, by leveraging the
knowledge that the underlying storage's pack method is also called:

    >>> import six
    >>> def dummy_pack(time, ref):
    ...     six.print_(
    ...         "_blobs_pack_is_in_progress =",
    ...         blob_storage._blobs_pack_is_in_progress)
    ...     return base_pack(time, ref)
    >>> base_pack = base_storage.pack
    >>> base_storage.pack = dummy_pack
    >>> blob_storage.pack(packtime, referencesf)
    _blobs_pack_is_in_progress = True
    >>> blob_storage._blobs_pack_is_in_progress
    False
    >>> base_storage.pack = base_pack
