Metadata-Version: 2.1
Name: wendelin.core
Version: 2.0a2.post1
Summary: Out-of-core NumPy arrays
Home-page: https://lab.nexedi.com/nexedi/wendelin.core
Author: Kirill Smelkov
Author-email: kirr@nexedi.com
License: GPLv3+ with wide exception for Open-Source
Description: ==========================================
         Wendelin.core - Out-of-core NumPy arrays
        ==========================================
        
        Wendelin.core allows you to work with arrays bigger than RAM and local disk.
        Bigarrays are persisted to storage, and can be changed in transactional manner.
        
        In other words bigarrays are something like `numpy.memmap`_ for numpy.ndarray
        and OS files, but support transactions and files bigger than disk. The whole
        bigarray cannot generally be used as a drop-in replacement for numpy arrays,
        but bigarray *slices* are real ndarrays and can be used everywhere ndarray can
        be used, including in C/Cython/Fortran code. Slice size is limited by
        virtual address-space size, which is ~ max 127TB on Linux/amd64.
        
        The main class to work with is `ZBigArray` and is used like `ndarray` from
        `NumPy`_:
        
        1. create array::
        
            from wendelin.bigarray.array_zodb import ZBigArray
            import transaction
        
            # root is connected to opened database
            root['A'] = A = ZBigArray(shape=..., dtype=...)
            transaction.commit()
        
        2. view array as a real ndarray::
        
            a = A[:]        # view which covers all array, if it fits into address-space
            b = A[10:100]
        
           data for views will be loaded lazily on memory access.
        
        3. work with views, including using C/Cython/Fortran functions from NumPy
           and other libraries to read/modify data::
        
            a[2] = 1
            a[10:20] = numpy.arange(10)
            numpy.mean(a)
        
           | the amount of modifications in one transaction should be less than available RAM.
           | the amount of data read is limited only by virtual address-space size.
        
        4. data can be appended to array in O(δ) time::
        
            values                  # ndarray to append of shape  (δ,)
            A.append(values)
        
           and array itself can be resized in O(1) time::
        
            A.resize(newshape)
        
        5. changes to array data can be either discarded or saved back to DB::
        
            transaction.abort()     # discard all made changes
            transaction.commit()    # atomically save all changes
        
        
        
        When using NEO_ or ZEO_ as a database, bigarrays can be simultaneously used by
        several nodes in a cluster.
        
        
        Please see `demo/demo_zbigarray.py`__ for a complete example.
        
        __ demo/demo_zbigarray.py
        
        
        Current state and Roadmap
        =========================
        
        Wendelin.core works in real life for workloads Nexedi_ is using in production,
        including 24/7 projects. We are, however, aware of the following
        limitations and things that need to be improved:
        
        - wendelin.core is currently not very fast
        - there are big - proportional to input in size - temporary array allocations
          in third-party libraries (NumPy_, `scikit-learn`_, ...) which might practically
          prevent processing out-of-core arrays depending on the functionality used.
        
        Thus
        
        - we are currently working on improved wendelin.core design and implementation,
          which uses kernel virtual memory manager (complemented by one implemented__ in__
          userspace__) with arrays backend presented to kernel via FUSE as virtual
          filesystem implemented in Go.
        
          As of 2021 November `this filesystem`__ reached its alpha state and is staged
          to be tried for real.
        
        __  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h
        __  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c
        __  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c
        __  https://lab.nexedi.com/nexedi/wendelin.core/blob/master/wcfs/wcfs.go
        
        In parallel we will also:
        
        - try wendelin.core 1.0 on large data sets
        - identify and incrementally fix big-temporaries allocation issues in NumPy and
          scikit-learn
        
        We are open to community help with the above.
        
        
        Additional materials
        ====================
        
        - Wendelin.core tutorial__
        - Slides__ (pdf__) from presentation about wendelin.core in PyData Paris 2015
        
        __  https://www.nexedi.com/wendelin-Core.Tutorial.2016
        __  https://wendelin.nexedi.com/NXD-Wendelin.Core.Non.Secret/asEntireHTML
        __  https://wendelin.nexedi.com/NXD-Wendelin.Core.Non.Secret?format=pdf
        
        
        .. _NumPy:          http://www.numpy.org/
        .. _scikit-learn:   http://scikit-learn.org/
        .. _numpy.memmap:   http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
        .. _NEO:            http://www.neoppod.org/
        .. _ZEO:            https://pypi.python.org/pypi/ZEO
        .. _Nexedi:         https://www.nexedi.com/
        
        ----
        
        Wendelin.core change history
        ============================
        
        - `2.0.alpha2`__ (2022-01-27)  Fix several crashes discovered by first on-field usage.
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/compare/49f826b1...a36cdcc3
        
        
        2.0.alpha1 (2021-11-16)
        -----------------------
        
        This is a major pre-release that reduces wendelin.core RAM consumption
        dramatically:
        
        The project switches to be mainly using kernel virtual memory manager.
        Bigfiles are now primarily accessed with plain OS-level mmap to files from
        synthetic WCFS filesystem. This makes bigfile's cache (now it is the kernel's
        pagecache) to be shared in between several processes.
        
        In addition a custom coherency protocol is provided, which allows clients,
        that want to receive isolation guarantee ("I" from ACID), to still use the shared
        cache and at the same time get bigfile view isolated from other's changes.
        
        By default wendelin.core python client continues to provide full ACID semantics as
        before.
        
        In addition to being significantly more efficient, WCFS also fixes
        data-corruption bugs that were discovered__ in how Wendelin.core 1 handles
        invalidations on BTree topology change.
        
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/8c32c9f6
        
        Please see wcfs.go__ for description of the new filesystem.
        
        __ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/wcfs/wcfs.go
        
        Major steps: 1__, 2__, 3__, 4__, 5__, 6__, 7__, 8__, 9__, 10__, 11__, 12__,
        13__, 14__, 15__, 16__, 17__, 18__, 19__, 20__, 21__.
        
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2c152d41?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e3f2ee2d?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0e829874?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/a8595565?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/b87edcfe?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1f2cd49d?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/27df5a3b?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/80153aa5?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ab4be93?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f980471f?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/4430de41?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6f0cdaff?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/10f7153a?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fae045cc?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/23362204?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ceadfcc7?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1dba3a9a?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1f866c00?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e11edc70?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/986cf86e?expanded=1
        __ https://lab.nexedi.com/nexedi/wendelin.core/commit/c5e18c74?expanded=1
        
        
        0.13 (2019-06-18)
        -----------------
        
        - Add support for Python 3.7 (commit__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/bca5f79e6f
        
        - Add `RAMArray` class which is compatible to `ZBigArray` in API and semantic,
          but stores its data in RAM only (`commit 1`__, 2__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7365979b9d
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc9b69d8e1
        
        - Add `lib.xnumpy.structured` - utility to create structured view of an array (`commit 1`__, 2__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6a5dfefaf8
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/32ca80e2d5
        
        - Fix logic to keep `ZBigFileH` in sync with ZODB connection (commit__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d9d6adec1b
        
        - Fix crash on PyVMA deallocation (commit__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d97641d2ba
        
        - Move `py.bench` to pygolang__ so that it can be used not only in
          Wendelin.core (commit__).
        
          __ https://pypi.org/project/pygolang/
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/318efce0bf
        
        - Enhance `t/qemu-runlinux` - utility that is used to work on combined
          kernel/user-space workloads (`commit 1`__, 2__, 3__, 4__, 5__, 6__).
          This was in particular useful to develop Linux kernel fixes that are needed
          for Wendelin.core 2.0 (`kernel commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fe541453f8
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ccca055cfe
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6ab952207e
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/a568d6d999
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/208aca62ae
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/89fb89929a
        
          __ https://git.kernel.org/linus/ad2ba64dd489
          __ https://git.kernel.org/linus/10dce8af3422
          __ https://git.kernel.org/linus/bbd84f33652f
          __ https://git.kernel.org/linus/c5bf68fe0c86
          __ https://git.kernel.org/linus/438ab720c675
          __ https://git.kernel.org/linus/7640682e67b3
          __ https://git.kernel.org/linus/d4b13963f217
        
        - Various bugfixes.
        
        0.12 (2018-04-16)
        -----------------
        
        - Update licensing to be in line with whole Nexedi stack (`commit`__). Please
          see https://www.nexedi.com/licensing for details, rationale and options.
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f11386a4
        
        - Add `ArrayRef` utility to find out for a NumPy array its top-level root
          parent and how to recreate the array as some view of the root;
          this builds the foundation for e.g. sending arrays as references without copy
          in CMFActivity joblib backend
          (`commit 1`__, 2__, 3__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e9d61a89
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53371b6
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/450ad804
        
        
        - Don't crash if during `loadblk()` garbage collection was run twice at tricky
          times (`commit 1`__, 2__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/4228d8b6
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3804cc39
        
        - Don't crash on writeout if previously `storeblk()` resulted in error
          (`commit`__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/87bf4908
        
        
        
        - Fix `py.bench` and rework it to produce output in Go benchmarking format
          (`commit 1`__, 2__, 3__, 4__, 5__); add benchmarks for handling pagefaults
          (`commit`__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3cfc2728
        
        - Use zodbtools/zodburi, if available, to open database by URL
          (`commit`__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f785ac07
        
        - Start to make sure it works with ZODB5 too (`commit 1`__, 2__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/808b59b7
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0dbf3c44
        
        - Various bugfixes.
        
        0.11 (2017-03-28)
        -----------------
        
        - Switch back to using ZBlk0 format by default (`commit`__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178
        
        0.10 (2017-03-16)
        -----------------
        
        - Tell the world `dtype=object` is not supported (`commit`__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e44bd761
        
        0.9 (2017-01-17)
        ----------------
        
        - Avoid deadlocks via doing `storeblk()` calls with virtmem lock released
          (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/8bb7f2f2
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fb4bfb32
        
        - Don't crash if in `loadblk()` implementation an exception is internally
          raised & caught
          (`commit 1`__, 2__, 3__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9aa6a5d7
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/61b18a40
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/024c246c
        
        0.8 (2016-09-28)
        ----------------
        
        - Do not leak memory when loading data in ZBlk1 format (`commit`__).
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/542917d1
        
        0.7 (2016-07-14)
        ------------------
        
        - Add support for Python 3.5 (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/20115391
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e6beab19
        
        - Fix bug in pagemap code which could lead to crashes and other issues (`commit`__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ee9bcd00
        
        - Various bugfixes
        
        0.6 (2016-06-13)
        ----------------
        
        - Add support for FORTRAN ordering (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ab9ca2df
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ca0f076
        
        
        - Avoid deadlocks via doing `loadblk()` calls with virtmem lock released
          (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f49c11a3
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0231a65d
        
        - Various bugfixes
        
        0.5 (2015-10-02)
        ----------------
        
        - Introduce another storage format, which is optimized for small changes, and
          make it the default.
          (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/13c0c17c
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae42085
        
        - Various bugfixes and documentation improvements
        
        
        0.4 (2015-08-19)
        ----------------
        
        - Add support for O(δ) in-place BigArray.append() (commit__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1245acc9
        
        - Implement proper multithreading support (commit__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53271b9
        
        - Implement proper RAM pages invalidation when backing ZODB objects are changed
          from outside (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/cb779c7b
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/92bfd03e
        
        - Fix all kind of failures that could happen when ZODB connection changes
          worker thread in-between handling requests (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/c7c01ce4
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/64d1f40b
        
        - Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages
          (`commit 1`__, 2__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/010eeb35
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7fc4ec66
        
        - Various bugfixes
        
        
        
        0.3 (2015-06-12)
        ----------------
        
        - Add support for automatic BigArray -> ndarray conversion, so that e.g. the
          following::
        
            A = BigArray(...)
            numpy.mean(A)       # passing BigArray to plain NumPy function
        
          either succeeds, or raises MemoryError if not enough address space is
          available to cover whole A. (current limitation is ~ 127TB on linux/amd64)
        
          (commit__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/00db08d6
        
        - Various bugfixes (build-fixes, crashes, overflows, etc)
        
        
        0.2 (2015-05-25)
        ----------------
        
        - Add support for O(1) in-place BigArray.resize() (commit__)
        
          __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ca064f75
        
        - Various build bugfixes (older systems, non-std python, etc)
        
        
        0.1 (2015-04-03)
        ----------------
        
        - Initial release
        
Keywords: bigdata out-of-core numpy virtual-memory
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Framework :: ZODB
Provides-Extra: test
