Metadata-Version: 2.1
Name: iterfilesystem
Version: 1.1.0
Summary: Threaded directory iteration via os.scandir() with progress indicator and resume function.
Home-page: https://github.com/jedie/iterfilesystem
Author: Jens Diemer
Author-email: python@jensdiemer.de
License: GNU General Public License v3
Keywords: iterfilesystem
Platform: UNKNOWN
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Requires-Python: >=3.6

--------------
IterFilesystem
--------------

Multiprocess directory iteration via ``os.scandir()``:

* "stats" processes:

    * only counts up all directories and files.

    * accumulates the sizes of all files.

* "worker" process:

    * Filesystem walk and process the real action with dir/files

among other things these packages are used:

* progress bar `tqdm <https://pypi.org/project/tqdm/>`_

Requirement:

* Python 3.6 or newer.

* `Pipenv <https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv>`_. Packages and virtual environment manager.

Please: try, fork and contribute! ;)

+--------------------------------------+----------------------------------------------------------+
| |Build Status on travis-ci.org|      | `travis-ci.org/jedie/IterFilesystem`_                    |
+--------------------------------------+----------------------------------------------------------+
| |Build Status on appveyor.com|       | `ci.appveyor.com/project/jedie/IterFilesystem`_          |
+--------------------------------------+----------------------------------------------------------+
| |Coverage Status on codecov.io|      | `codecov.io/gh/jedie/IterFilesystem`_                    |
+--------------------------------------+----------------------------------------------------------+
| |Coverage Status on coveralls.io|    | `coveralls.io/r/jedie/IterFilesystem`_                   |
+--------------------------------------+----------------------------------------------------------+
| |Requirements Status on requires.io| | `requires.io/github/jedie/IterFilesystem/requirements/`_ |
+--------------------------------------+----------------------------------------------------------+

.. |Build Status on travis-ci.org| image:: https://travis-ci.org/jedie/IterFilesystem.svg
.. _travis-ci.org/jedie/IterFilesystem: https://travis-ci.org/jedie/IterFilesystem/
.. |Build Status on appveyor.com| image:: https://ci.appveyor.com/api/projects/status/py5sl38ql3xciafc?svg=true
.. _ci.appveyor.com/project/jedie/IterFilesystem: https://ci.appveyor.com/project/jedie/IterFilesystem/history
.. |Coverage Status on codecov.io| image:: https://codecov.io/gh/jedie/IterFilesystem/branch/master/graph/badge.svg
.. _codecov.io/gh/jedie/IterFilesystem: https://codecov.io/gh/jedie/IterFilesystem
.. |Coverage Status on coveralls.io| image:: https://coveralls.io/repos/jedie/IterFilesystem/badge.svg
.. _coveralls.io/r/jedie/IterFilesystem: https://coveralls.io/r/jedie/IterFilesystem
.. |Requirements Status on requires.io| image:: https://requires.io/github/jedie/IterFilesystem/requirements.svg?branch=master
.. _requires.io/github/jedie/IterFilesystem/requirements/: https://requires.io/github/jedie/IterFilesystem/requirements/

-------
Example
-------

Use example CLI, e.g.:

::

    ~$ git clone https://github.com/jedie/IterFilesystem.git
    ~$ cd IterFilesystem
    ~/IterFilesystem$ pipenv install
    ~/IterFilesystem$ pipenv shell
    (IterFilesystem) ~/IterFilesystem$ print_fs_stats --help
    (IterFilesystem) ~/IterFilesystem$ pip install -e .
    ...
    Successfully installed iterfilesystem

    (IterFilesystem) ~/IterFilesystem$ $ print_fs_stats --help
    usage: print_fs_stats.py [-h] [-v] [--debug] [--path PATH]
                             [--skip_dir_patterns [SKIP_DIR_PATTERNS [SKIP_DIR_PATTERNS ...]]]
                             [--skip_file_patterns [SKIP_FILE_PATTERNS [SKIP_FILE_PATTERNS ...]]]

    Scan filesystem and print some information

    optional arguments:
      -h, --help            show this help message and exit
      -v, --version         show program's version number and exit
      --debug               enable DEBUG
      --path PATH           The file path that should be scanned e.g.: "~/foobar/"
                            default is "~"
      --skip_dir_patterns [SKIP_DIR_PATTERNS [SKIP_DIR_PATTERNS ...]]
                            Directory names to exclude from scan.
      --skip_file_patterns [SKIP_FILE_PATTERNS [SKIP_FILE_PATTERNS ...]]
                            File names to ignore.

example output looks like this:

::

    (IterFilesystem) ~/IterFilesystem$ $ print_fs_stats --path ~/IterFilesystem --skip_dir_patterns ".*" "*.egg-info" --skip_file_patterns ".*"
    Read/process: '~/IterFilesystem'...
    Skip directory patterns:
    	* .*
    	* *.egg-info

    Skip file patterns:
    	* .*

    Filesystem items..:Read/process: '~/IterFilesystem'...

    ...

    Filesystem items..: 100%|█████████████████████████████████████████|135/135 13737.14entries/s [00:00<00:00, 13737.14entries/s]
    File sizes........: 100%|██████████████████████████████████████████████████████████████|843k/843k [00:00<00:00, 88.5MBytes/s]
    Average progress..: 100%|███████████████████████████████████████████████████████████████████████████████████████|00:00<00:00
    Current File......:, /home/jens/repos/IterFilesystem/Pipfile


    Processed 135 filesystem items in 0.02 sec
    SHA515 hash calculated over all file content: 10f9475b21977f5aea1d4657a0e09ad153a594ab30abc2383bf107dbc60c430928596e368ebefab3e78ede61dcc101cb638a845348fe908786cb8754393439ef
    File count: 109
    Total file size: 843.5 KB
    6 directories skipped.
    6 files skipped.

-------
History
-------

* **dev** - `compare v1.1.0...master <https://github.com/jedie/IterFilesystem/compare/v1.1.0...master>`_ 

    * TBC

* 12.10.2019 - `compare v1.0.0...v1.1.0 <https://github.com/jedie/IterFilesystem/compare/v1.0.0...v1.1.0>`_ 

    * don't create separate process for worker: Just do the work in main process

    * dir/file filter uses now ``fnmatch``

* 12.10.2019 - `compare v0.2.0...v1.0.0 <https://github.com/jedie/IterFilesystem/compare/v0.2.0...v1.0.0>`_ 

    * refactoring:

        * don't use ``persist-queue``

        * switch from threading to multiprocessing

        * enhance progress display with multiple ``tqdm`` process bars

* 15.09.2019 - `compare v0.1.0...v0.2.0 <https://github.com/jedie/IterFilesystem/compare/v0.1.0...v0.2.0>`_ 

    * store persist queue in temp directory

    * Don't catch ``process_path_item`` errors, this should be made in child class

* 15.09.2019 - `compare v0.0.1...v0.1.0 <https://github.com/jedie/IterFilesystem/compare/v0.0.1...v0.1.0>`_ 

    * add some project meta files and tests

    * setup CI

    * fix tests

* 15.09.2019 - v0.0.1

    * first Release on PyPi

-----
Links
-----

* `https://pypi.python.org/pypi/IterFilesystem/ <https://pypi.python.org/pypi/IterFilesystem/>`_

* `https://github.com/jedie/IterFilesystem/ <https://github.com/jedie/IterFilesystem/>`_

--------
Donating
--------

* `paypal.me/JensDiemer <https://www.paypal.me/JensDiemer>`_

* `Flattr This! <https://flattr.com/submit/auto?uid=jedie&url=https%3A%2F%2Fgithub.com%2Fjedie%2FIterFilesystem%2F>`_

* Send `Bitcoins <http://www.bitcoin.org/>`_ to `1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6F <https://blockexplorer.com/address/1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6F>`_

