Metadata-Version: 2.1
Name: pytorch_zoo
Version: 1.2.2
Summary: A collection of useful modules and utilities for kaggle not available in Pytorch
Home-page: https://github.com/bkahn-github/pytorch_zoo
Author: Bilal Khan
Author-email: bk@tinymanager.com
License: UNKNOWN
Description: <h1 align='center'>
            Pytorch Zoo
        </h1>
        
        <h4 align='center'>
            A collection of useful modules and utilities (especially helpful for kaggling) not available in <a href="https://pytorch.org">Pytorch</a>
        </h4>
        
        <p align='center'>
            <a href="https://forthebadge.com">
                <img src="https://forthebadge.com/images/badges/made-with-python.svg" alt="forthebadge">
            </a>
            <a href="https://lgtm.com/projects/g/bkkaggle/pytorch_zoo/context:python">
                <img alt="Language grade: Python" src="https://img.shields.io/lgtm/grade/python/g/bkkaggle/pytorch_zoo.svg?logo=lgtm&logoWidth=18"/>
            </a>
            <a href="https://lgtm.com/projects/g/bkkaggle/pytorch_zoo/alerts/">
                <img alt="Total alerts" src="https://img.shields.io/lgtm/alerts/g/bkkaggle/pytorch_zoo.svg?logo=lgtm&logoWidth=18"/>
            </a>
            <a href="https://github.com/prettier/prettier">
                <img src="https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square" alt="code style: prettier" />
            </a>
            <a href="https://opensource.org/licenses/MIT">
                <img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT">
            </a>
            <a href="http://makeapullrequest.com">
                <img src="https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square" alt="PRs Welcome">
            </a>
            <a href="https://github.com/bkkaggle/pytorch_zoo/issues">
                <img alt="GitHub issues" src="https://img.shields.io/github/issues/bkkaggle/pytorch_zoo.svg?style=flat">
            </a>
            <a href="https://github.com/bkkaggle/pytorch_zoo/pulls">
                <img alt="GitHub pull requests" src="https://img.shields.io/github/issues-pr/bkkaggle/pytorch_zoo.svg">
            </a>
            <a href="https://github.com/bkkaggle/pytorch_zoo/issues">
                <img alt="GitHub last commit" src="https://img.shields.io/github/last-commit/bkkaggle/pytorch_zoo.svg">
            </a>
        
        </p>
        
        <p align='center'>
            <a href='#overview'>Overview</a> •
            <a href='#installation'>Installation</a> •
            <a href='#documentation'>Documentation</a> •
            <a href='#contributing'>Contributing</a> •
            <a href='#authors'>Authors</a> •
            <a href='#license'>License</a> •
            <a href='#acknowledgements'>Acknowledgements</a>
        </p>
        
        <div>
            <img src="./screenshot.png" />
        </div>
        
        <p align='center'><strong>Made by <a href='https://github.com/bkkaggle'>Bilal Khan</a> • https://bilal.software</strong></p>
        
        <!-- START doctoc generated TOC please keep comment here to allow auto update -->
        <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
        
        - [Installation](#installation)
        - [Documentation](#documentation)
          - [Notifications](#notifications)
            - [Sending yourself notifications when your models finish training](#sending-yourself-notifications-when-your-models-finish-training)
            - [Viewing training progress with tensorboard in a kaggle kernel](#viewing-training-progress-with-tensorboard-in-a-kaggle-kernel)
          - [Data](#data)
              - [DynamicSampler(sampler, batch_size=32)](#dynamicsamplersampler-batch_size32)
              - [trim_tensors(tensors)](#trim_tensorstensors)
          - [Loss](#loss)
              - [lovasz_hinge(logits, labels, per_image=True)](#lovasz_hingelogits-labels-per_imagetrue)
              - [DiceLoss()](#diceloss)
          - [Metrics](#metrics)
          - [Modules](#modules)
              - [SqueezeAndExcitation(in_ch, r=16)](#squeezeandexcitationin_ch-r16)
              - [ChannelSqueezeAndSpatialExcitation(in_ch)](#channelsqueezeandspatialexcitationin_ch)
              - [ConcurrentSpatialAndChannelSqueezeAndChannelExcitation(in_ch)](#concurrentspatialandchannelsqueezeandchannelexcitationin_ch)
              - [GaussianNoise(0.1)](#gaussiannoise01)
          - [Schedulers](#schedulers)
              - [CyclicalMomentum(optimizer, base_momentum=0.8, max_momentum=0.9, step_size=2000, mode="triangular")](#cyclicalmomentumoptimizer-base_momentum08-max_momentum09-step_size2000-modetriangular)
          - [Utils](#utils)
              - [notify({'value1': 'Notification title', 'value2': 'Notification body'}, key)](#notifyvalue1-notification-title-value2-notification-body-key)
              - [seed_environment(seed=42)](#seed_environmentseed42)
              - [gpu_usage(device, digits=4)](#gpu_usagedevice-digits4)
              - [n_params(model)](#n_paramsmodel)
              - [save_model(model, fold=0)](#save_modelmodel-fold0)
              - [load_model(model, fold=0)](#load_modelmodel-fold0)
              - [save(obj, 'obj.pkl')](#saveobj-objpkl)
              - [load('obj.pkl')](#loadobjpkl)
              - [masked_softmax(logits, mask, dim=-1)](#masked_softmaxlogits-mask-dim-1)
              - [masked_log_softmax(logits, mask, dim=-1)](#masked_log_softmaxlogits-mask-dim-1)
        - [Contributing](#contributing)
        - [Authors](#authors)
        - [License](#license)
        - [Acknowledgements](#acknowledgements)
        
        <!-- END doctoc generated TOC please keep comment here to allow auto update -->
        
        ## Installation
        
        pytorch_zoo can be installed from pip
        
        ```
        pip install pytorch_zoo
        ```
        
        ## Documentation
        
        ### Notifications
        
        #### Sending yourself notifications when your models finish training
        
        IFTTT allows you to easily do this. Follow https://medium.com/datadriveninvestor/monitor-progress-of-your-training-remotely-f9404d71b720 to setup an IFTTT webhook and get a secret key.
        
        Once you have a key, you can send yourself a notification with:
        
        ```python
        from pytorch_zoo.utils import notify
        
        message = f'Validation loss: {val_loss}'
        obj = {'value1': 'Training Finished', 'value2': message}
        
        notify(obj, [YOUR_SECRET_KEY_HERE])
        ```
        
        #### Viewing training progress with tensorboard in a kaggle kernel
        
        Make sure tensorboard is installed in the kernel and run the following in a code cell near the beginning of your kernel:
        
        ```python
        !mkdir logs
        !wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
        !unzip -o ngrok-stable-linux-amd64.zip
        LOG_DIR = './logs'
        get_ipython().system_raw(
            'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'
            .format(LOG_DIR)
        )
        get_ipython().system_raw('./ngrok http 6006 &')
        
        !curl -s http://localhost:4040/api/tunnels | python3 -c \
            "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
        
        temp = !curl -s http://localhost:4040/api/tunnels | python3 -c "import sys,json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
        
        from pytorch_zoo.utils import notify
        
        obj = {'value1': 'Tensorboard URL', 'value2': temp[0]}
        notify(obj, [YOUR_SECRET_KEY_HERE])
        
        !rm ngrok
        !rm ngrok-stable-linux-amd64.zip
        ```
        
        This will start tensorboard, set up a http tunnel, and send you a notification with a url where you can access tensorboard.
        
        ### Data
        
        ##### [DynamicSampler(sampler, batch_size=32)](./pytorch_zoo/data.py#L4)
        
        A dynamic batch length data sampler. To be used with `trim_tensors`.
        
        Implementation adapted from https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/94779 and https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py
        
        ```python
        train_dataset = data.TensorDataset(data)
        sampler = data.RandomSampler(train_dataset)
        sampler = DynamicSampler(sampler, batch_size=32, drop_last=False)
        train_loader = torch.utils.data.DataLoader(train_dataset, batch_sampler=len_sampler)
        
        for epoch in range(10):
            for batch in train_loader:
                batch = trim_tensors(batch)
                train_batch(...)
        ```
        
        _Arguments_:  
        `sampler` (torch.utils.data.Sampler): Base sampler.  
        `batch_size` (int): Size of minibatch.  
        `drop_last` (bool): If `True`, the sampler will drop the last batch if its size would be less than `batch_size`.
        
        ##### [trim_tensors(tensors)](./pytorch_zoo/data.py#L48)
        
        Trim padding off of a batch of tensors to the smallest possible length. To be used with `DynamicSampler`.
        
        Implementation adapted from https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/94779
        
        ```python
        train_dataset = data.TensorDataset(data)
        sampler = data.RandomSampler(train_dataset)
        sampler = DynamicSampler(sampler, batch_size=32, drop_last=False)
        train_loader = torch.utils.data.DataLoader(train_dataset, batch_sampler=len_sampler)
        
        for epoch in range(10):
            for batch in train_loader:
                batch = trim_tensors(batch)
                train_batch(...)
        ```
        
        _Arguments_:  
        `tensors` ([torch.tensor]): list of tensors to trim.
        
        _Returns_:  
        ([torch.tensor]): list of trimmed tensors.
        
        ### Loss
        
        ##### [lovasz_hinge(logits, labels, per_image=True)](./pytorch_zoo/loss.py#L84)
        
        The binary Lovasz Hinge loss for semantic segmentation.
        
        Implementation adapted from https://github.com/bermanmaxim/LovaszSoftmax
        
        ```python
        loss = lovasz_hinge(logits, labels)
        ```
        
        _Arguments_:  
        `logits` (torch.tensor): Logits at each pixel (between -\infty and +\infty).  
        `labels` (torch.tensor): Binary ground truth masks (0 or 1).  
        `per_image` (bool, optional): Compute the loss per image instead of per batch. Defaults to True.
        
        _Shape_:
        
        -   Input:
            -   logits: (batch, height, width)
            -   labels: (batch, height, width)
        -   Output: (batch)
        
        _Returns_:  
        (torch.tensor): The lovasz hinge loss
        
        ##### [DiceLoss()](./pytorch_zoo/loss.py#L115)
        
        The dice loss for semantic segmentation
        
        Implementation adapted from https://www.kaggle.com/soulmachine/siim-deeplabv3
        
        ```python
        criterion = DiceLoss()
        loss = criterion(logits, targets)
        ```
        
        _Shape_:
        
        -   Input:
            -   logits: (batch, \*)
            -   targets: (batch, \*) _same as logits_
        -   Output: (1)
        
        _Returns_:  
        (torch.tensor): The dice loss
        
        ### Metrics
        
        ### Modules
        
        ##### [SqueezeAndExcitation(in_ch, r=16)](./pytorch_zoo/modules.py#L6)
        
        The channel-wise SE (Squeeze and Excitation) block from the [Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507) paper.
        
        Implementation adapted from https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/65939 and https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178
        
        ```python
        # in __init__()
        self.SE = SqueezeAndExcitation(in_ch, r=16)
        
        # in forward()
        x = self.SE(x)
        ```
        
        _Arguments_:  
        `in_ch` (int): The number of channels in the feature map of the input.  
        `r` (int): The reduction ratio of the intermidiate channels. Default: 16.
        
        _Shape_:
        
        -   Input: (batch, channels, height, width)
        -   Output: (batch, channels, height, width) (same shape as input)
        
        ##### [ChannelSqueezeAndSpatialExcitation(in_ch)](./pytorch_zoo/modules.py#L41)
        
        The sSE (Channel Squeeze and Spatial Excitation) block from the [Concurrent Spatial and Channel ‘Squeeze & Excitation’ in Fully Convolutional Networks](https://arxiv.org/abs/1803.02579) paper.
        
        Implementation adapted from https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178
        
        ```python
        # in __init__()
        self.sSE = ChannelSqueezeAndSpatialExcitation(in_ch)
        
        # in forward()
        x = self.sSE(x)
        ```
        
        _Arguments_:  
        `in_ch` (int): The number of channels in the feature map of the input.
        
        _Shape_:
        
        -   Input: (batch, channels, height, width)
        -   Output: (batch, channels, height, width) (same shape as input)
        
        ##### [ConcurrentSpatialAndChannelSqueezeAndChannelExcitation(in_ch)](./pytorch_zoo/modules.py#L71)
        
        The scSE (Concurrent Spatial and Channel Squeeze and Channel Excitation) block from the [Concurrent Spatial and Channel ‘Squeeze & Excitation’ in Fully Convolutional Networks](https://arxiv.org/abs/1803.02579) paper.
        
        Implementation adapted from https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178
        
        ```python
        # in __init__()
        self.scSE = ConcurrentSpatialAndChannelSqueezeAndChannelExcitation(in_ch, r=16)
        
        # in forward()
        x = self.scSE(x)
        ```
        
        _Arguments_:  
        `in_ch` (int): The number of channels in the feature map of the input.  
        `r` (int): The reduction ratio of the intermidiate channels. Default: 16.
        
        _Shape_:
        
        -   Input: (batch, channels, height, width)
        -   Output: (batch, channels, height, width) (same shape as input)
        
        ##### [GaussianNoise(0.1)](./pytorch_zoo/modules.py#L104)
        
        A gaussian noise module.
        
        ```python
        # in __init__()
        self.gaussian_noise = GaussianNoise(0.1)
        
        # in forward()
        if self.training:
            x = self.gaussian_noise(x)
        ```
        
        _Arguments_:  
        `stddev` (float): The standard deviation of the normal distribution. Default: 0.1.
        
        _Shape_:
        
        -   Input: (batch, \*)
        -   Output: (batch, \*) (same shape as input)
        
        ### Schedulers
        
        ##### [CyclicalMomentum(optimizer, base_momentum=0.8, max_momentum=0.9, step_size=2000, mode="triangular")](./pytorch_zoo/schedulers.py#L7)
        
        Pytorch's [cyclical learning rates](https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py), but for momentum, which leads to better results when used with cyclic learning rates, as shown in [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://arxiv.org/abs/1803.09820).
        
        ```python
        optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
        scheduler = torch.optim.CyclicMomentum(optimizer)
        data_loader = torch.utils.data.DataLoader(...)
        for epoch in range(10):
            for batch in data_loader:
                scheduler.batch_step()
                train_batch(...)
        ```
        
        _Arguments_:  
        `optimizer` (Optimizer): Wrapped optimizer.  
        `base_momentum` (float or list): Initial momentum which is the lower boundary in the cycle for each param groups. Default: 0.8  
        `max_momentum` (float or list): Upper boundaries in the cycle for each parameter group. scaling function. Default: 0.9  
        `step_size` (int): Number of training iterations per half cycle. Authors suggest setting step_size 2-8 x training iterations in epoch. Default: 2000  
        `mode` (str): One of {triangular, triangular2, exp_range}. Default: 'triangular'  
        `gamma` (float): Constant in 'exp_range' scaling function. Default: 1.0  
        `scale_fn` (function): Custom scaling policy defined by a single argument lambda function. Mode paramater is ignored Default: None  
        `scale_mode` (str): {'cycle', 'iterations'}. Defines whether scale_fn is evaluated on cycle number or cycle iterations (training iterations since start of cycle). Default: 'cycle'  
        `last_batch_iteration` (int): The index of the last batch. Default: -1
        
        ### Utils
        
        ##### [notify({'value1': 'Notification title', 'value2': 'Notification body'}, key)](./pytorch_zoo/utils.py#L13)
        
        Send a notification to your phone with IFTTT
        
        Setup a IFTTT webhook with https://medium.com/datadriveninvestor/monitor-progress-of-your-training-remotely-f9404d71b720
        
        ```python
        notify({'value1': 'Notification title', 'value2': 'Notification body'}, key=[YOUR_PRIVATE_KEY_HERE])
        ```
        
        _Arguments_:  
        `obj` (Object): Object to send to IFTTT  
        `key` ([type]): IFTTT webhook key
        
        ##### [seed_environment(seed=42)](./pytorch_zoo/utils.py#L25)
        
        Set random seeds for python, numpy, and pytorch to ensure reproducible research.
        
        ```python
        seed_envirionment(42)
        ```
        
        _Arguments_:  
        `seed` (int): The random seed to set.
        
        ##### [gpu_usage(device, digits=4)](./pytorch_zoo/utils.py#L39)
        
        Prints the amount of GPU memory currently allocated in GB.
        
        ```python
        gpu_usage(device, digits=4)
        ```
        
        _Arguments_:  
        `device` (torch.device, optional): The device you want to check. Defaults to device.  
        `digits` (int, optional): The number of digits of precision. Defaults to 4.
        
        ##### [n_params(model)](./pytorch_zoo/utils.py#L53)
        
        Return the number of parameters in a pytorch model.
        
        ```python
        print(n_params(model))
        ```
        
        _Arguments_:  
        `model` (nn.Module): The model to analyze.
        
        _Returns_:  
        (int): The number of parameters in the model.
        
        ##### [save_model(model, fold=0)](./pytorch_zoo/utils.py#L71)
        
        Save a trained pytorch model on a particular cross-validation fold to disk.
        
        Implementation adapted from https://github.com/floydhub/save-and-resume.
        
        ```python
        save_model(model, fold=0)
        ```
        
        _Arguments_:  
        `model` (nn.Module): The model to save.  
        `fold` (int): The cross-validation fold the model was trained on.
        
        ##### [load_model(model, fold=0)](./pytorch_zoo/utils.py#L84)
        
        Load a trained pytorch model saved to disk using `save_model`.
        
        ```python
        model = load_model(model, fold=0)
        ```
        
        _Arguments_:
        `model` (nn.Module): The model to save.  
        `fold` (int): Which saved model fold to load.
        
        _Returns_:  
        (nn.Module): The same model that was passed in, but with the pretrained weights loaded.
        
        ##### [save(obj, 'obj.pkl')](./pytorch_zoo/utils.py#L99)
        
        Save an object to disk.
        
        ```python
        save(tokenizer, 'tokenizer.pkl')
        ```
        
        _Arguments_:  
        `obj` (Object): The object to save.  
        `filename` (String): The name of the file to save the object to.
        
        ##### [load('obj.pkl')](./pytorch_zoo/utils.py#L110)
        
        Load an object saved to disk with `save`.
        
        ```python
        tokenizer = load('tokenizer.pkl')
        ```
        
        _Arguments_:  
        `path` (String): The path to the saved object.
        
        _Returns_:  
        (Object): The loaded object.
        
        ##### [masked_softmax(logits, mask, dim=-1)](./pytorch_zoo/utils.py#L124)
        
        A masked softmax module to correctly implement attention in Pytorch.
        
        Implementation adapted from: https://github.com/allenai/allennlp/blob/master/allennlp/nn/util.py
        
        ```python
        out = masked_softmax(logits, mask, dim=-1)
        ```
        
        _Arguments_:  
        `vector` (torch.tensor): The tensor to softmax.  
        `mask` (torch.tensor): The tensor to indicate which indices are to be masked and not included in the softmax operation.  
        `dim` (int, optional): The dimension to softmax over. Defaults to -1.  
        `memory_efficient` (bool, optional): Whether to use a less precise, but more memory efficient implementation of masked softmax. Defaults to False.  
        `mask_fill_value` ([type], optional): The value to fill masked values with if `memory_efficient` is `True`. Defaults to -1e32.
        
        _Returns_:  
        (torch.tensor): The masked softmaxed output
        
        ##### [masked_log_softmax(logits, mask, dim=-1)](./pytorch_zoo/utils.py#L175)
        
        A masked log-softmax module to correctly implement attention in Pytorch.
        
        Implementation adapted from: https://github.com/allenai/allennlp/blob/master/allennlp/nn/util.py
        
        ```python
        out = masked_log_softmax(logits, mask, dim=-1)
        ```
        
        _Arguments_:  
        `vector` (torch.tensor): The tensor to log-softmax.  
        `mask` (torch.tensor): The tensor to indicate which indices are to be masked and not included in the log-softmax operation.  
        `dim` (int, optional): The dimension to log-softmax over. Defaults to -1.
        
        _Returns_:  
        (torch.tensor): The masked log-softmaxed output
        
        ## Contributing
        
        This repository is still a work in progress, so if you find a bug, think there is something missing, or have any suggestions for new features or modules, feel free to open an issue or a pull request. Feel free to use the library or code from it in your own projects, and if you feel that some code used in this project hasn't been properly accredited, please open an issue.
        
        ## Authors
        
        -   _Bilal Khan_ - _Initial work_
        
        ## License
        
        This project is licensed under the MIT License - see the [license](LICENSE) file for details
        
        ## Acknowledgements
        
        This project contains code adapted from:
        
        -   https://github.com/bermanmaxim/LovaszSoftmax
        -   https://www.kaggle.com/aglotero/another-iou-metric
        -   https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/66178
        -   https://github.com/bckenstler/CLR
        -   https://github.com/floydhub/save-and-resume
        -   https://github.com/allenai/allennlp
        -   https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/94779
        
        This README is based on:
        
        -   https://github.com/mxbi/mlcrate
        -   https://github.com/athityakumar/colorls
        -   https://github.com/amitmerchant1990/electron-markdownify/blob/master/README.md
        -   https://github.com/rish-16/gpt2client
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
