Metadata-Version: 2.1
Name: hystoc
Version: 0.1
Summary: Toolkit for obtaining word-level confidences
Author-email: Karel Beneš <ibenes@fit.vutbr.cz>
License: MIT License
        
        Copyright (c) 2023 BUT Speech@FIT
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: repository, https://github.com/BUTSpeechFIT/hystoc
Keywords: Confidence estimation,ASR
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy

# hystoc
Getting confidences from any end-to-end systems, developed in context of Automatic Speech Recognition.
The underlying technique was previously sucessfuly applied to [semi-supervised learning in OCR](https://arxiv.org/abs/2104.13037).
Hystoc is oblivious to the underlying task, but please note that no special care is provided for non-monotonic tasks such as Machine Translation.

When using Hystoc, please cite (currently redacted because the paper is in a double-blind review).

## Installation

So far, installation is limited to manually downloading the package.

## Usage

To obtain confidences, Hystoc needs two inputs describing the competing hypotheses:

A text file with the desired level of tokenization given by whitespace:
```
uttA-1 Some example text
uttA-2 Mom example text
uttB-1 Nice bowl of rice
uttB-2 Rice bowl of nice
```

A score file with (possibly un-normalized) log-probabilities of the hypotheses.
```
uttA-1 -0.264534
uttA-2 -9.381741
uttB-1 -0.185739
uttB-2 -1.294320
```

Then, confidences can obtained with:
```
hystoc-confidences --temperature 1.0 hypotheses scores output
```

### Performing direct fusion with Hystoc
Hystoc also allows to directly fuse outputs of multiple systems into a single one.

To this end a list of pairs needs to be provided like this:
```
hystoc-fusion --confidence-file fused.txt --method normalize-per-system example/a.score example/a.txt example/b.score example/b.txt
```

Please note that our experiments did not show Hystoc fusion to consistently outperform Rover.
