Metadata-Version: 2.4
Name: context-extension
Version: 0.1.2
Summary: Zero-Training Context Extension for Transformer Encoders via Nonlinear Absolute Positional Embeddings Interpolation
Author-email: Ivan Danylenko <kowd.pauuh@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/Kowd-PauUh
Project-URL: Repository, https://github.com/Kowd-PauUh/encoders-context-extension
Keywords: sentence transformers,context extension
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: torch
Requires-Dist: sentence-transformers
Requires-Dist: scipy
Dynamic: license-file

# Zero-Training Context Extension for Transformer Encoders via Nonlinear Absolute Positional Embeddings Interpolation

Official implementation of "Zero-Training Context Extension for Transformer Encoders via Nonlinear Absolute Positional Embeddings Interpolation". Paper preprint is coming soon.

This implementation currently supports only models compatible with [Sentence Transformers](https://www.sbert.net/) library.

## Models

Models are available at HuggingFace:

| Model                                                                                   | Context length | Language |
| --------------------------------------------------------------------------------------- | -------------- | -------- |
| [idanylenko/e5-large-v2-ctx1024](https://huggingface.co/idanylenko/e5-large-v2-ctx1024) | 1024           | English  |

## Installation

To install the package, use pip:

```bash
pip install "context-extension>=0.1.1"
```

## Usage

After installing the package you may use `extend-context-spline` (recommended) or `extend-context-linear` scripts for embeddings interpolation.

### Spline Interpolation

Use this for smooth, nonlinear interpolation to support arbitrary context lengths:

```bash
extend-context-spline \
  --model_name_or_path="intfloat/e5-large-v2" \
  --max_seq_length=1024 \
  --embeddings_attr_name="embeddings.position_embeddings" \
  --offset=0 \
  --output_dir="intfloat/e5-large-v2-ctx1024-spline"
```

### Linear Interpolation

Use this to double the model's positional embedding range using linear averaging between consecutive embeddings:

```bash
extend-context-linear \
  --model_name_or_path="intfloat/e5-large-v2" \
  --embeddings_attr_name="embeddings.position_embeddings" \
  --offset=0 \
  --output_dir="intfloat/e5-large-v2-ctx1024-linear"
```

Both commands modify the positional embeddings of a model and save the updated model to the specified directory. You can then upload the resulting model to Hugging Face or use it locally for inference.

For models like RoBERTa that use special tokens in the first few positions, remember to set appropriate `--offset` argument.
