Metadata-Version: 2.4
Name: llm_insight_forge
Version: 0.1.2
Summary: A comprehensive toolkit for LLM evaluation, prompt engineering, fine-tuning, and inference optimization
Home-page: https://github.com/biswanathroul/llm_insight_forge
Author: Biswanath Roul
License: MIT
Project-URL: Homepage, https://github.com/biswanathroul/llm_insight_forge
Project-URL: Bug Tracker, https://github.com/biswanathroul/llm_insight_forge/issues
Project-URL: Documentation, https://llm-insight-forge.readthedocs.io/
Keywords: llm,nlp,evaluation,prompt-engineering,fine-tuning,quantization
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=1.10.0
Requires-Dist: transformers>=4.20.0
Requires-Dist: numpy>=1.19.0
Requires-Dist: scikit-learn>=1.0.0
Requires-Dist: nltk>=3.6.0
Requires-Dist: rouge-score>=0.1.0
Requires-Dist: datasets>=2.0.0
Requires-Dist: tqdm>=4.62.0
Provides-Extra: fine-tuning
Requires-Dist: peft>=0.4.0; extra == "fine-tuning"
Requires-Dist: accelerate>=0.20.0; extra == "fine-tuning"
Requires-Dist: bitsandbytes>=0.40.0; extra == "fine-tuning"
Provides-Extra: quantization
Requires-Dist: bitsandbytes>=0.40.0; extra == "quantization"
Requires-Dist: optimum>=1.8.0; extra == "quantization"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: isort>=5.10.0; extra == "dev"
Requires-Dist: flake8>=4.0.0; extra == "dev"
Requires-Dist: mypy>=0.940; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=4.4.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0.0; extra == "docs"
Requires-Dist: myst-parser>=0.18.0; extra == "docs"
Provides-Extra: all
Requires-Dist: peft>=0.4.0; extra == "all"
Requires-Dist: accelerate>=0.20.0; extra == "all"
Requires-Dist: bitsandbytes>=0.40.0; extra == "all"
Requires-Dist: optimum>=1.8.0; extra == "all"
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# LLM Insight Forge

A comprehensive toolkit for working with Large Language Models (LLMs), offering advanced capabilities for evaluation, prompt engineering, fine-tuning, and inference optimization.

[![PyPI version](https://badge.fury.io/py/llm-insight-forge.svg)](https://badge.fury.io/py/llm-insight-forge)
[![Python Version](https://img.shields.io/pypi/pyversions/llm-insight-forge.svg)](https://pypi.org/project/llm-insight-forge/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Features

LLM Insight Forge provides a modular toolkit for various LLM operations:

### 📊 Evaluation

- Text similarity metrics (BLEU, ROUGE)
- Semantic similarity using embeddings
- Factuality assessment
- Hallucination detection
- Bias detection
- Coherence and fluency scoring
- Comprehensive benchmarking

### ✨ Prompt Engineering

- Structured prompt templates
- Prompt optimization techniques
- Jailbreak detection and prevention
- Cross-model prompt compatibility

### 🔧 Fine-Tuning

- Dataset preparation and preprocessing
- Supervised fine-tuning (SFT)
- Parameter-efficient tuning methods (LoRA, P-Tuning, etc.)
- Training job management and monitoring

### ⚡ Inference Optimization

- Model quantization techniques
- Batched inference processing
- Caching strategies
- Hardware-specific optimizations

## Installation

```bash
pip install llm-insight-forge
```

For development:

```bash
pip install llm-insight-forge[dev]
```

For building documentation:

```bash
pip install llm-insight-forge[docs]
```

### Checking the Version

You can check the installed version using the command-line interface:

```bash
llm-insight-forge --version
```

Or from within Python:

```python
import llm_insight_forge as lif
print(lif.__version__)
```

## Quick Start

```python
import llm_insight_forge as lif

# Evaluate model responses
score = lif.evaluate_response(
    response="The Earth orbits around the Sun in 365.25 days.",
    reference="The Earth completes one orbit around the Sun in approximately 365.25 days.",
    metrics=["bleu", "semantic_similarity", "factuality"]
)
print(f"Evaluation score: {score}")

# Create and optimize prompts
template = lif.PromptTemplate(
    "Answer the following question about {topic}: {question}"
)
prompt = template.format(
    topic="astronomy",
    question="How long does it take for Earth to orbit the Sun?"
)
optimized_prompt = lif.optimize_prompt(prompt, target_model="gpt-4")

# Prepare datasets for fine-tuning
dataset = lif.prepare_dataset(
    data_path="path/to/data.jsonl",
    instruction_field="instruction",
    input_field="input",
    output_field="output"
)

# Train a model
lif.train_model(
    model_name="meta-llama/Llama-2-7b-hf",
    dataset=dataset,
    method="lora",
    output_dir="./fine_tuned_model"
)

# Optimize inference
quantized_model = lif.quantize_model(
    model_path="./fine_tuned_model",
    bits=4
)
```



## Example Scripts

Check out the `examples/` directory for more usage examples:

- `basic_evaluation.py`: Simple response evaluation workflow
- `advanced_metrics.py`: Using advanced hallucination and bias metrics
- `prompt_optimization.py`: Optimizing prompts for different models
- `fine_tuning_example.py`: Complete fine-tuning workflow
- `inference_optimization.py`: Quantization and batch inference

## Contributing

Contributions are welcome! Please check out our [contributing guidelines](CONTRIBUTING.md).

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Citation

If you use LLM Insight Forge in your research, please cite:

```bibtex
@software{biswanath2025llminsightforge,
  author = {Roul, Biswanath},
  title = {LLM Insight Forge: A Toolkit for LLM Evaluation and Optimization},
  year = {2025},
  url = {https://github.com/biswanathroul/llm_insight_forge}
}
```
