"""
The **SapiensBench** algorithm is a structured benchmarking framework designed to evaluate the performance of language models in a consistent, automated,
and scalable way. It focuses on measuring how accurately different models respond to predefined input–output pairs, enabling objective comparison across multiple systems.

At its core, the algorithm operates by loading or generating a dataset composed of prompts (inputs) and their expected answers (outputs).
These datasets can come from external JSON files or be synthetically generated through specialized modules such as context-length evaluation or mathematical precision testing.
This flexibility allows the framework to assess models under different types of cognitive demands, from long-context reasoning to exact-answer problems.

Before evaluation begins, each data sample is normalized and validated to ensure consistency.
The system also dynamically generates a “system prompt” tailored to the language of the input, enforcing a strict answering policy:
models must return only the correct answer, with no explanations or additional text. This constraint is critical for eliminating ambiguity during evaluation and ensuring fair scoring.

During benchmarking, each model is tested iteratively across all input–output pairs.
For every prompt, the model may attempt multiple responses (controlled by the *n-shots* parameter), increasing robustness against stochastic variability.
The algorithm then compares the model’s response to the expected output using a normalization process that removes punctuation, accents,
and formatting differences. A response is considered correct if it matches or closely aligns with the expected answer under these normalized conditions.

Performance is quantified as a probability score, calculated as the ratio of correct answers to the total number of test cases.
This score is continuously updated during execution and ultimately represents the model’s accuracy on the given benchmark.
The results are saved in a structured JSON file and optionally visualized as a bar chart, allowing easy comparison between models.

One of the key strengths of SapiensBench is its modularity. It supports multiple model backends, including locally loaded models and API-based systems,
without requiring changes to the evaluation logic.
Additionally, its ability to normalize outputs and enforce strict response formats ensures that results are not biased by stylistic differences between models.

Another important benefit is reproducibility. By standardizing datasets, prompts, and evaluation criteria,
the framework enables consistent benchmarking across different environments and over time.
This makes it particularly useful for tracking model improvements or comparing newly released architectures.

The inclusion of synthetic data generators, such as those for context length and mathematical precision, further enhances its utility.
These generators create controlled testing scenarios that isolate specific capabilities, allowing developers to diagnose weaknesses and optimize models more effectively.

In summary, SapiensBench provides a reliable, extensible, and precise benchmarking solution for language models.
Its combination of strict evaluation rules, flexible data sourcing, multi-model support,
and clear performance metrics makes it a powerful tool for both research and production-level model assessment.
"""
# --------------------------> A SAPIENS TECHNOLOGY®️ PRODUCTION) <--------------------------
from setuptools import setup, find_packages
package_name = 'sapiens_bench'
version = '1.0.1'
setup(
    name=package_name,
    version=version,
    author='SAPIENS TECHNOLOGY',
    packages=find_packages(),
    install_requires=[
        'matplotlib',
        'utilities-nlp==7.0.7',
        'SAPI-ARCHITECTURE==1.0.5',
        'context-length==1.0.7',
        'math-precision==1.0.1'
    ],
    url='https://github.com/sapiens-technology/SapiensBench',
    license='Proprietary Software'
)
"""
The **SapiensBench** algorithm is a structured benchmarking framework designed to evaluate the performance of language models in a consistent, automated,
and scalable way. It focuses on measuring how accurately different models respond to predefined input–output pairs, enabling objective comparison across multiple systems.

At its core, the algorithm operates by loading or generating a dataset composed of prompts (inputs) and their expected answers (outputs).
These datasets can come from external JSON files or be synthetically generated through specialized modules such as context-length evaluation or mathematical precision testing.
This flexibility allows the framework to assess models under different types of cognitive demands, from long-context reasoning to exact-answer problems.

Before evaluation begins, each data sample is normalized and validated to ensure consistency.
The system also dynamically generates a “system prompt” tailored to the language of the input, enforcing a strict answering policy:
models must return only the correct answer, with no explanations or additional text. This constraint is critical for eliminating ambiguity during evaluation and ensuring fair scoring.

During benchmarking, each model is tested iteratively across all input–output pairs.
For every prompt, the model may attempt multiple responses (controlled by the *n-shots* parameter), increasing robustness against stochastic variability.
The algorithm then compares the model’s response to the expected output using a normalization process that removes punctuation, accents,
and formatting differences. A response is considered correct if it matches or closely aligns with the expected answer under these normalized conditions.

Performance is quantified as a probability score, calculated as the ratio of correct answers to the total number of test cases.
This score is continuously updated during execution and ultimately represents the model’s accuracy on the given benchmark.
The results are saved in a structured JSON file and optionally visualized as a bar chart, allowing easy comparison between models.

One of the key strengths of SapiensBench is its modularity. It supports multiple model backends, including locally loaded models and API-based systems,
without requiring changes to the evaluation logic.
Additionally, its ability to normalize outputs and enforce strict response formats ensures that results are not biased by stylistic differences between models.

Another important benefit is reproducibility. By standardizing datasets, prompts, and evaluation criteria,
the framework enables consistent benchmarking across different environments and over time.
This makes it particularly useful for tracking model improvements or comparing newly released architectures.

The inclusion of synthetic data generators, such as those for context length and mathematical precision, further enhances its utility.
These generators create controlled testing scenarios that isolate specific capabilities, allowing developers to diagnose weaknesses and optimize models more effectively.

In summary, SapiensBench provides a reliable, extensible, and precise benchmarking solution for language models.
Its combination of strict evaluation rules, flexible data sourcing, multi-model support,
and clear performance metrics makes it a powerful tool for both research and production-level model assessment.
"""
# --------------------------> A SAPIENS TECHNOLOGY®️ PRODUCTION) <--------------------------
