"""
The **SapiensBench** algorithm is a structured benchmarking framework designed to evaluate the performance of language models in a consistent, automated,
and scalable way. It focuses on measuring how accurately different models respond to predefined input–output pairs, enabling objective comparison across multiple systems.

At its core, the algorithm operates by loading or generating a dataset composed of prompts (inputs) and their expected answers (outputs).
These datasets can come from external JSON files or be synthetically generated through specialized modules such as context-length evaluation or mathematical precision testing.
This flexibility allows the framework to assess models under different types of cognitive demands, from long-context reasoning to exact-answer problems.

Before evaluation begins, each data sample is normalized and validated to ensure consistency.
The system also dynamically generates a “system prompt” tailored to the language of the input, enforcing a strict answering policy:
models must return only the correct answer, with no explanations or additional text. This constraint is critical for eliminating ambiguity during evaluation and ensuring fair scoring.

During benchmarking, each model is tested iteratively across all input–output pairs.
For every prompt, the model may attempt multiple responses (controlled by the *n-shots* parameter), increasing robustness against stochastic variability.
The algorithm then compares the model’s response to the expected output using a normalization process that removes punctuation, accents,
and formatting differences. A response is considered correct if it matches or closely aligns with the expected answer under these normalized conditions.

Performance is quantified as a probability score, calculated as the ratio of correct answers to the total number of test cases.
This score is continuously updated during execution and ultimately represents the model’s accuracy on the given benchmark.
The results are saved in a structured JSON file and optionally visualized as a bar chart, allowing easy comparison between models.

One of the key strengths of SapiensBench is its modularity. It supports multiple model backends, including locally loaded models and API-based systems,
without requiring changes to the evaluation logic.
Additionally, its ability to normalize outputs and enforce strict response formats ensures that results are not biased by stylistic differences between models.

Another important benefit is reproducibility. By standardizing datasets, prompts, and evaluation criteria,
the framework enables consistent benchmarking across different environments and over time.
This makes it particularly useful for tracking model improvements or comparing newly released architectures.

The inclusion of synthetic data generators, such as those for context length and mathematical precision, further enhances its utility.
These generators create controlled testing scenarios that isolate specific capabilities, allowing developers to diagnose weaknesses and optimize models more effectively.

In summary, SapiensBench provides a reliable, extensible, and precise benchmarking solution for language models.
Its combination of strict evaluation rules, flexible data sourcing, multi-model support,
and clear performance metrics makes it a powerful tool for both research and production-level model assessment.
"""
# --------------------------> A SAPIENS TECHNOLOGY®️ PRODUCTION) <--------------------------
class SapiensBench:
	def __init__(self, show_errors=True, display_error_point=False):
		try:
			self.__show_errors = bool(show_errors) if type(show_errors) in (bool, int, float) else True
			self.__display_error_point = bool(display_error_point) if type(display_error_point) in (bool, int, float) else False
			try:
				from warnings import simplefilter, filterwarnings
				from logging import getLogger, ERROR, disable, CRITICAL
				from os import environ
				from dotenv import load_dotenv
				simplefilter('ignore')
				filterwarnings('ignore')
				filterwarnings('ignore', category=UserWarning, module='torch.distributed')
				getLogger('torch.distributed.elastic.multiprocessing.redirects').setLevel(ERROR)
				environ['DISABLE_MODEL_SOURCE_CHECK'] = 'True'
				load_dotenv()
				disable(CRITICAL)
			except: pass
			from traceback import print_exc
			self.__print_exc = print_exc
			from SAPI_ARCHITECTURE import SAPI
			from utilities_nlp import UtilitiesNLP as SapiensUtilities
			self.__sapiens_utilities = SapiensUtilities(show_errors=self.__show_errors, display_error_point=self.__display_error_point)
			self.__SAPI_ARCHITECTURE = SAPI(show_errors=self.__show_errors, display_error_point=self.__display_error_point)
			self.__benchmarking_name = ''
			self.__inputs_outputs = []
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.__init__: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
	def __validate_input_output(self, dictionary={}):
		try:
			input_output = {'input': '', 'output': ''}
			if not dictionary: return input_output
			if type(input_output) != dict: return input_output
			dictionary = {str(key_name).lower(): key_value for key_name, key_value in dictionary.items()}
			input_keys = ['input', 'prompt', 'question', 'ask', 'asking', 'entry']
			output_keys = ['output', 'answer', 'response']
			for input_key in input_keys:
				input_string = str(dictionary.get(input_key, '')).strip()
				if input_string: input_output['input'] = input_string
			for output_key in output_keys:
				output_string = str(dictionary.get(output_key, '')).strip()
				if output_string: input_output['output'] = output_string
			return input_output
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.__validate_input_output: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return {'input': '', 'output': ''}
	def __get_system(self, prompt=''):
		try:
			system = ''
			prompt = str(prompt).strip()
			language = str(self.__sapiens_utilities.getLanguage(string=prompt)).lower().strip()
			system_en = "You are a “benchmarking” system that receives an input request and responds only with the correct answer in a short and objective way, without explanations. If the request is a question with alternatives, your answer must contain only the correct alternative repeated exactly as in the statement, without adding or removing anything at all."
			system_es = "Usted es un sistema de “benchmarking” que recibe una solicitud de entrada y responde únicamente con la respuesta correcta de manera breve y objetiva, sin explicaciones. Si la solicitud es una pregunta con alternativas, su respuesta debe contener solo la alternativa correcta repetida exactamente como en el enunciado, sin agregar ni quitar absolutamente nada."
			system_pt = "Você é um sistema de “benchmarking” que recebe uma solicitação de entrada e responde somente com a resposta correta de forma curta e objetiva, sem explicações. Se a solicitação for uma pergunta com alternativas, a sua resposta deve conter apenas a alternativa correta repetida exatamente como no enunciado, sem adicionar ou remover absolutamente nada."
			if language.startswith('en'): system = system_en
			elif language.startswith('es'): system = system_es
			elif language.startswith('pt'): system = system_pt
			if not system: system = self.__sapiens_utilities.translate(string=system_en, source_language='en', target_language=language)
			return system
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.__get_system: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return ''
	def __save_benchmark_chart(self, probabilities=[], chart_path='', benchmarking_name=''):
		try:
			saved_benchmark_chart = False
			probabilities = list(probabilities) if type(probabilities) in (tuple, list) else []
			chart_path = str(chart_path).strip()
			benchmarking_name = str(benchmarking_name).strip()
			if not probabilities: return saved_benchmark_chart
			if not benchmarking_name: benchmarking_name = 'Benchmarking Models'
			if not chart_path: chart_path = f'{benchmarking_name}.png'
			if '.' not in chart_path: chart_path += '.png'
			from os import makedirs
			from os.path import dirname, exists
			directory_path = dirname(chart_path)
			if directory_path and not exists(directory_path): makedirs(directory_path, exist_ok=True)
			from matplotlib import pyplot as plot
			model_names = [item['model_name'] for item in probabilities]
			probability_values = [item['probability'] for item in probabilities]
			colors_palette = ['#ee5071', '#f6b059', '#5fad57', '#6d8cdf', '#c57cdf']
			colors = [colors_palette[i % len(colors_palette)] for i in range(len(probabilities))]
			plot.figure()
			bars = plot.bar(model_names, probability_values, color=colors)
			max_value = max(probability_values)
			offset = max_value * 0.02 if max_value > 0 else 0.01
			plot.ylim(0, max_value * 1.1 if max_value > 0 else 1)
			for bar in bars:
				height = bar.get_height()
				plot.text(bar.get_x() + bar.get_width() / 2, height + offset, f'{height:.2f}', ha='center', va='bottom')
			plot.title(benchmarking_name)
			plot.xlabel('Model')
			plot.ylabel('Score')
			plot.xticks(rotation=45, ha='right')
			plot.tight_layout()
			plot.savefig(chart_path)
			plot.close()
			saved_benchmark_chart = True
			return saved_benchmark_chart
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.__save_benchmark_chart: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return False
	def loadDataset(self, database_path=''):
		try:
			loaded_json_database = False
			database_path = str(database_path).strip()
			if not database_path: return loaded_json_database
			from pathlib import Path
			path_object = Path(database_path)
			file_extension = path_object.suffix
			json_file = file_extension.lower().strip().endswith('.json')
			notice1 = f'The specified path does NOT correspond to a JSON file: {database_path}'
			notice2 = 'The list of objects was not found!'
			if not json_file or not path_object.is_file():
				if self.__show_errors: print(notice1)
				return loaded_json_database
			json_content, data = '', []
			with open(database_path, 'r', encoding='utf-8') as file: json_content =  str(file.read()).strip()
			dictionary = self.__sapiens_utilities.stringToDictionaryOrJSON(string=json_content)
			if type(dictionary) in (tuple, list): data = dictionary
			else:
				try: data = list(dictionary.get('data', []))
				except: data = []
				if not data:
					key_names = list(dictionary.keys())
					for key_name in key_names:
						key_value = dictionary.get(key_name, None)
						if type(key_value) in (tuple, list):
							data = key_value
							break
				if not data:
					if self.__show_errors: print(notice2)
					return loaded_json_database
			self.__benchmarking_name = str(path_object.stem).strip()
			self.__inputs_outputs = data
			loaded_json_database = len(self.__inputs_outputs) > 0
			return loaded_json_database
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.loadDataset: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return False
	def loadContextLengthBenchmarking(self, n_samples=10, n_tokens=10000):
		try:
			loaded_json_database = False
			n_samples = max(1, int(n_samples)) if type(n_samples) in (int, float) else 10
			n_tokens = max(1, int(n_tokens)) if type(n_tokens) in (int, float) else 10000
			from context_length import ContextLength
			context_length = ContextLength(show_errors=self.__show_errors, display_error_point=self.__display_error_point)
			return_dictionary = context_length.generateDatabase(n_samples=n_samples, n_tokens=n_tokens)
			self.__benchmarking_name = 'Context Length - Benchmarking'
			self.__inputs_outputs = list(return_dictionary.get('data', []))
			loaded_json_database = len(self.__inputs_outputs) > 0
			return loaded_json_database
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.loadContextLengthBenchmarking: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return False
	def loadMathPrecisionBenchmarking(self, n_samples=10):
		try:
			loaded_json_database = False
			n_samples = max(1, int(n_samples)) if type(n_samples) in (int, float) else 10
			from math_precision import MathPrecision
			math_precision = MathPrecision(show_errors=self.__show_errors, display_error_point=self.__display_error_point)
			return_dictionary = math_precision.generateDatabase(n_samples=n_samples)
			self.__benchmarking_name = 'Math Precision - Benchmarking'
			self.__inputs_outputs = list(return_dictionary.get('data', []))
			loaded_json_database = len(self.__inputs_outputs) > 0
			return loaded_json_database
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.loadMathPrecisionBenchmarking: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return False
	def benchmarking(self, models_list=[], output_path='', n_shots=1, progress=True):
		try:
			result_benchmarking = []
			models_list = list(models_list) if type(models_list) in (tuple, list) else []
			output_path = str(output_path).strip()
			n_shots = max(1, int(n_shots)) if type(n_shots) in (int, float) else 1
			progress = bool(progress) if type(progress) in (bool, int, float) else True
			if not models_list: return result_benchmarking
			if not self.__inputs_outputs:
				print('No valid database was loaded!')
				return result_benchmarking
			if not output_path: output_path = 'output'
			inputs_outputs_length = len(self.__inputs_outputs)
			models_length = len(models_list)
			def _normalize_text(text=''):
				from unicodedata import normalize, combining, category
				normalized_text = normalize('NFD', str(text).replace('*', '').replace('$', '').strip())
				return str(''.join(character for character in normalized_text if not combining(character) and not category(character).startswith('P'))).strip()
			probabilities = []
			from tqdm import tqdm
			from pathlib import Path
			for index, configuration in enumerate(models_list, start=1):
				correct_answers, final_probability = 0, 0.0
				if configuration and type(configuration) == dict:
					model_path = ''
					model_route, model_name = str(configuration.get('model_route', '')).strip(), ''
					if model_route: model_name = str(model_route.split('/')[-1]).strip()
					if not model_name:
						model_name, loaded = str(configuration.get('model_name', '')).strip(), False
						if model_name.startswith('replicate-'): model_name = str(model_name.split('replicate-')[-1]).strip()
						elif model_name.startswith('deepinfra-'): model_name = str(model_name.split('deepinfra-')[-1]).strip()
						if not model_name:
							for key_name in list(configuration.keys()):
								_key_name = str(key_name).lower().strip()
								if _key_name and _key_name.endswith('model_path'): model_path = str(configuration.get(_key_name, '')).strip()
								if model_path:
									loaded = self.__SAPI_ARCHITECTURE.loadModel(model_path=model_path, progress=progress)
									if not loaded and self.__show_errors: print(f'ERROR loading {model_name}.')
									model_name = str(Path(model_path).stem).strip()
							if not model_name: model_name = f'model-{index}'
					position = str(index).rjust(len(str(models_length)), '0')
					if progress: progress_bar = tqdm(total=1.0, desc=f'{position}/{models_length} {model_name}', disable=not progress)
					for n_pair, inputs_output in enumerate(self.__inputs_outputs, start=1):
						shots_count, hit_the_target = 0, False
						while shots_count < n_shots:
							current_probability = 0.0
							if inputs_output:
								inputs_output = self.__validate_input_output(dictionary=inputs_output)
								_input, _output = str(inputs_output.get('input', '')).strip(), str(inputs_output.get('output', '')).lower().strip()
								if _input and _output:
									_output = _normalize_text(text=_output)
									system = self.__get_system(prompt=_input)
									messages = [{'role': 'system', 'content': system}, {'role': 'user', 'content': _input}] if system else [{'role': 'user', 'content': _input}]
									try:
										if model_path: dictionary = self.__SAPI_ARCHITECTURE.inference(messages=messages, task_names=[], stream=False)
										else: dictionary = self.__sapiens_utilities.executeTask(messages=messages, task_names=[], config=configuration, stream=False)
										answer = str(dictionary.get('answer', '')).lower().strip()
									except: answer = ''
									if not answer and self.__show_errors: print(f'ERROR inferring {model_name}.')
									if answer and answer != _input.lower().strip():
										if '<think>' in answer and '</think>' in answer: answer = str(answer.split('</think>')[-1]).strip()
										answer = _normalize_text(text=answer)
										n_tokens = len(answer.split(chr(32)))
										if n_tokens < 2 and _output.startswith(answer):
											correct_answers += 1
											hit_the_target = True
										elif n_tokens > 1:
											answer_lines = answer.split('\n')
											for answer_line in answer_lines:
												answer_line = str(answer_line).strip()
												if answer_line:
													if answer_line.startswith(_output):
														correct_answers += 1
														hit_the_target = True
														break
													elif answer_line.endswith(_output):
														break
														correct_answers += 1
														hit_the_target = True
										current_probability = float(min(1.0, max(0.0, correct_answers/max(1, inputs_outputs_length))))
										if progress:
											progress_bar.set_postfix(pair=f'{n_pair}/{inputs_outputs_length}', shot=shots_count+1, probability=current_probability)
											progress_bar.n = current_probability
											progress_bar.refresh()
							shots_count += 1
							if hit_the_target: break
						if progress and current_probability > 0.0:
							progress_bar.set_postfix(pair=f'{n_pair}/{inputs_outputs_length}', shot=shots_count, probability=current_probability)
							progress_bar.n = current_probability
							progress_bar.refresh()
				final_probability = float(min(1.0, max(0.0, correct_answers/max(1, inputs_outputs_length))))
				probabilities.append({'model_name': model_name, 'probability': final_probability})
				if progress:
					progress_bar.refresh()
					progress_bar.close()
					extraction_path = self.__SAPI_ARCHITECTURE._SAPI__sapiens_sapi._SapiensSAPI__semantic_ai_with_pretrained_integration._SemanticAIWithPretrainedIntegration__extraction_path
					closed = self.__SAPI_ARCHITECTURE._SAPI__sapiens_sapi._SapiensSAPI__semantic_ai_with_pretrained_integration._SemanticAIWithPretrainedIntegration__delete_path(extraction_path) if model_path else True
					if not closed and self.__show_errors: print('ERROR closing the model.')
			result_benchmarking = probabilities
			def _save_dictionary_to_json(result_dictionary={}, file_path=''):
				from json import dump
				from os.path import dirname, exists
				from os import makedirs
				directory_path = dirname(file_path)
				if directory_path and not exists(directory_path): makedirs(directory_path, exist_ok=True)
				with open(file_path, 'w', encoding='utf-8') as file: dump(result_dictionary, file, ensure_ascii=False, indent=4)
				return exists(file_path)
			from pathlib import Path
			if '.' not in output_path: file_path = f'{output_path}.json'
			else: file_path = str(Path(output_path).with_suffix('.json'))
			saved_dictionary_to_json = _save_dictionary_to_json(result_dictionary=result_benchmarking, file_path=file_path)
			if output_path.endswith('.json'): output_path = str(Path(output_path).with_suffix('.png'))
			saved_benchmark_chart = self.__save_benchmark_chart(probabilities=probabilities, chart_path=output_path, benchmarking_name=f'{self.__benchmarking_name}\n{n_shots} shot(s)')
			if not saved_dictionary_to_json and self.__show_errors: print('ERROR saving the JSON.')
			if not saved_benchmark_chart and self.__show_errors: print('ERROR saving the chart.')
			return result_benchmarking
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.benchmarking: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return False
	def __close(self):
		try:
			if self.__sapiens_utilities: del self.__sapiens_utilities
			if self.__inputs_outputs: del self.__inputs_outputs
			return self.__SAPI_ARCHITECTURE.close()
		except Exception as error:
			try:
				if self.__show_errors:
					error_message = 'ERROR in SapiensBench.__close: '+str(error)
					print(error_message)
					try: self.__print_exc() if self.__display_error_point else None
					except: pass
			except: pass
			return False
"""
The **SapiensBench** algorithm is a structured benchmarking framework designed to evaluate the performance of language models in a consistent, automated,
and scalable way. It focuses on measuring how accurately different models respond to predefined input–output pairs, enabling objective comparison across multiple systems.

At its core, the algorithm operates by loading or generating a dataset composed of prompts (inputs) and their expected answers (outputs).
These datasets can come from external JSON files or be synthetically generated through specialized modules such as context-length evaluation or mathematical precision testing.
This flexibility allows the framework to assess models under different types of cognitive demands, from long-context reasoning to exact-answer problems.

Before evaluation begins, each data sample is normalized and validated to ensure consistency.
The system also dynamically generates a “system prompt” tailored to the language of the input, enforcing a strict answering policy:
models must return only the correct answer, with no explanations or additional text. This constraint is critical for eliminating ambiguity during evaluation and ensuring fair scoring.

During benchmarking, each model is tested iteratively across all input–output pairs.
For every prompt, the model may attempt multiple responses (controlled by the *n-shots* parameter), increasing robustness against stochastic variability.
The algorithm then compares the model’s response to the expected output using a normalization process that removes punctuation, accents,
and formatting differences. A response is considered correct if it matches or closely aligns with the expected answer under these normalized conditions.

Performance is quantified as a probability score, calculated as the ratio of correct answers to the total number of test cases.
This score is continuously updated during execution and ultimately represents the model’s accuracy on the given benchmark.
The results are saved in a structured JSON file and optionally visualized as a bar chart, allowing easy comparison between models.

One of the key strengths of SapiensBench is its modularity. It supports multiple model backends, including locally loaded models and API-based systems,
without requiring changes to the evaluation logic.
Additionally, its ability to normalize outputs and enforce strict response formats ensures that results are not biased by stylistic differences between models.

Another important benefit is reproducibility. By standardizing datasets, prompts, and evaluation criteria,
the framework enables consistent benchmarking across different environments and over time.
This makes it particularly useful for tracking model improvements or comparing newly released architectures.

The inclusion of synthetic data generators, such as those for context length and mathematical precision, further enhances its utility.
These generators create controlled testing scenarios that isolate specific capabilities, allowing developers to diagnose weaknesses and optimize models more effectively.

In summary, SapiensBench provides a reliable, extensible, and precise benchmarking solution for language models.
Its combination of strict evaluation rules, flexible data sourcing, multi-model support,
and clear performance metrics makes it a powerful tool for both research and production-level model assessment.
"""
# --------------------------> A SAPIENS TECHNOLOGY®️ PRODUCTION) <--------------------------
