med3pa subpackage — Med3pa documentation 0.1.0 documentation

# med3pa subpackage

## Overview

The `med3pa` subpackage represents a modularized and enhanced version of the original “MED3PA code”, designed to augment clinical decision-making by providing a robust framework for evaluating and managing model uncertainty in healthcare applications. It introduces a sophisticated approach to assessing model performance that transcends traditional global metrics, focusing instead on the concept of predictive confidence at both individual and aggregated levels.

### Key Components:

- Individualized Predictive Confidence (IPC): This component employs `regression` models to estimate the predictive confidence for individual data points. The IPC model, adaptable with various regression algorithms like `Random Forest`, is particularly aimed at quantifying the uncertainty associated with each prediction, allowing for a detailed analysis of model reliability.

- Aggregated Predictive Confidence (APC): Contrasting with IPC, APC focuses on groups of similar data points, using a `decision tree regressor` to analyze uncertainty in aggregated profiles. This method helps identify patterns or groups where the model’s performance might be suboptimal, facilitating targeted improvements.

- Mixed Predictive Confidence (MPC): This model combines results from IPC and APC to derive a composite measure of confidence. MPC values are then used to further scrutinize the model’s performance and to identify problematic profiles where the base model might fail.

### Advanced Analysis with MDR:

The med3pa approach innovatively employs metrics by declaration rate (MDR), which evaluates model metrics at various confidence thresholds. This methodology not only highlights how the model performs across different confidence levels but also pinpoints specific profiles (groups of data points) that might be problematic. By doing so, it aids in the proactive identification and mitigation of potential model failures.

### Extensibility for Integration:

Our package is meticulously designed to facilitate and accommodate integration with other methods, such as `Detectron`. This extensibility allows `med3pa` to assess shifts in data distributions, especially focusing on the problematic profiles identified through MPC. Such integration enhances the comprehensive assessment of how external changes or shifts might affect model reliability over time, ensuring that the model remains robust and accurate in dynamic clinical environments.

In essence, `med3pa` is dedicated to advancing safer clinical deployments by providing tools that not only predict outcomes but also critically analyze and improve the understanding of where and why predictions might fail. This helps ensure that deployed models are not just effective but are also reliable and trustworthy in real-world settings.

this subpackage includes the following classes:

## uncertainty module

This module handles the computation of uncertainty metrics. It defines an abstract base class `UncertaintyMetric` and concrete implementations such as `AbsoluteError` for calculating uncertainty based on the difference between predicted probabilities and actual outcomes. An `UncertaintyCalculator` class is provided, which allows users to specify which uncertainty metric to use, thereby facilitating the use of customized uncertainty metrics for different analytical needs.

class MED3pa.med3pa.uncertainty.AbsoluteError

  Bases: `UncertaintyMetric`

  Concrete implementation of the UncertaintyMetric class using absolute error.

  calculate(predicted_prob: ndarray, y_true: ndarray) → ndarray

    Calculates the absolute error between predicted probabilities and true labels, providing a measure of prediction accuracy.

    Parameters:

      - x (np.ndarray) – Input features (not used in this metric but included for interface consistency).

      - predicted_prob (np.ndarray) – Predicted probabilities.

      - y_true (np.ndarray) – True labels.

    Returns:

      Absolute errors between predicted probabilities and true labels.

    Return type:

      np.ndarray

class MED3pa.med3pa.uncertainty.UncertaintyCalculator(metric_name: str)

  Bases: `object`

  Class for calculating uncertainty using a specified uncertainty metric.

  calculate_uncertainty(x: ndarray, predicted_prob: ndarray, y_true: ndarray) → ndarray

    Calculates uncertainty for a set of predictions using the configured uncertainty metric.

    Parameters:

      - x (np.ndarray) – Input features.

      - predicted_prob (np.ndarray) – Predicted probabilities.

      - y_true (np.ndarray) – True labels.

    Returns:

      Uncertainty values for each prediction, computed using the specified metric.

    Return type:

      np.ndarray

  metric_mapping = {'absolute_error': <class 'MED3pa.med3pa.uncertainty.AbsoluteError'>}
  classmethod supported_metrics() → list

    Returns a list of supported uncertainty metrics.

    Returns:

      A list of strings representing the names of the supported uncertainty metrics.

    Return type:

      list

class MED3pa.med3pa.uncertainty.UncertaintyMetric

  Bases: `ABC`

  Abstract base class for uncertainty metrics. Defines the structure that all uncertainty metrics should follow.

  abstract calculate(predicted_prob: ndarray, y_true: ndarray) → ndarray

    Calculates the uncertainty metric based on input observations, predicted probabilities, and true labels.

    Parameters:

      - x (np.ndarray) – Input observations.

      - predicted_prob (np.ndarray) – Predicted probabilities by the model.

      - y_true (np.ndarray) – True labels.

    Returns:

      An array of uncertainty values for each prediction.

    Return type:

      np.ndarray

## models module

Defines the models used within the med3pa framework. It includes classes for Individualized Predictive Confidence (IPC) models that predict uncertainty at an individual level, where the regressor type can be specified by the user. Additionally, it includes Aggregated Predictive Confidence (APC) models that predict uncertainty for groups of similar data points, and Mixed Predictive Confidence (MPC) models that combine the predictions from IPC and APC models.

class MED3pa.med3pa.models.APCModel(features: List[str], params: Dict[str, Any] | None = None, tree_file_path: str | None = None, pretrained_model: str | None = None)

  Bases: `object`

  APCModel class used to predict the Aggregated predicted confidence. ie, the base model confidence for a group of similar data points.

  default_params = {'max_depth': 3, 'min_samples_leaf': 1, 'random_state': 54288}
  evaluate(X: ndarray, y: ndarray, eval_metrics: List[str], print_results: bool = False) → Dict[str, float]

    Evaluates the model using specified metrics.

    Parameters:

      - X (np.ndarray) – observations for evaluation.

      - y (np.ndarray) – True labels for evaluation.

      - eval_metrics (List[str]) – Metrics to use for evaluation.

      - print_results (bool) – Whether to print the evaluation results.

    Returns:

      A dictionary with metric names and their evaluated scores.

    Return type:

      Dict[str, float]

  get_info() → Dict[str, Any]

    Returns information about the APCModel instance.

    Returns:

      A dictionary containing the model name, parameters, whether the model was optimized, and other relevant details.

    Return type:

      Dict[str, Any]

  load_model(file_path: str) → None

    Loads a pre-trained model from a pickle file.

    Parameters:

      file_path (str) – The path to the pickle file.

  load_tree(file_path: str) → None

    Loads the tree structure from a JSON file and initializes the tree representation.

    Parameters:

      file_path (str) – The file path from which the tree structure will be loaded.

  optimize(param_grid: dict, cv: int, x: ndarray, error_prob: ndarray, sample_weight: ndarray = None) → None

    Optimizes the model parameters using GridSearchCV.

    Parameters:

      - param_grid (Dict[str, Any]) – The parameter grid to explore.

      - cv (int) – The number of cross-validation folds.

      - x (np.ndarray) – Training data.

      - y (np.ndarray) – Target data.

      - sample_weight (Optional[np.ndarray]) – Weights for the training samples.

  predict(X: ndarray) → ndarray

    Predicts error probabilities using the tree representation for the given input observations.

    Parameters:

      - x (np.ndarray) – Feature matrix for which to predict error probabilities.

      - depth (Optional[int]) – The maximum depth of the tree to use for predictions.

    Returns:

      Predicted error probabilities based on the aggregated confidence levels.

    Return type:

      np.ndarray

  print_decision_tree_structure(feature_names=None)

    Prints the structure of a trained DecisionTreeRegressor.

    Args: tree_model: The trained DecisionTreeRegressor model. feature_names: List of feature names to use in the output. Default is None.

  save_model(file_path: str) → None

    Saves the trained model to a pickle file.

    Parameters:

      file_path (str) – The path to the file where the model will be saved.

  classmethod supported_models_params() → Dict[str, Dict[str, Any]]

    Returns a dictionary containing the supported models and their parameters and grid search parameters.

    Returns:

      A dictionary with model names as keys and another dictionary as value containing

        ’params’ and ‘grid_search_params’ for each model.

    Return type:

      Dict[str, Dict[str, Any]]

  supported_params = {'DecisionTreeRegressor': {'grid_params': [{'default': ['squared_error', 'friedman_mse', 'absolute_error', 'poisson'], 'description': 'The function to measure the quality of a split.', 'name': 'criterion', 'type': 'string'}, {'default': ['best', 'random'], 'description': 'The strategy used to choose the split at each node.', 'name': 'splitter', 'type': 'string'}, {'default': [2, 3, 4], 'description': 'The maximum depth of the tree. Increasing this value will make the model more complex.', 'name': 'max_depth', 'type': 'int'}, {'default': [2, 5, 10], 'description': 'The minimum number of samples required to split an internal node.', 'name': 'min_samples_split', 'type': 'int'}, {'default': [5, 10, 15, 20], 'description': 'The minimum number of samples required to be at a leaf node.', 'name': 'min_samples_leaf', 'type': 'int'}, {'default': [0.0, 0.1, 0.2], 'description': 'The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.', 'name': 'min_weight_fraction_leaf', 'type': 'float'}, {'default': ['auto', 'sqrt', 'log2'], 'description': 'The number of features to consider when looking for the best split.', 'name': 'max_features', 'type': 'string'}, {'default': [10, 20, 30], 'description': 'Grow trees with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity.', 'name': 'max_leaf_nodes', 'type': 'int'}, {'default': [0.0, 0.1, 0.2], 'description': 'A node will be split if this split induces a decrease of the impurity greater than or equal to this value.', 'name': 'min_impurity_decrease', 'type': 'float'}, {'default': [0.0, 0.01, 0.1], 'description': 'Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen.', 'name': 'ccp_alpha', 'type': 'float'}], 'params': [{'choices': ['squared_error', 'friedman_mse', 'absolute_error', 'poisson'], 'default': 'squared_error', 'description': 'The function to measure the quality of a split.', 'name': 'criterion', 'type': 'string'}, {'choices': ['best', 'random'], 'default': 'best', 'description': 'The strategy used to choose the split at each node.', 'name': 'splitter', 'type': 'string'}, {'default': 4, 'description': 'The maximum depth of the tree. Increasing this value will make the model more complex.', 'name': 'max_depth', 'type': 'int'}, {'default': 2, 'description': 'The minimum number of samples required to split an internal node.', 'name': 'min_samples_split', 'type': 'int'}, {'default': 1, 'description': 'The minimum number of samples required to be at a leaf node.', 'name': 'min_samples_leaf', 'type': 'int'}, {'default': 0.0, 'description': 'The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.', 'name': 'min_weight_fraction_leaf', 'type': 'float'}, {'choices': ['auto', 'sqrt', 'log2'], 'default': 'auto', 'description': 'The number of features to consider when looking for the best split.', 'name': 'max_features', 'type': 'string'}, {'default': 100, 'description': 'Grow trees with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity.', 'name': 'max_leaf_nodes', 'type': 'int'}, {'default': 0.0, 'description': 'A node will be split if this split induces a decrease of the impurity greater than or equal to this value.', 'name': 'min_impurity_decrease', 'type': 'float'}, {'default': 0.0, 'description': 'Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen.', 'name': 'ccp_alpha', 'type': 'float'}]}}
  train(x: ndarray, error_prob: ndarray) → None

    Trains the model using the provided data and error probabilities and builds the tree representation.

    Parameters:

      - x (np.ndarray) – Feature matrix for training.

      - error_prob (np.ndarray) – Error probabilities corresponding to each training instance.

class MED3pa.med3pa.models.IPCModel(model_name: str = 'RandomForestRegressor', params: Dict[str, Any] | None = None, pretrained_model: str | None = None)

  Bases: `object`

  IPCModel class used to predict the Individualized predicted confidence. ie, the base model confidence for each data point.

  default_params = {'random_state': 54288}
  evaluate(X: ndarray, y: ndarray, eval_metrics: List[str], print_results: bool = False) → Dict[str, float]

    Evaluates the model using specified metrics.

    Parameters:

      - X (np.ndarray) – observations for evaluation.

      - y (np.ndarray) – True labels for evaluation.

      - eval_metrics (List[str]) – Metrics to use for evaluation.

      - print_results (bool) – Whether to print the evaluation results.

    Returns:

      A dictionary with metric names and their evaluated scores.

    Return type:

      Dict[str, float]

  get_info() → Dict[str, Any]

    Returns information about the IPCModel instance.

    Returns:

      A dictionary containing the model name, parameters, whether the model was optimized, and other relevant details.

    Return type:

      Dict[str, Any]

  load_model(file_path: str) → None

    Loads a pre-trained model from a pickle file.

    Parameters:

      file_path (str) – The path to the pickle file.

  optimize(param_grid: dict, cv: int, x: ndarray, error_prob: ndarray, sample_weight: ndarray = None) → None

    Optimizes the model parameters using GridSearchCV.

    Parameters:

      - param_grid (Dict[str, Any]) – The parameter grid to explore.

      - cv (int) – The number of cross-validation folds.

      - x (np.ndarray) – Training data.

      - y (np.ndarray) – Target data.

      - sample_weight (Optional[np.ndarray]) – Weights for the training samples.

  predict(x: ndarray) → ndarray

    Predicts error probabilities for the given input observations using the trained model.

    Parameters:

      x (np.ndarray) – Feature matrix for which to predict error probabilities.

    Returns:

      Predicted error probabilities.

    Return type:

      np.ndarray

  save_model(file_path: str) → None

    Saves the trained model to a pickle file.

    Parameters:

      file_path (str) – The path to the file where the model will be saved.

  classmethod supported_ipc_models() → list

    Returns a list of supported IPC models.

    Returns:

      A list of supported regression model names.

    Return type:

      list

  classmethod supported_models_params() → Dict[str, Dict[str, Any]]

    Returns a dictionary containing the supported models and their parameters and grid search parameters.

    Returns:

      A dictionary with model names as keys and another dictionary as value containing

        ’params’ and ‘grid_search_params’ for each model.

    Return type:

      Dict[str, Dict[str, Any]]

  supported_regressors_mapping = {'RandomForestRegressor': <class 'MED3pa.models.concrete_regressors.RandomForestRegressorModel'>}
  supported_regressos_params = {'RandomForestRegressor': {'grid_params': [{'default': [10, 20, 50, 100, 200], 'description': 'The number of trees in the forest.', 'name': 'n_estimators', 'type': 'int'}, {'default': [2, 3, 4], 'description': 'The maximum depth of the tree. Increasing this value will make the model more complex.', 'name': 'max_depth', 'type': 'int'}, {'default': [2, 5, 10], 'description': 'The minimum number of samples required to split an internal node.', 'name': 'min_samples_split', 'type': 'int'}, {'default': [5, 10, 15, 20], 'description': 'The minimum number of samples required to be at a leaf node.', 'name': 'min_samples_leaf', 'type': 'int'}, {'default': ['auto', 'sqrt', 'log2'], 'description': 'The number of features to consider when looking for the best split.', 'name': 'max_features', 'type': 'string'}, {'default': [True, False], 'description': 'Whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree.', 'name': 'bootstrap', 'type': 'bool'}], 'params': [{'default': 100, 'description': 'The number of trees in the forest.', 'name': 'n_estimators', 'type': 'int'}, {'choices': ['squared_error', 'absolute_error', 'poisson'], 'default': 'squared_error', 'description': 'The function to measure the quality of a split.', 'name': 'criterion', 'type': 'string'}, {'default': 4, 'description': 'The maximum depth of the tree. Increasing this value will make the model more complex.', 'name': 'max_depth', 'type': 'int'}, {'default': 2, 'description': 'The minimum number of samples required to split an internal node.', 'name': 'min_samples_split', 'type': 'int'}, {'default': 1, 'description': 'The minimum number of samples required to be at a leaf node.', 'name': 'min_samples_leaf', 'type': 'int'}, {'default': 0.0, 'description': 'The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node.', 'name': 'min_weight_fraction_leaf', 'type': 'float'}, {'choices': ['auto', 'sqrt', 'log2'], 'default': 'auto', 'description': 'The number of features to consider when looking for the best split.', 'name': 'max_features', 'type': 'string'}, {'default': 100, 'description': 'Grow trees with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity.', 'name': 'max_leaf_nodes', 'type': 'int'}, {'default': 0.0, 'description': 'A node will be split if this split induces a decrease of the impurity greater than or equal to this value.', 'name': 'min_impurity_decrease', 'type': 'float'}, {'default': True, 'description': 'Whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree.', 'name': 'bootstrap', 'type': 'bool'}, {'default': False, 'description': 'Whether to use out-of-bag samples to estimate the generalization score.', 'name': 'oob_score', 'type': 'bool'}, {'default': 1, 'description': 'The number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context.', 'name': 'n_jobs', 'type': 'int'}, {'default': 42, 'description': 'Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True) and the sampling of the features to consider when looking for the best split at each node (if max_features < n_features).', 'name': 'random_state', 'type': 'int'}, {'default': 0, 'description': 'Controls the verbosity when fitting and predicting.', 'name': 'verbose', 'type': 'int'}, {'default': False, 'description': 'When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new forest.', 'name': 'warm_start', 'type': 'bool'}, {'default': 0.0, 'description': 'Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen.', 'name': 'ccp_alpha', 'type': 'float'}, {'default': 0.9, 'description': 'If bootstrap is True, the number of samples to draw from X to train each base estimator.', 'name': 'max_samples', 'type': 'float'}]}}
  train(x: ndarray, error_prob: ndarray) → None

    Trains the model on the provided training data and error probabilities.

    Parameters:

      - x (np.ndarray) – Feature matrix for training.

      - error_prob (np.ndarray) – Error probabilities corresponding to each training instance.

  underlying_models_mapping = {'RandomForestRegressor': <class 'sklearn.ensemble._forest.RandomForestRegressor'>}

class MED3pa.med3pa.models.MPCModel(IPC_values: ndarray = None, APC_values: ndarray = None)

  Bases: `object`

  MPCModel class used to predict the Mixed predicted confidence. ie, the minimum between the APC and IPC values.

  predict() → ndarray

    Combines IPC and APC values to predict MPC values.

    Returns:

      Combined MPC values.

    Return type:

      np.ndarray

## tree module

Manages the tree representation for the APC model. It includes the `TreeRepresentation` class which handles the construction and manipulation of decision trees and `TreeNode` class that represents a node in the tree. This module is crucial for profiling aggregated data and extracting valuable insights

class MED3pa.med3pa.tree.TreeRepresentation(features: list)

  Bases: `object`

  Represents the structure of a decision tree for a given set of features.

  build_tree(dtr: DecisionTreeRegressorModel, X: DataFrame, y: Series, node_id: int = 0, path: list = ['*']) → _TreeNode

    Recursively builds the tree representation starting from the specified node.

    Parameters:

      - dtr (DecisionTreeRegressorModel) – Trained decision tree regressor model.

      - X (DataFrame) – Training data observations.

      - y (Series) – Training data labels.

      - node_id (int) – Node ID to start building from. Defaults to 0.

      - path (Optional[List[str]]) – Path to the current node. Defaults to [‘*’].

    Returns:

      The root node of the tree representation.

    Return type:

      _TreeNode

  get_all_nodes() → list

    Retrieves all nodes from the tree with their paths.

    Returns:

      A list of dictionaries representing nodes with their paths.

    Return type:

      List[dict]

  get_all_profiles(min_ca: float = 0, min_samples_ratio: float = 0) → list

    Retrieves all profiles from the tree that meet the minimum criteria for value and sample ratio.

    Parameters:

      - min_ca (float) – Minimum value threshold for profiles. Defaults to 0.

      - min_samples_ratio (float) – Minimum sample ratio threshold for profiles. Defaults to 0.

    Returns:

      A list of Profile instances meeting the specified criteria.

    Return type:

      List[Profile]

  save_tree(file_path: str) → None

    Saves the tree structure to a JSON file.

    Parameters:

      file_path (str) – The file path where the tree structure will be saved.

MED3pa.med3pa.tree.to_serializable(obj: Any, additional_arg: Any = None) → Any

  Convert an object to a JSON-serializable format.

  Parameters:

    obj (Any) – The object to convert.

  Returns:

    The JSON-serializable representation of the object.

  Return type:

    Any

## Profiles module

Handles the management and storage of profiles derived from the tree representation. It defines a `Profile` class to encapsulate metrics and values associated with a specific node in the tree and a `ProfilesManager` class to manage collections of profiles and track lost profiles during analysis.

class MED3pa.med3pa.profiles.Profile(node_id: int, path: list[str])

  Bases: `object`

  Represents a profile containing metrics and values associated with a specific node.

  to_dict(save_all: bool = True)

    Converts the Profile instance into a dictionary format suitable for serialization.

    Returns:

      A dictionary representation of the Profile instance including the node ID, path, mean value,

        metrics, and any detectron results.

    Return type:

      dict

  update_detectron_results(detectron_results: dict)

    Updates the detectron results associated with this profile.

    Parameters:

      detectron_results (dict) – The results from the Detectron experiment to be added to the profile.

  update_metrics_results(metrics: dict)

    Updates the metrics associated with this profile.

    Parameters:

      detectron_results (dict) – The results from the Detectron experiment to be added to the profile.

  update_node_information(info: dict)

    Updates the information associated with this profile.

    Parameters:

      info (dict) – The updated node information.

class MED3pa.med3pa.profiles.ProfilesManager(features: list[str])

  Bases: `object`

  Manages the records of profiles and lost profiles based on declaration rates and minimal samples ratio.

  get_lost_profiles(min_samples_ratio=None, dr=None)

    Retrieves lost profiles based on the specified minimum sample ratio and dr value.

    Parameters:

      - dr (int) – Desired declaration rate as a percentage.

      - min_samples_ratio (int) – Minimum samples ratio.

  get_profiles(min_samples_ratio: int = None, dr: int = None)

    Retrieves profiles based on the specified minimum sample ratio and dr value.

    Parameters:

      - dr (int) – Desired declaration rate as a percentage.

      - min_samples_ratio (int) – Minimum samples ratio.

  insert_lost_profiles(dr: int, min_samples_ratio: int, profiles: list[Profile])

    Inserts lost profiles into the records under a specific dr value and minimum sample ratio.

    Parameters:

      - dr (int) – Desired declaration rate as a percentage.

      - min_samples_ratio (int) – Minimum samples ratio.

      - profiles (list[Profile]) – The profiles to insert.

  insert_profiles(dr: int, min_samples_ratio: int, profiles: list[Profile])

    Inserts profiles into the records under a specific dr value and minimum sample ratio.

    Parameters:

      - dr (int) – Desired declaration rate as a percentage.

      - min_samples_ratio (int) – Minimum samples ratio.

      - profiles (list[Profile]) – The profiles to insert.

  static transform_to_profiles(profiles_list: list[dict], to_dict: bool = False)

    Transforms a list of profile data into instances of the Profile class or dictionaries.

    Parameters:

      - profiles_list (list[dict]) – List of profiles data.

      - to_dict (bool, optional) – If True, transforms profiles to dictionaries. Defaults to True.

    Returns:

      List of transformed profiles.

    Return type:

      list[Union[dict, Profile]]

## MDR module

Contains functionality for calculating metrics based on the predicted confidence and declaration rates (MDR). The `MDRCalculator` class offers methods to assess model performance across different declaration rates, and to extract problematic profiles under specific declaration rates.

class MED3pa.med3pa.mdr.MDRCalculator

  Bases: `object`

  Class to calculate various metrics and profiles for the MED3PA method.

  static calc_metrics_by_dr(dataset: MaskedDataset, confidence_scores: ndarray, metrics_list: list)

    Calculate metrics by declaration rates (DR), evaluating model performance at various thresholds of predicted accuracies.

    Parameters:

      - dataset (MaskedDataset) – The dataset to filter.

      - confidence_scores (np.ndarray) – the confidence scores used for filtering.

      - metrics_list (list) – List of metric names to be calculated (e.g., ‘AUC’, ‘Accuracy’).

      - set (str) – the set to calculate the metrics on.

    Returns:

      A dictionary containing metrics computed for each declaration rate from 100% to 0%, including metrics and population percentage.

    Return type:

      dict

  static calc_metrics_by_profiles(profiles_manager, dataset: MaskedDataset, features: list, confidence_scores: ndarray, min_samples_ratio: int, metrics_list)

    Calculates various metrics for different profiles and declaration rates based on provided datasets.

    Parameters:

      - profiles_manager (ProfilesManager) – Manager handling profiles.

      - dataset (MaskedDataset) – the dataset to use.

      - set (str) – the set to calculate the metrics on.

      - metrics_list (list) – List of metrics to calculate.

  calc_profiles(tree: TreeRepresentation, dataset: MaskedDataset, features: list, confidence_scores: ndarray, min_samples_ratio: int) → Dict[int, float]

    Calculates profiles for different declaration rates and minimum sample ratios. This method assesses how profiles change across different confidence levels derived from predicted accuracies.

    Parameters:

      - profiles_manager (ProfilesManager) – Manager for storing and retrieving profile information.

      - tree (TreeRepresentation) – Tree structure from which profiles are derived.

      - dataset (MaskedDataset) – The dataset to filter.

      - features (list) – the list of features to filter on.

      - confidence_scores (np.ndarray) – Array of predicted accuracy values used for thresholding profiles.

      - min_samples_ratio (int) – Minimum sample ratio to consider for including a profile.

    Returns:

      A dictionary with declaration rates as keys and their corresponding minimum confidence levels as values.

    Return type:

      Dict[int, float]

  static calc_profiles_deprecated(profiles_manager: ProfilesManager, tree: TreeRepresentation, confidence_scores: ndarray, min_samples_ratio: int)

    Calculates profiles for different declaration rates and minimum sample ratios. This method assesses how profiles change across different confidence levels derived from predicted accuracies.

    Parameters:

      - profiles_manager (ProfilesManager) – Manager for storing and retrieving profile information.

      - tree (TreeRepresentation) – Tree structure from which profiles are derived.

      - confidence_scores (np.ndarray) – Array of predicted accuracy values used for thresholding profiles.

      - min_samples_ratio (int) – Minimum sample ratio to consider for including a profile.

  detectron_by_profiles(profiles_manager: ProfilesManager, confidence_scores: ndarray, training_params: Dict, base_model_manager: BaseModelManager, strategies: Type[DetectronStrategy] | List[Type[DetectronStrategy]], samples_size: int = 20, ensemble_size: int = 10, num_calibration_runs: int = 100, patience: int = 3, allow_margin: bool = False, margin: float = 0.05, all_dr: bool = True) → Dict

    Runs the Detectron method on the different testing set profiles.

    Parameters:

      - datasets (DatasetsManager) – The datasets manager instance.

      - profiles_manager (ProfilesManager) – the manager containing the profiles of the testing set.

      - training_params (dict) – Parameters for training the models.

      - base_model_manager (BaseModelManager) – The base model manager instance.

      - testing_mpc_values (np.ndarray) – MPC values for the testing data.

      - reference_mpc_values (np.ndarray) – MPC values for the reference data.

      - samples_size (int, optional) – Sample size for the Detectron experiment, by default 20.

      - ensemble_size (int, optional) – Number of models in the ensemble, by default 10.

      - num_calibration_runs (int, optional) – Number of calibration runs, by default 100.

      - patience (int, optional) – Patience for early stopping, by default 3.

      - strategies (Union[Type[DetectronStrategy], List[Type[DetectronStrategy]]]) – The strategies for testing disagreement.

      - allow_margin (bool, optional) – Whether to allow a margin in the test, by default False.

      - margin (float, optional) – Margin value for the test, by default 0.05.

      - all_dr (bool, optional) – Whether to run for all declaration rates, by default False.

    Returns:

      Dictionary of med3pa profiles with detectron results.

    Return type:

      Dict

  static detectron_by_profiles_deprecated(datasets: DatasetsManager, profiles_manager: ProfilesManager, confidence_scores: ndarray, training_params: Dict, base_model_manager: BaseModelManager, strategies: Type[DetectronStrategy] | List[Type[DetectronStrategy]], samples_size: int = 20, ensemble_size: int = 10, num_calibration_runs: int = 100, patience: int = 3, allow_margin: bool = False, margin: float = 0.05, all_dr: bool = True) → Dict

    Runs the Detectron method on the different testing set profiles.

    Parameters:

      - datasets (DatasetsManager) – The datasets manager instance.

      - profiles_manager (ProfilesManager) – the manager containing the profiles of the testing set.

      - training_params (dict) – Parameters for training the models.

      - base_model_manager (BaseModelManager) – The base model manager instance.

      - testing_mpc_values (np.ndarray) – MPC values for the testing data.

      - reference_mpc_values (np.ndarray) – MPC values for the reference data.

      - samples_size (int, optional) – Sample size for the Detectron experiment, by default 20.

      - ensemble_size (int, optional) – Number of models in the ensemble, by default 10.

      - num_calibration_runs (int, optional) – Number of calibration runs, by default 100.

      - patience (int, optional) – Patience for early stopping, by default 3.

      - strategies (Union[Type[DetectronStrategy], List[Type[DetectronStrategy]]]) – The strategies for testing disagreement.

      - allow_margin (bool, optional) – Whether to allow a margin in the test, by default False.

      - margin (float, optional) – Margin value for the test, by default 0.05.

      - all_dr (bool, optional) – Whether to run for all declaration rates, by default False.

    Returns:

      Dictionary of med3pa profiles with detectron results.

    Return type:

      Dict

  static update_min_confidence(metrics_by_dr: dict, confidence_scores: ndarray)

    Calculate metrics by declaration rates (DR), evaluating model performance at various thresholds of predicted accuracies.

    Parameters:

      - dataset (MaskedDataset) – The dataset to filter.

      - confidence_scores (np.ndarray) – the confidence scores used for filtering.

      - metrics_list (list) – List of metric names to be calculated (e.g., ‘AUC’, ‘Accuracy’).

      - set (str) – the set to calculate the metrics on.

    Returns:

      A dictionary containing metrics computed for each declaration rate from 100% to 0%, including metrics and population percentage.

    Return type:

      dict

## experiment module

Orchestrates the execution of the med3pa method and integrates the functionality of other modules to run comprehensive experiments. It includes classes to manage and store results `Med3paResults`, execute experiments like `Med3paExperiment` and `Med3paDetectronExperiment`, and integrate results from the Detectron method `Med3paDetectronResults`

class MED3pa.med3pa.experiment.Med3paDetectronExperiment

  Bases: `object`

  static run(datasets: DatasetsManager, base_model_manager: BaseModelManager, uncertainty_metric: str = 'absolute_error', training_params: Dict = None, samples_size: int = 20, samples_size_profiles: int = 10, ensemble_size: int = 10, num_calibration_runs: int = 100, patience: int = 3, test_strategies: str | List[str] = 'enhanced_disagreement_strategy', allow_margin: bool = False, margin: float = 0.05, ipc_type: str = 'RandomForestRegressor', ipc_params: Dict = None, ipc_grid_params: Dict = None, ipc_cv: int = None, pretrained_ipc: str = None, apc_params: Dict = None, apc_grid_params: Dict = None, apc_cv: int = None, pretrained_apc: str = None, samples_ratio_min: int = 0, samples_ratio_max: int = 50, samples_ratio_step: int = 5, med3pa_metrics: List[str] = ['Accuracy', 'BalancedAccuracy', 'Precision', 'F1Score', 'Specificity', 'Sensitivity', 'Auc', 'LogLoss', 'Auprc', 'MCC'], evaluate_models: bool = False, use_ref_models: bool = False, models_metrics: List[str] = ['MSE', 'RMSE', 'MAE'], mode: str = 'mpc', all_dr: bool = False) → Med3paResults

    Runs the MED3PA and Detectron experiment.

    Parameters:

      - datasets (DatasetsManager) – The datasets manager instance.

      - training_params (dict) – Parameters for training the models.

      - base_model_manager (BaseModelManager) – The base model manager instance.

      - uncertainty_metric (str, optional) – the uncertainty metric ysed to calculate uncertainty, by default absolute_error.

      - samples_size (int, optional) – Sample size for the Detectron experiment, by default 20.

      - samples_size_profiles (int, optional) – Sample size for Profiles Detectron experiment, by default 10.

      - ensemble_size (int, optional) – Number of models in the ensemble, by default 10.

      - num_calibration_runs (int, optional) – Number of calibration runs, by default 100.

      - patience (int, optional) – Patience for early stopping, by default 3.

      - test_strategies (Union[str, List[str]) – strategies for testing disagreement, by default enhanced_disagreement_strategies.

      - allow_margin (bool, optional) – Whether to allow a margin in the test, by default False.

      - margin (float, optional) – Margin value for the test, by default 0.05.

      - ipc_type (str, optional) – The regressor model to use for IPC, by default RandomForestRegressor.

      - ipc_params (dict, optional) – Parameters for initializing the IPC regressor model, by default None.

      - ipc_grid_params (dict, optional) – Grid search parameters for optimizing the IPC model, by default None.

      - ipc_cv (int, optional) – Number of cross-validation folds for optimizing the IPC model, by default None.

      - pretrained_ipc (str, optional) – path to a pretrained ipc, by default None.

      - apc_params (dict, optional) – Parameters for initializing the APC regressor model, by default None.

      - apc_grid_params (dict, optional) – Grid search parameters for optimizing the APC model, by default None.

      - pretrained_apc (str, optional) – path to a pretrained apc, by default None.

      - apc_cv (int, optional) – Number of cross-validation folds for optimizing the APC model, by default None.

      - samples_ratio_min (int, optional) – Minimum sample ratio, by default 0.

      - samples_ratio_max (int, optional) – Maximum sample ratio, by default 50.

      - samples_ratio_step (int, optional) – Step size for sample ratio, by default 5.

      - med3pa_metrics (list of str, optional) – List of metrics to calculate, by default [‘Auc’, ‘Accuracy’, ‘BalancedAccuracy’].

      - evaluate_models (bool, optional) – Whether to evaluate the models, by default False.

      - models_metrics (list of str, optional) – List of metrics for model evaluation, by default [‘MSE’, ‘RMSE’].

      - all_dr (bool, optional) – Whether to run for all declaration rates, by default False.

    Returns:

      Results of MED3pa on reference and testing sets, plus Detectron Results.

    Return type:

      Med3paResults

class MED3pa.med3pa.experiment.Med3paExperiment

  Bases: `object`

  Class to run the MED3PA method experiment.

  static run(datasets_manager: DatasetsManager, base_model_manager: BaseModelManager = None, uncertainty_metric: str = 'absolute_error', ipc_type: str = 'RandomForestRegressor', ipc_params: Dict = None, ipc_grid_params: Dict = None, ipc_cv: int = 4, pretrained_ipc: str = None, apc_params: Dict = None, apc_grid_params: Dict = None, apc_cv: int = 4, pretrained_apc: str = None, samples_ratio_min: int = 0, samples_ratio_max: int = 50, samples_ratio_step: int = 5, med3pa_metrics: List[str] = ['Accuracy', 'BalancedAccuracy', 'Precision', 'F1Score', 'Specificity', 'Sensitivity', 'Auc', 'LogLoss', 'Auprc', 'MCC'], evaluate_models: bool = False, use_ref_models: bool = False, mode: str = 'mpc', models_metrics: List[str] = ['MSE', 'RMSE', 'MAE']) → Med3paResults

    Runs the MED3PA experiment on both reference and testing sets.

    Parameters:

      - datasets_manager (DatasetsManager) – the datasets manager containing the dataset to use in the experiment.

      - base_model_manager (BaseModelManager, optional) – Instance of BaseModelManager to get the base model, by default None.

      - uncertainty_metric (str, optional) – the uncertainty metric ysed to calculate uncertainty, by default absolute_error.

      - ipc_type (str, optional) – The regressor model to use for IPC, by default RandomForestRegressor.

      - ipc_params (dict, optional) – Parameters for initializing the IPC regressor model, by default None.

      - ipc_grid_params (dict, optional) – Grid search parameters for optimizing the IPC model, by default None.

      - ipc_cv (int, optional) – Number of cross-validation folds for optimizing the IPC model, by default None.

      - pretrained_ipc (str, optional) – path to a pretrained ipc, by default None.

      - apc_params (dict, optional) – Parameters for initializing the APC regressor model, by default None.

      - apc_grid_params (dict, optional) – Grid search parameters for optimizing the APC model, by default None.

      - apc_cv (int, optional) – Number of cross-validation folds for optimizing the APC model, by default None.

      - pretrained_apc (str, optional) – path to a pretrained apc, by default None.

      - use_ref_models (bool, optional) – whether or not to use the trained IPC and APC models from the reference set on the test set.

      - samples_ratio_min (int, optional) – Minimum sample ratio, by default 0.

      - samples_ratio_max (int, optional) – Maximum sample ratio, by default 50.

      - samples_ratio_step (int, optional) – Step size for sample ratio, by default 5.

      - med3pa_metrics (list of str, optional) – List of metrics to calculate, by default [‘Auc’, ‘Accuracy’, ‘BalancedAccuracy’].

      - evaluate_models (bool, optional) – Whether to evaluate the models, by default False.

      - models_metrics (list of str, optional) – List of metrics for model evaluation, by default [‘MSE’, ‘RMSE’].

    Returns:

      the results of the MED3PA experiment on the reference set and testing set.

    Return type:

      Med3paResults

class MED3pa.med3pa.experiment.Med3paRecord

  Bases: `object`

  Class to store and manage results from the MED3PA method on one set.

  get_confidence_scores(mode='str') → ndarray
  get_profiles_manager() → ProfilesManager

    Retrieves the profiles manager for this Med3paResults instance

  save(file_path: str) → None

    Saves the experiment results.

    Parameters:

      file_path (str) – The file path to save the JSON files.

  set_confidence_scores(scores: ndarray, mode='str') → None
  set_dataset(mode: str, dataset: MaskedDataset) → None

    Saves the dataset for a given sample ratio.

    Parameters:

      - samples_ratio (int) – The sample ratio.

      - dataset (MaskedDataset) – The MaskedDataset instance.

  set_metrics_by_dr(metrics_by_dr: Dict) → None

    Set the calculated metrics by declaration rate.

    Parameters:

      metrics_by_dr (Dict) – Dictionary of metrics by declaration rate.

  set_models_evaluation(ipc_evaluation: Dict, apc_evaluation: Dict = None) → None

    Set models evaluation metrics.

    Parameters:

      - ipc_evaluation (Dict) – Evaluation metrics for IPC model.

      - apc_evaluation (Dict) – Evaluation metrics for APC model.

  set_profiles_manager(profile_manager: ProfilesManager) → None

    Set the profile manager for this Med3paResults instance.

    Parameters:

      profile_manager (ProfilesManager) – The ProfileManager instance.

  set_tree(tree: TreeRepresentation)

    Sets the constructed tree

class MED3pa.med3pa.experiment.Med3paResults(reference_record: Med3paRecord, test_record: Med3paRecord)

  Bases: `object`

  Class to store and manage results from the MED3PA complete experiment.

  os = <module 'os' (frozen)>
  save(file_path: str) → None

    Saves the experiment results.

    Parameters:

      file_path (str) – The file path to save the JSON files.

  save_models(file_path: str, mode: str = 'all', id: str = None) → None

    Saves the experiment ipc and apc models as .pkl files, alongside the tree structure for the test set.

    Parameters:

      - file_path (str) – The file path to save the pickled files.

      - mode (str) – Defines the type of models to save, either ipc, apc, or both.

      - id (str) – Optional identifier to append to the filenames.

  set_detectron_results(detectron_results: DetectronResult = None) → None

    Sets the detectron results for the Med3paDetectron experiment.

    Parameters:

      detectron_results (DetectronResult) – The structure holding the detectron results.

  set_experiment_config(config: Dict[str, Any]) → None

    Sets or updates the configuration for the Med3pa experiment.

    Parameters:

      config (Dict[str, Any]) – A dictionary of experiment configuration.

  set_models(ipc_model: IPCModel, apc_model: APCModel = None)

    Sets the models.

MED3pa.med3pa.experiment.to_serializable(obj: Any, additional_arg: Any = None) → Any

  Convert an object to a JSON-serializable format.

  Parameters:

    obj (Any) – The object to convert.

  Returns:

    The JSON-serializable representation of the object.

  Return type:

    Any

## compraison module

Compares between two experiments, either two `Med3paExperiment` or two `Med3paDetectronExperiment`

class MED3pa.med3pa.comparaison.Med3paComparison(results1_path: str, results2_path: str)

  Bases: `object`

  Class to compare the results of two Med3paExperiment instances.

  compare_config()

    Compares the config files of the two experiments.

  compare_experiments()

    Compares the experiments by global metrics, profiles, and Detectron results if applicable.

  compare_global_metrics()

    Compares global metrics between two sets of results and stores them in a dictionary.

  compare_models_evaluation()

    Compares IPC and APC evaluation between two experiments.

  compare_profiles_detectron_results()

    Compares Detectron results between two sets of profiles and stores them in a dictionary, using only the shared profiles.

  compare_profiles_metrics()

    Compares profile metrics between two sets of results and stores them in a dictionary, using only the shared profiles.

  identify_shared_profiles()

    Identifies the shared profiles between the two experiments and stores them in shared_profiles.

  is_comparable() → bool

    Determines whether the two experiments can be compared based on the given criteria. Two experiments can be compared if either: - The datasets are different, but the base model and experiment parameters are the same. - The base model is different, but the datasets and experiment parameters are the same.

    Returns:

      True if the experiments can be compared, False otherwise.

    Return type:

      bool

  save(directory_path: str) → None

    Saves the comparison results to a specified directory.

    Parameters:

      directory_path (str) – The directory where the comparison results will be saved.