Welcome to MED3pa documentation! — Med3pa documentation 0.1.0 documentation

# Welcome to MED3pa documentation!

# Overview

## Use and Application of the package

The `MED3pa` package is specifically designed to address critical challenges in deploying machine learning models, particularly focusing on the robustness and reliability of models under real-world conditions. It provides comprehensive tools for evaluating model stability and performance in the face of covariate shifts, and problematic data profiles.

## Key functionalities

- Covariate Shift Detection: Utilizing the detectron subpackage, MED3pa can identify significant shifts in data distributions that might affect the model’s predictions. This feature is crucial for applications such as healthcare, where early detection of shifts can prevent erroneous decisions.

- Uncertainty and Confidence Estimation: Through the med3pa subpackage, the package measures the uncertainty and predictive confidence at both individual and group levels. This helps in understanding the reliability of model predictions and in making informed decisions based on model outputs.

- Identification of Problematic Profiles: MED3pa analyzes data profiles that consistently lead to poor model performance. This capability allows developers to refine training datasets or retrain models to handle these edge cases effectively.

## Software Engineering Principles

The `MED3pa` package is constructed with a strong emphasis on software engineering principles, making it a robust and scalable solution for machine learning model evaluation:

- Modular Design: The package is structured into four distinct subpackages (detectron, med3pa, models, and datasets), each focusing on different aspects of model training and evaluation. This modularity allows users to easily integrate and utilize specific functionalities tailored to their needs without dealing with unnecessary complexities.

- Extensibility: Thanks to its modular architecture, the package can be effortlessly extended to include more functionalities or adapt to different use cases. New models, methods, or data handling procedures can be added with minimal impact on the existing system structure.

- Use of Design Patterns: MED3pa employs various design patterns that enhance its maintainability and usability. For example, the use of factory patterns in model creation and strategy patterns in handling different file extensions ensures that the system remains flexible and adaptable to new requirements.

Installation Guide

- Installation Guide
  - Prerequisites
  - Installation
    - Step 1: Install via pip
    - Step 2: Verify the Installation
  - Usage
    - Basic Example

# Subpackages

The package is structured into four distinct subpackages : `datasets`, `models`, `detectron` and `med3pa`, each focusing on different aspects of the package goals. This modularity allows users to easily integrate and utilize specific functionalities tailored to their needs without dealing with unnecessary complexities.

Subpackages

- datasets subpackage
  - Overview
  - loading_context module
    - `DataLoadingContext`
      - `DataLoadingContext.get_strategy()`
      - `DataLoadingContext.load_as_np()`
      - `DataLoadingContext.set_strategy()`
      - `DataLoadingContext.strategies`
    - `supported_file_formats()`
  - loading_strategies module
    - `CSVDataLoadingStrategy`
      - `CSVDataLoadingStrategy.execute()`
    - `DataLoadingStrategy`
      - `DataLoadingStrategy.execute()`
  - manager module
    - `DatasetsManager`
      - `DatasetsManager.combine()`
      - `DatasetsManager.get_column_labels()`
      - `DatasetsManager.get_dataset_by_type()`
      - `DatasetsManager.get_info()`
      - `DatasetsManager.reset()`
      - `DatasetsManager.reset_datasets()`
      - `DatasetsManager.save_dataset_to_csv()`
      - `DatasetsManager.set_column_labels()`
      - `DatasetsManager.set_from_data()`
      - `DatasetsManager.set_from_file()`
      - `DatasetsManager.summarize()`
  - masked module
    - `MaskedDataset`
      - `MaskedDataset.clone()`
      - `MaskedDataset.combine()`
      - `MaskedDataset.get_confidence_scores()`
      - `MaskedDataset.get_file_path()`
      - `MaskedDataset.get_info()`
      - `MaskedDataset.get_observations()`
      - `MaskedDataset.get_pseudo_labels()`
      - `MaskedDataset.get_pseudo_probabilities()`
      - `MaskedDataset.get_sample_counts()`
      - `MaskedDataset.get_true_labels()`
      - `MaskedDataset.refine()`
      - `MaskedDataset.reset_indices()`
      - `MaskedDataset.sample_random()`
      - `MaskedDataset.sample_uniform()`
      - `MaskedDataset.save_to_csv()`
      - `MaskedDataset.set_confidence_scores()`
      - `MaskedDataset.set_file_path()`
      - `MaskedDataset.set_pseudo_labels()`
      - `MaskedDataset.set_pseudo_probs_labels()`
      - `MaskedDataset.summarize()`
      - `MaskedDataset.to_dataframe()`
- models subpackage
  - Overview
  - factories module
    - `ModelFactory`
      - `ModelFactory.create_model_from_pickled()`
      - `ModelFactory.create_model_with_hyperparams()`
      - `ModelFactory.factories`
      - `ModelFactory.get_factory()`
      - `ModelFactory.get_supported_models()`
      - `ModelFactory.model_mapping`
    - `XGBoostFactory`
      - `XGBoostFactory.check_version()`
      - `XGBoostFactory.create_model_from_pickled()`
      - `XGBoostFactory.create_model_with_hyperparams()`
      - `XGBoostFactory.extract_params()`
  - abstract_models module
    - `ClassificationModel`
      - `ClassificationModel.balance_train_weights()`
      - `ClassificationModel.predict()`
      - `ClassificationModel.train()`
    - `Model`
      - `Model.model`
      - `Model.model_class`
      - `Model.params`
      - `Model.data_preparation_strategy`
      - `Model.pickled_model`
      - `Model.evaluate()`
      - `Model.get_data_strategy()`
      - `Model.get_info()`
      - `Model.get_model()`
      - `Model.get_model_type()`
      - `Model.get_params()`
      - `Model.get_path()`
      - `Model.is_pickled()`
      - `Model.print_evaluation_results()`
      - `Model.save()`
      - `Model.set_data_strategy()`
      - `Model.set_file_path()`
      - `Model.set_model()`
      - `Model.set_params()`
      - `Model.update_params()`
      - `Model.validate_params()`
    - `RegressionModel`
      - `RegressionModel.predict()`
      - `RegressionModel.train()`
  - concrete_classifiers module
    - `XGBoostModel`
      - `XGBoostModel.evaluate()`
      - `XGBoostModel.predict()`
      - `XGBoostModel.train()`
      - `XGBoostModel.train_to_disagree()`
  - concrete_regressors module
    - `DecisionTreeRegressorModel`
      - `DecisionTreeRegressorModel.evaluate()`
      - `DecisionTreeRegressorModel.predict()`
      - `DecisionTreeRegressorModel.train()`
    - `RandomForestRegressorModel`
      - `RandomForestRegressorModel.evaluate()`
      - `RandomForestRegressorModel.predict()`
      - `RandomForestRegressorModel.train()`
  - abstract_metrics module
    - `EvaluationMetric`
      - `EvaluationMetric.get_metric()`
      - `EvaluationMetric.supported_metrics()`
  - classification_metrics module
    - `ClassificationEvaluationMetrics`
      - `ClassificationEvaluationMetrics.accuracy()`
      - `ClassificationEvaluationMetrics.average_precision()`
      - `ClassificationEvaluationMetrics.balanced_accuracy()`
      - `ClassificationEvaluationMetrics.f1_score()`
      - `ClassificationEvaluationMetrics.get_metric()`
      - `ClassificationEvaluationMetrics.log_loss()`
      - `ClassificationEvaluationMetrics.matthews_corrcoef()`
      - `ClassificationEvaluationMetrics.npv()`
      - `ClassificationEvaluationMetrics.ppv()`
      - `ClassificationEvaluationMetrics.precision()`
      - `ClassificationEvaluationMetrics.recall()`
      - `ClassificationEvaluationMetrics.roc_auc()`
      - `ClassificationEvaluationMetrics.sensitivity()`
      - `ClassificationEvaluationMetrics.specificity()`
      - `ClassificationEvaluationMetrics.supported_metrics()`
  - regression_metrics module
    - `RegressionEvaluationMetrics`
      - `RegressionEvaluationMetrics.get_metric()`
      - `RegressionEvaluationMetrics.mean_absolute_error()`
      - `RegressionEvaluationMetrics.mean_squared_error()`
      - `RegressionEvaluationMetrics.r2_score()`
      - `RegressionEvaluationMetrics.root_mean_squared_error()`
      - `RegressionEvaluationMetrics.supported_metrics()`
  - data_strategies module
    - `DataPreparingStrategy`
      - `DataPreparingStrategy.execute()`
    - `ToDataframesStrategy`
      - `ToDataframesStrategy.execute()`
    - `ToDmatrixStrategy`
      - `ToDmatrixStrategy.execute()`
      - `ToDmatrixStrategy.is_supported_data()`
    - `ToNumpyStrategy`
      - `ToNumpyStrategy.execute()`
  - base module
    - `BaseModelManager`
      - `BaseModelManager.clone_base_model()`
      - `BaseModelManager.get_instance()`
      - `BaseModelManager.reset()`
      - `BaseModelManager.set_base_model()`
- detectron subpackage
  - Overview
  - ensemble module
    - `DetectronEnsemble`
      - `DetectronEnsemble.evaluate_ensemble()`
  - record module
    - `DetectronRecord`
      - `DetectronRecord.to_dict()`
      - `DetectronRecord.update()`
    - `DetectronRecordsManager`
      - `DetectronRecordsManager.count_quantile()`
      - `DetectronRecordsManager.counts()`
      - `DetectronRecordsManager.freeze()`
      - `DetectronRecordsManager.get_evaluation()`
      - `DetectronRecordsManager.get_record()`
      - `DetectronRecordsManager.load()`
      - `DetectronRecordsManager.predicted_probabilities()`
      - `DetectronRecordsManager.rejected_count_quantile()`
      - `DetectronRecordsManager.rejected_counts()`
      - `DetectronRecordsManager.rejection_rates()`
      - `DetectronRecordsManager.save()`
      - `DetectronRecordsManager.seed()`
      - `DetectronRecordsManager.set_evaluation()`
      - `DetectronRecordsManager.update()`
  - stopper module
    - `EarlyStopper`
      - `EarlyStopper.update()`
  - strategies module
  - experiment module
  - comparaison module
- med3pa subpackage
  - Overview
    - Key Components:
    - Advanced Analysis with MDR:
    - Extensibility for Integration:
  - uncertainty module
    - `AbsoluteError`
      - `AbsoluteError.calculate()`
    - `UncertaintyCalculator`
      - `UncertaintyCalculator.calculate_uncertainty()`
      - `UncertaintyCalculator.metric_mapping`
      - `UncertaintyCalculator.supported_metrics()`
    - `UncertaintyMetric`
      - `UncertaintyMetric.calculate()`
  - models module
    - `APCModel`
      - `APCModel.default_params`
      - `APCModel.evaluate()`
      - `APCModel.get_info()`
      - `APCModel.load_model()`
      - `APCModel.load_tree()`
      - `APCModel.optimize()`
      - `APCModel.predict()`
      - `APCModel.print_decision_tree_structure()`
      - `APCModel.save_model()`
      - `APCModel.supported_models_params()`
      - `APCModel.supported_params`
      - `APCModel.train()`
    - `IPCModel`
      - `IPCModel.default_params`
      - `IPCModel.evaluate()`
      - `IPCModel.get_info()`
      - `IPCModel.load_model()`
      - `IPCModel.optimize()`
      - `IPCModel.predict()`
      - `IPCModel.save_model()`
      - `IPCModel.supported_ipc_models()`
      - `IPCModel.supported_models_params()`
      - `IPCModel.supported_regressors_mapping`
      - `IPCModel.supported_regressos_params`
      - `IPCModel.train()`
      - `IPCModel.underlying_models_mapping`
    - `MPCModel`
      - `MPCModel.predict()`
  - tree module
    - `TreeRepresentation`
      - `TreeRepresentation.build_tree()`
      - `TreeRepresentation.get_all_nodes()`
      - `TreeRepresentation.get_all_profiles()`
      - `TreeRepresentation.save_tree()`
    - `to_serializable()`
  - Profiles module
    - `Profile`
      - `Profile.to_dict()`
      - `Profile.update_detectron_results()`
      - `Profile.update_metrics_results()`
      - `Profile.update_node_information()`
    - `ProfilesManager`
      - `ProfilesManager.get_lost_profiles()`
      - `ProfilesManager.get_profiles()`
      - `ProfilesManager.insert_lost_profiles()`
      - `ProfilesManager.insert_profiles()`
      - `ProfilesManager.transform_to_profiles()`
  - MDR module
    - `MDRCalculator`
      - `MDRCalculator.calc_metrics_by_dr()`
      - `MDRCalculator.calc_metrics_by_profiles()`
      - `MDRCalculator.calc_profiles()`
      - `MDRCalculator.calc_profiles_deprecated()`
      - `MDRCalculator.detectron_by_profiles()`
      - `MDRCalculator.detectron_by_profiles_deprecated()`
      - `MDRCalculator.update_min_confidence()`
  - experiment module
    - `Med3paDetectronExperiment`
      - `Med3paDetectronExperiment.run()`
    - `Med3paExperiment`
      - `Med3paExperiment.run()`
    - `Med3paRecord`
      - `Med3paRecord.get_confidence_scores()`
      - `Med3paRecord.get_profiles_manager()`
      - `Med3paRecord.save()`
      - `Med3paRecord.set_confidence_scores()`
      - `Med3paRecord.set_dataset()`
      - `Med3paRecord.set_metrics_by_dr()`
      - `Med3paRecord.set_models_evaluation()`
      - `Med3paRecord.set_profiles_manager()`
      - `Med3paRecord.set_tree()`
    - `Med3paResults`
      - `Med3paResults.os`
      - `Med3paResults.save()`
      - `Med3paResults.save_models()`
      - `Med3paResults.set_detectron_results()`
      - `Med3paResults.set_experiment_config()`
      - `Med3paResults.set_models()`
    - `to_serializable()`
  - compraison module
    - `Med3paComparison`
      - `Med3paComparison.compare_config()`
      - `Med3paComparison.compare_experiments()`
      - `Med3paComparison.compare_global_metrics()`
      - `Med3paComparison.compare_models_evaluation()`
      - `Med3paComparison.compare_profiles_detectron_results()`
      - `Med3paComparison.compare_profiles_metrics()`
      - `Med3paComparison.identify_shared_profiles()`
      - `Med3paComparison.is_comparable()`
      - `Med3paComparison.save()`

# Tutorials

Welcome to the tutorials section of the med3pa documentation. Here, we offer comprehensive, step-by-step guides to help you effectively utilize the various subpackages within `med3pa`. Each tutorial is designed to enhance your understanding of the package’s capabilities and to provide practical examples of how to integrate these tools into your data science and machine learning workflows.

Explore each tutorial to learn how to make the most of MED3pa’s robust functionalities for your projects.

Tutorials

- Working with datasets subpackage
  - Using the DatasetsManager class
    - Step 1: Importing the DatasetsManager
    - Step 2: Creating an Instance of DatasetsManager
    - Step 3: Loading Datasets
    - Step 4: Ensuring Feature Consistency
    - Step 5: Retrieving Data
    - Step 6: Getting a Summary
    - Step 7: Saving and Resetting Datasets
    - Summary of Outputs
  - Using the MaskedDataset Class
    - Step 1: Importing Necessary Modules
    - Step 2: Loading Data with DatasetsManager
    - Step 3: Performing Operations on MaskedDataset
- Working with the Models Subpackage
  - Using the ModelFactory Class
    - Step 1: Importing Necessary Modules
    - Step 2: Creating an Instance of ModelFactory
    - Step 3: Discovering Supported Models
    - Step 4: Specifying and Creating a Model
    - Step 5: Loading a Model from a Serialized State
  - Using the Model Class
    - Step 1: Training the Model
    - Step 2: Predicting Using the Trained Model
    - Step 3: Evaluating the Model
    - Step 4: Retrieving Model Information
    - Step 5: Saving Model Information
- Working with the Detectron Subpackage
  - Step 1: Setting up the Datasets
  - Step 2: Configuring the Model
  - Step 3: Running the Detectron Experiment
  - Step 4: Analyzing the Results
  - Step 5: Saving the Results
  - Step 6: Comparing the Results
- Working with the med3pa Subpackage
  - Running the MED3pa Experiment
    - Step 1: Setting up the Datasets
    - Step 2: Configuring the Model
    - Step 3: Running the med3pa Experiment
    - Step 4: Analyzing and Saving the Results
    - Step 5: Running experiments from pretrained models
    - Step 6: Comparing two experiments
  - Running the MED3pa and Detectron Experiment

# Indices and tables

- Index

- Module Index

- Search Page