Models — hlink 4.0.0a1 documentation

# Models

These are the machine learning models available for use in the model evaluation and training tasks and in their household counterparts.

There are a few attributes available for all models.

- `type` – Type: `string`. The name of the model type. The available model types are listed below.

- `threshold` – Type: `float`. The “alpha threshold”. This is the probability score required for a potential match to be labeled a match. `0 ≤ threshold ≤ 1`.

- `threshold_ratio` – Type: `float`. The threshold ratio or “beta threshold”. This applies to records which have multiple potential matches when `training.decision` is set to `"drop_duplicate_with_threshold_ratio"`. For each record, only potential matches which have the highest probability, have a probability of at least `threshold`, and whose probabilities are at least `threshold_ratio` times larger than the second-highest probability are matches. This is sometimes called the “de-duplication distance ratio”. `1 ≤ threshold_ratio < ∞`.

In addition, any model parameters documented in a model type’s Spark documentation can be passed as parameters to the model through hlink’s `training.chosen_model` and `training.model_exploration` configuration sections.

Here is an example `training.chosen_model` configuration. The `type`, `threshold`, and `threshold_ratio` attributes are hlink specific. `maxDepth` is a parameter to the random forest model which hlink passes through to the underlying Spark classifier.

    [training.chosen_model]
    type = "random_forest"
    threshold = 0.2
    threshold_ratio = 1.2
    maxDepth = 5

## random_forest

Uses pyspark.ml.classification.RandomForestClassifier.

- Parameters:

  - `maxDepth` – Type: `int`. Maximum depth of the tree. Spark default value is 5.

  - `numTrees` – Type: `int`. The number of trees to train. Spark default value is 20, must be >= 1.

  - `featureSubsetStrategy` – Type: `string`. Per the Spark docs: “The number of features to consider for splits at each tree node. Supported options: auto, all, onethird, sqrt, log2, (0.0-1.0], [1-n].”

    [training.chosen_model]
    type = "random_forest"
    threshold = 0.15
    threshold_ratio = 1.0
    maxDepth = 5
    numTrees = 75
    featureSubsetStrategy = "sqrt"

## probit

Uses pyspark.ml.regression.GeneralizedLinearRegression with `family="binomial"` and `link="probit"`.

    [training.chosen_model]
    type = "probit"
    threshold = 0.85
    threshold_ratio = 1.2

## logistic_regression

Uses pyspark.ml.classification.LogisticRegression

    [training.chosen_model]
    type = "logistic_regression"
    threshold = 0.5
    threshold_ratio = 1.0

## decision_tree

Uses pyspark.ml.classification.DecisionTreeClassifier.

- Parameters:

  - `maxDepth` – Type: `int`. Maximum depth of the tree.

  - `minInstancesPerNode` – Type `int`. Per the Spark docs: “Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.”

  - `maxBins` – Type: `int`. Per the Spark docs: “Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.”

    [training.chosen_model]
    type = "decision_tree"
    threshold = 0.5
    threshold_ratio = 1.5
    maxDepth = 6
    minInstancesPerNode = 2
    maxBins = 4

## gradient_boosted_trees

Uses pyspark.ml.classification.GBTClassifier.

- Parameters:

  - `maxDepth` – Type: `int`. Maximum depth of the tree.

  - `minInstancesPerNode` – Type `int`. Per the Spark docs: “Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Should be >= 1.”

  - `maxBins` – Type: `int`. Per the Spark docs: “Max number of bins for discretizing continuous features. Must be >=2 and >= number of categories for any categorical feature.”

    [training.chosen_model]
    type = "gradient_boosted_trees"
    threshold = 0.7
    threshold_ratio = 1.3
    maxDepth = 4
    minInstancesPerNode = 1
    maxBins = 6

## xgboost

Added in version 3.8.0.

XGBoost is an alternate, high-performance implementation of gradient boosting. It uses xgboost.spark.SparkXGBClassifier. Since the XGBoost-PySpark integration which the xgboost Python package provides is currently unstable, support for the xgboost model type is disabled in hlink by default. hlink will stop with an error if you try to use this model type without enabling support for it. To enable support for xgboost, install hlink with the `xgboost` extra.

    pip install hlink[xgboost]

This installs the xgboost package and its Python dependencies. Depending on your machine and operating system, you may also need to install the libomp library, which is another dependency of xgboost. xgboost should raise a helpful error if it detects that you need to install libomp.

You can view a list of xgboost’s parameters here.

    [training.chosen_model]
    type = "xgboost"
    threshold = 0.8
    threshold_ratio = 1.5
    max_depth = 5
    eta = 0.5
    gamma = 0.05

## lightgbm

Added in version 3.8.0.

LightGBM is another alternate, high-performance implementation of gradient boosting. It uses synapse.ml.lightgbm.LightGBMClassifier. `synapse.ml` is a library which provides various integrations with PySpark, including integrations between the C++ LightGBM library and PySpark.

LightGBM requires some additional Scala libraries that hlink does not usually install, so support for the lightgbm model is disabled in hlink by default. hlink will stop with an error if you try to use this model type without enabling support for it. To enable support for lightgbm, install hlink with the `lightgbm` extra.

    pip install hlink[lightgbm]

This installs the lightgbm package and its Python dependencies. Depending on your machine and operating system, you may also need to install the libomp library, which is another dependency of lightgbm. If you encounter errors when training a lightgbm model, please try installing libomp if you do not have it installed.

lightgbm has an enormous number of available parameters. Many of these are available as normal in hlink, via the LightGBMClassifier class. Others are available through the special `passThroughArgs` parameter, which passes additional parameters through to the C++ library. You can see a full list of the supported parameters here.

    [training.chosen_model]
    type = "lightgbm"
    # hlink's threshold and threshold_ratio
    threshold = 0.8
    threshold_ratio = 1.5
    # LightGBMClassifier supports these parameters (and many more).
    maxDepth = 5
    learningRate = 0.5
    # LightGBMClassifier does not directly support this parameter,
    # so we have to send it to the C++ library with passThroughArgs.
    passThroughArgs = "force_row_wise=true"