Predicting probabilities in classification tasks allows us to adjust the probability thresholds required for assigning an observation to a certain class. This can lead to improved predictive performance.

Predicting probabilities in classification tasks allows us to adjust the probability thresholds required for assigning an observation to a certain class. This can lead to improved classification performance, especially for cases where we e.g. aim to balance off metrics such as false positive and false negative rates.

This is for example often done in ROC Analysis. The mlr3book also has a chapter on ROC Analysis) for the interested reader. This post does not focus on ROC analysis, but instead focusses on the general problem of adjusting classification thresholds for arbitrary metrics.

This post assumes some familiarity with the mlr3, and also the mlr3pipelines and mlr3tuning packages, as both are used during the post. The mlr3book contains more details on those two packages. This post is a more in-depth version of the article on threshold tuning in the mlr3book.

We load the mlr3verse package which pulls in the most important packages for this example.

We initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.

```
set.seed(7832)
lgr::get_logger("mlr3")$set_threshold("warn")
lgr::get_logger("bbotk")$set_threshold("warn")
```

In order to understand thresholds, we will quickly showcase the effect of setting different thresholds:

First we create a learner that predicts probabilities and use it to predict on holdout data, storing the prediction.

```
learner = lrn("classif.rpart", predict_type = "prob")
rr = resample(tsk("pima"), learner, rsmp("holdout"))
prd = rr$prediction()
prd
```

```
<PredictionClassif> for 256 observations:
row_ids truth response prob.pos prob.neg
4 neg neg 0.1057692 0.8942308
6 neg neg 0.0200000 0.9800000
10 pos neg 0.1428571 0.8571429
---
764 neg neg 0.2777778 0.7222222
766 neg neg 0.0200000 0.9800000
767 pos pos 0.8000000 0.2000000
```

If we now look at the confusion matrix, the off-diagonal elements are errors made by our model (*false positives* and *false negatives*) while on-diagol ements are where our model predicted correctly.

```
# Print confusion matrix
prd$confusion
```

```
truth
response pos neg
pos 53 27
neg 37 139
```

```
classif.fp classif.fn
27 37
```

By adjusting the **classification threshold**, in this case the probability required to predict the positive class, we can now trade off predicting more positive cases (first row) against predicting fewer negative cases (second row) or vice versa.

```
# Lower threshold: More positives
prd$set_threshold(0.25)$confusion
```

```
truth
response pos neg
pos 78 71
neg 12 95
```

```
# Higher threshold: Fewer positives
prd$set_threshold(0.75)$confusion
```

```
truth
response pos neg
pos 52 20
neg 38 146
```

This threshold value can now be adjusted optimally for a given measure, such as accuracy. How this can be done is discussed in the following section.

Currently mlr3pipelines offers two main strategies towards adjusting `classification thresholds`

. We can either expose the thresholds as a `hyperparameter`

of the Learner by using `PipeOpThreshold`

. This allows us to tune the `thresholds`

via an outside optimizer from mlr3tuning.

Alternatively, we can also use `PipeOpTuneThreshold`

which automatically tunes the threshold after each learner fit.

In this blog-post, we’ll go through both strategies.

`PipeOpThreshold`

can be put directly after a `Learner`

.

A simple example would be:

```
gr = lrn("classif.rpart", predict_type = "prob") %>>% po("threshold")
l = GraphLearner$new(gr)
```

Note, that `predict_type`

= “prob” is required for `po("threshold")`

to have any effect.

The `thresholds`

are now exposed as a `hyperparameter`

of the `GraphLearner`

we created:

```
l$param_set
```

id | class | lower | upper | nlevels |
---|---|---|---|---|

classif.rpart.cp | ParamDbl | 0 | 1 | Inf |

classif.rpart.keep_model | ParamLgl | NA | NA | 2 |

classif.rpart.maxcompete | ParamInt | 0 | Inf | Inf |

classif.rpart.maxdepth | ParamInt | 1 | 30 | 30 |

classif.rpart.maxsurrogate | ParamInt | 0 | Inf | Inf |

classif.rpart.minbucket | ParamInt | 1 | Inf | Inf |

classif.rpart.minsplit | ParamInt | 1 | Inf | Inf |

classif.rpart.surrogatestyle | ParamInt | 0 | 1 | 2 |

classif.rpart.usesurrogate | ParamInt | 0 | 2 | 3 |

classif.rpart.xval | ParamInt | 0 | Inf | Inf |

threshold.thresholds | ParamUty | NA | NA | Inf |

We can now tune those thresholds from the outside as follows:

Before `tuning`

, we have to define which hyperparameters we want to tune over. In this example, we only tune over the `thresholds`

parameter of the `threshold`

`PipeOp`

. you can easily imagine, that we can also jointly tune over additional hyperparameters, i.e. rpart’s `cp`

parameter.

As the `Task`

we aim to optimize for is a binary task, we can simply specify the threshold parameter:

We now create a `AutoTuner`

, which automatically tunes the supplied learner over the `ParamSet`

we supplied above.

```
at = auto_tuner(
method = "random_search",
learner = l,
resampling = rsmp("cv", folds = 3L),
measure = msr("classif.ce"),
search_space = search_space,
term_evals = 5L,
)
at$train(tsk("german_credit"))
```

For multi-class `Tasks`

, this is a little more complicated. We have to use a `trafo`

to transform a set of `ParamDbl`

into the desired format for `threshold.thresholds`

: A named numeric vector containing the thresholds. This can be easily achieved via a `trafo`

function:

Inside the `.exta_trafo`

, we simply collect all set params into a named vector via `map_dbl`

and store it in the `threshold.thresholds`

slot expected by the learner.

Again, we create a `AutoTuner`

, which automatically tunes the supplied learner over the `ParamSet`

we supplied above.

```
at_2 = auto_tuner(
method = "random_search",
learner = l,
resampling = rsmp("cv", folds = 3L),
measure = msr("classif.ce"),
search_space = search_space,
term_evals = 5L,
)
at_2$train(tsk("iris"))
```

One drawback of this strategy is, that this requires us to fit a new model for each new threshold setting. While setting a threshold and computing performance is relatively cheap, fitting the learner is often more computationally demanding. A better strategy is therefore often to optimize the thresholds separately after each model fit.

`PipeOpTuneThreshold`

on the other hand works together with `PipeOpLearnerCV`

. It directly optimizes the `cross-validated`

predictions made by this `PipeOp`

.

A simple example would be:

```
gr = po("learner_cv", lrn("classif.rpart", predict_type = "prob")) %>>%
po("tunethreshold")
l2 = GraphLearner$new(gr)
```

Note, that `predict_type`

= “prob” is required for `po("tunethreshold")`

to have any effect. Additionally, note that this time no `threshold`

parameter is exposed, it is automatically tuned internally.

```
l2$param_set
```

id | class | lower | upper | nlevels |
---|---|---|---|---|

classif.rpart.resampling.method | ParamFct | NA | NA | 2 |

classif.rpart.resampling.folds | ParamInt | 2 | Inf | Inf |

classif.rpart.resampling.keep_response | ParamLgl | NA | NA | 2 |

classif.rpart.cp | ParamDbl | 0 | 1 | Inf |

classif.rpart.keep_model | ParamLgl | NA | NA | 2 |

classif.rpart.maxcompete | ParamInt | 0 | Inf | Inf |

classif.rpart.maxdepth | ParamInt | 1 | 30 | 30 |

classif.rpart.maxsurrogate | ParamInt | 0 | Inf | Inf |

classif.rpart.minbucket | ParamInt | 1 | Inf | Inf |

classif.rpart.minsplit | ParamInt | 1 | Inf | Inf |

classif.rpart.surrogatestyle | ParamInt | 0 | 1 | 2 |

classif.rpart.usesurrogate | ParamInt | 0 | 2 | 3 |

classif.rpart.xval | ParamInt | 0 | Inf | Inf |

classif.rpart.affect_columns | ParamUty | NA | NA | Inf |

tunethreshold.measure | ParamUty | NA | NA | Inf |

tunethreshold.optimizer | ParamUty | NA | NA | Inf |

tunethreshold.log_level | ParamUty | NA | NA | Inf |

If we now use the `GraphLearner`

, it automatically adjusts the `thresholds`

during prediction.

Note that we can set `ResamplingInsample`

as a resampling strategy for `PipeOpLearnerCV`

in order to evaluate predictions on the “training” data. This is generally not advised, as it might lead to over-fitting on the thresholds but can significantly reduce runtime.

Finally, we can compare no threshold tuning to the `tunethreshold`

approach:

nr | task_id | learner_id | resampling_id | iters | classif.ce | classif.fnr |
---|---|---|---|---|---|---|

1 | german_credit | classif.rpart | cv | 3 | 0.2760095 | 0.1272398 |

2 | german_credit | classif.rpart.tunethreshold | cv | 3 | 0.2879916 | 0.0448533 |

For attribution, please cite this work as

Pfisterer (2020, Oct. 14). mlr-org: Threshold Tuning for Classification Tasks. Retrieved from https://mlr-org.github.io/mlr-org-website/gallery/2020-10-14-threshold-tuning/

BibTeX citation

@misc{pfisterer2020threshold, author = {Pfisterer, Florian}, title = {mlr-org: Threshold Tuning for Classification Tasks}, url = {https://mlr-org.github.io/mlr-org-website/gallery/2020-10-14-threshold-tuning/}, year = {2020} }