library(mlr3)
library(mlr3pipelines)
library(mlr3tuning)Intro
Predicting probabilities in classification tasks allows us to adjust the probability thresholds required for assigning an observation to a certain class. This can lead to improved classification performance, especially for cases where we e.g. aim to balance off metrics such as false positive and false negative rates.
This is for example often done in ROC Analysis. The mlr3book also has a chapter on ROC Analysis) for the interested reader. This post does not focus on ROC analysis, but instead focusses on the general problem of adjusting classification thresholds for arbitrary metrics.
This post assumes some familiarity with the mlr3, and also the mlr3pipelines and mlr3tuning packages, as both are used during the post. The mlr3book contains more details on those two packages. This post is a more in-depth version of the article on threshold tuning in the mlr3book.
Prerequisites
We load the mlr3verse package which pulls in the most important packages for this example.
We initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.
set.seed(7832)
lgr::get_logger("mlr3")$set_threshold("warn")
lgr::get_logger("bbotk")$set_threshold("warn")Thresholds: A short intro
In order to understand thresholds, we will quickly showcase the effect of setting different thresholds:
First we create a learner that predicts probabilities and use it to predict on holdout data, storing the prediction.
learner = lrn("classif.rpart", predict_type = "prob")
rr = resample(tsk("diabetes"), learner, rsmp("holdout"))
prd = rr$prediction()
prd
── <PredictionClassif> for 43 observations: ────────────────────────────────────────────────────────────────────────────
row_ids truth response prob.pos prob.neg
2 pos pos 0.6000000 0.4000000
3 neg neg 0.2333333 0.7666667
4 pos neg 0.2333333 0.7666667
--- --- --- --- ---
117 neg neg 0.2333333 0.7666667
119 pos pos 0.6000000 0.4000000
123 neg neg 0.2333333 0.7666667If we now look at the confusion matrix, the off-diagonal elements are errors made by our model (false positives and false negatives) while on-diagol ements are where our model predicted correctly.
# Print confusion matrix
prd$confusion truth
response pos neg
pos 8 2
neg 8 25# Print False Positives and False Negatives
prd$score(list(msr("classif.fp"), msr("classif.fn")))classif.fp classif.fn
2 8 By adjusting the classification threshold, in this case the probability required to predict the positive class, we can now trade off predicting more positive cases (first row) against predicting fewer negative cases (second row) or vice versa.
# Lower threshold: More positives
prd$set_threshold(0.2)$confusion truth
response pos neg
pos 16 27
neg 0 0# Higher threshold: Fewer positives
prd$set_threshold(0.75)$confusion truth
response pos neg
pos 0 0
neg 16 27This threshold value can now be adjusted optimally for a given measure, such as accuracy. How this can be done is discussed in the following section.
Adjusting thresholds: Two strategies
Currently mlr3pipelines offers two main strategies towards adjusting classification thresholds. We can either expose the thresholds as a hyperparameter of the Learner by using PipeOpThreshold. This allows us to tune the thresholds via an outside optimizer from mlr3tuning.
Alternatively, we can also use PipeOpTuneThreshold which automatically tunes the threshold after each learner fit.
In this blog-post, we’ll go through both strategies.
PipeOpThreshold
PipeOpThreshold can be put directly after a Learner.
A simple example would be:
gr = lrn("classif.rpart", predict_type = "prob") %>>% po("threshold")
l = GraphLearner$new(gr)Note, that predict_type = “prob” is required for po("threshold") to have any effect.
The thresholds are now exposed as a hyperparameter of the GraphLearner we created:
as.data.table(l$param_set)[, .(id, class, lower, upper, nlevels)] id class lower upper nlevels
<char> <char> <num> <num> <num>
1: classif.rpart.cp ParamDbl 0 1 Inf
2: classif.rpart.keep_model ParamLgl NA NA 2
3: classif.rpart.maxcompete ParamInt 0 Inf Inf
4: classif.rpart.maxdepth ParamInt 1 30 30
5: classif.rpart.maxsurrogate ParamInt 0 Inf Inf
6: classif.rpart.minbucket ParamInt 1 Inf Inf
7: classif.rpart.minsplit ParamInt 1 Inf Inf
8: classif.rpart.surrogatestyle ParamInt 0 1 2
9: classif.rpart.usesurrogate ParamInt 0 2 3
10: classif.rpart.xval ParamInt 0 Inf Inf
11: threshold.thresholds ParamUty NA NA InfWe can now tune those thresholds from the outside as follows:
Before tuning, we have to define which hyperparameters we want to tune over. In this example, we only tune over the thresholds parameter of the threshold PipeOp. you can easily imagine, that we can also jointly tune over additional hyperparameters, i.e. rpart’s cp parameter.
As the Task we aim to optimize for is a binary task, we can simply specify the threshold parameter:
search_space = ps(
threshold.thresholds = p_dbl(lower = 0, upper = 1)
)We now create a AutoTuner, which automatically tunes the supplied learner over the ParamSet we supplied above.
at = auto_tuner(
tuner = tnr("random_search"),
learner = l,
resampling = rsmp("cv", folds = 3L),
measure = msr("classif.ce"),
search_space = search_space,
term_evals = 5L,
)
at$train(tsk("german_credit"))For multi-class Tasks, this is a little more complicated. We have to use a trafo to transform a set of ParamDbl into the desired format for threshold.thresholds: A named numeric vector containing the thresholds. This can be easily achieved via a trafo function:
search_space = ps(
versicolor = p_dbl(lower = 0, upper = 1),
setosa = p_dbl(lower = 0, upper = 1),
virginica = p_dbl(lower = 0, upper = 1),
.extra_trafo = function(x, param_set) {
list(threshold.thresholds = mlr3misc::map_dbl(x, identity))
}
)Inside the .exta_trafo, we simply collect all set params into a named vector via map_dbl and store it in the threshold.thresholds slot expected by the learner.
Again, we create a AutoTuner, which automatically tunes the supplied learner over the ParamSet we supplied above.
at_2 = auto_tuner(
tuner = tnr("random_search"),
learner = l,
resampling = rsmp("cv", folds = 3L),
measure = msr("classif.ce"),
search_space = search_space,
term_evals = 5L,
)
at_2$train(tsk("iris"))One drawback of this strategy is, that this requires us to fit a new model for each new threshold setting. While setting a threshold and computing performance is relatively cheap, fitting the learner is often more computationally demanding. A better strategy is therefore often to optimize the thresholds separately after each model fit.
PipeOpTuneThreshold
PipeOpTuneThreshold on the other hand works together with PipeOpLearnerCV. It directly optimizes the cross-validated predictions made by this PipeOp.
A simple example would be:
gr = po("learner_cv", lrn("classif.rpart", predict_type = "prob")) %>>%
po("tunethreshold")
l2 = GraphLearner$new(gr)Note, that predict_type = “prob” is required for po("tunethreshold") to have any effect. Additionally, note that this time no threshold parameter is exposed, it is automatically tuned internally.
as.data.table(l2$param_set)[, .(id, class, lower, upper, nlevels)] id class lower upper nlevels
<char> <char> <num> <num> <num>
1: classif.rpart.resampling.method ParamFct NA NA 2
2: classif.rpart.resampling.folds ParamInt 2 Inf Inf
3: classif.rpart.resampling.keep_response ParamLgl NA NA 2
4: classif.rpart.resampling.predict_method ParamFct NA NA 2
5: classif.rpart.resampling.prob_aggr ParamFct NA NA 2
6: classif.rpart.resampling.prob_aggr_eps ParamDbl 0 1 Inf
7: classif.rpart.cp ParamDbl 0 1 Inf
8: classif.rpart.keep_model ParamLgl NA NA 2
9: classif.rpart.maxcompete ParamInt 0 Inf Inf
10: classif.rpart.maxdepth ParamInt 1 30 30
11: classif.rpart.maxsurrogate ParamInt 0 Inf Inf
12: classif.rpart.minbucket ParamInt 1 Inf Inf
13: classif.rpart.minsplit ParamInt 1 Inf Inf
14: classif.rpart.surrogatestyle ParamInt 0 1 2
15: classif.rpart.usesurrogate ParamInt 0 2 3
16: classif.rpart.xval ParamInt 0 Inf Inf
17: classif.rpart.affect_columns ParamUty NA NA Inf
18: tunethreshold.measure ParamUty NA NA Inf
19: tunethreshold.optimizer ParamUty NA NA Inf
20: tunethreshold.log_level ParamUty NA NA Inf
id class lower upper nlevels
<char> <char> <num> <num> <num>If we now use the GraphLearner, it automatically adjusts the thresholds during prediction.
Note that we can set ResamplingInsample as a resampling strategy for PipeOpLearnerCV in order to evaluate predictions on the “training” data. This is generally not advised, as it might lead to over-fitting on the thresholds but can significantly reduce runtime.
Finally, we can compare no threshold tuning to the tunethreshold approach:
Comparison of the approaches
bmr = benchmark(benchmark_grid(
learners = list(no_tuning = lrn("classif.rpart"), internal = l2),
tasks = tsk("german_credit"),
rsmp("cv", folds = 3L)
))Warning:
✖ Multiple predict types detected, this will mean that you cannot evaluate the same measures on all learners.
→ Class: Mlr3WarningVaryingPredictTypesbmr$aggregate(list(msr("classif.ce"), msr("classif.fnr"))) nr task_id learner_id resampling_id iters classif.ce classif.fnr
<int> <char> <char> <char> <int> <num> <num>
1: 1 german_credit classif.rpart cv 3 0.2819766 0.1719254
2: 2 german_credit classif.rpart.tunethreshold cv 3 0.2819766 0.1763699
Hidden columns: resample_resultSession Information
sessioninfo::session_info(info = "packages")═ Session info ═══════════════════════════════════════════════════════════════════════════════════════════════════════
─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
backports 1.5.1 2026-04-03 [1] RSPM
base64url 1.4 2018-05-14 [1] RSPM
batchtools 0.9.18 2025-08-20 [1] RSPM
bbotk 1.12.0 2026-07-17 [1] RSPM
bit 4.6.0 2025-03-06 [1] RSPM
bit64 4.8.2 2026-05-19 [1] RSPM
brew 1.0-10 2023-12-16 [1] RSPM
callr 3.8.0 2026-06-05 [1] RSPM
checkmate 2.3.4 2026-02-03 [1] RSPM
class 7.3-23 2025-01-01 [2] CRAN (R 4.6.1)
classInt 0.4-11 2025-01-08 [1] RSPM
cli 3.6.6 2026-04-09 [1] RSPM
cluster 2.1.8.2 2026-02-05 [2] CRAN (R 4.6.1)
codetools 0.2-20 2024-03-31 [2] CRAN (R 4.6.1)
coro 1.1.0 2024-11-05 [1] RSPM
crayon 1.5.3 2024-06-20 [1] RSPM
data.table * 1.18.4 2026-05-06 [1] RSPM
DBI 1.3.0 2026-02-25 [1] RSPM
dictionar6 0.1.3 2026-02-23 [1] https://m~
digest 0.6.39 2025-11-19 [1] RSPM
distr6 1.8.4 2026-02-23 [1] https://m~
dplyr 1.2.1 2026-04-03 [1] RSPM
e1071 1.7-17 2025-12-18 [1] RSPM
evaluate 1.0.5 2025-08-27 [1] RSPM
farver 2.1.2 2024-05-13 [1] RSPM
fastmap 1.2.0 2024-05-15 [1] RSPM
future 1.75.0 2026-07-20 [1] RSPM
future.apply 1.20.2 2026-02-20 [1] RSPM
generics 0.1.4 2025-05-09 [1] RSPM
GenSA 1.1.15 2025-11-26 [1] RSPM
ggplot2 4.0.3 2026-04-22 [1] RSPM
globals 0.19.1 2026-03-13 [1] RSPM
glue 1.8.1 2026-04-17 [1] RSPM
gtable 0.3.6 2024-10-25 [1] RSPM
hms 1.1.4 2025-10-17 [1] RSPM
htmltools 0.5.9 2025-12-04 [1] RSPM
htmlwidgets 1.6.4 2023-12-06 [1] RSPM
jsonlite 2.0.0 2025-03-27 [1] RSPM
KernSmooth 2.23-26 2025-01-01 [2] CRAN (R 4.6.1)
knitr 1.51 2025-12-20 [1] RSPM
lattice 0.22-9 2026-02-09 [2] CRAN (R 4.6.1)
lgr 0.5.2 2026-01-30 [1] RSPM
lifecycle 1.0.5 2026-01-08 [1] RSPM
listenv 1.0.0 2026-06-22 [1] RSPM
magrittr 2.0.5 2026-04-04 [1] RSPM
Matrix 1.7-5 2026-03-21 [2] CRAN (R 4.6.1)
matrixStats 1.5.0 2025-01-07 [1] RSPM
mgcv 1.9-4 2025-11-07 [2] CRAN (R 4.6.1)
mlr3 * 1.7.1.9000 2026-08-07 [1] Github (mlr-org/mlr3@c63e546)
mlr3batchmark 0.2.2 2025-09-04 [1] RSPM
mlr3benchmark 0.1.7-9000 2026-08-07 [1] Github (mlr-org/mlr3benchmark@771107a)
mlr3cluster 0.4.1 2026-07-10 [1] RSPM
mlr3cmprsk 0.0.5 2026-04-11 [1] https://m~
mlr3data 0.9.0 2024-11-08 [1] RSPM
mlr3db 0.7.2 2026-05-22 [1] RSPM
mlr3extralearners 1.6.0 2026-07-14 [1] https://m~
mlr3fairness 0.4.0 2026-08-07 [1] Github (mlr-org/mlr3fairness@6946cbe)
mlr3fda 0.7.1 2026-07-15 [1] RSPM
mlr3filters 0.9.1 2026-04-23 [1] RSPM
mlr3fselect 1.6.0.9000 2026-08-07 [1] Github (mlr-org/mlr3fselect@3bc9771)
mlr3hyperband 1.1.1 2026-07-25 [1] RSPM
mlr3inferr 0.2.1 2025-11-26 [1] RSPM
mlr3learners 0.15.1 2026-07-25 [1] RSPM
mlr3mbo 1.2.1 2026-07-26 [1] RSPM
mlr3measures 1.3.0 2026-04-17 [1] RSPM
mlr3misc 0.22.0 2026-06-10 [1] RSPM
mlr3oml 0.12.0 2026-01-28 [1] RSPM
mlr3pipelines * 0.11.0-9000 2026-08-07 [1] Github (mlr-org/mlr3pipelines@3a48115)
mlr3proba 0.8.10 2026-06-05 [1] https://m~
mlr3spatial 0.7.0 2026-07-14 [1] RSPM
mlr3spatiotempcv 2.3.5 2026-08-03 [1] RSPM
mlr3torch 0.3.3 2026-01-31 [1] RSPM
mlr3tuning * 1.6.1 2026-07-26 [1] RSPM
mlr3tuningspaces 0.7.0 2026-07-25 [1] RSPM
mlr3verse 0.3.2 2026-06-22 [1] RSPM
mlr3viz 0.11.1 2026-07-26 [1] RSPM
mlr3website * 0.0.0.9000 2026-08-07 [1] Github (mlr-org/mlr3website@83dce5a)
moocore 0.3.2 2026-07-12 [1] RSPM
nlme 3.1-169 2026-03-27 [2] CRAN (R 4.6.1)
ooplah 0.2.0 2022-03-25 [1] https://m~
otel 0.2.0 2025-08-29 [1] RSPM
palmerpenguins 0.1.1 2022-08-15 [1] RSPM
paradox * 1.0.1 2024-07-09 [1] RSPM
parallelly 1.48.0 2026-06-29 [1] RSPM
param6 0.2.4 2026-02-23 [1] https://m~
pillar 1.11.1 2025-09-17 [1] RSPM
pkgconfig 2.0.3 2019-09-22 [1] RSPM
prettyunits 1.2.0 2023-09-24 [1] RSPM
processx 3.9.0 2026-04-22 [1] RSPM
progress 1.2.3 2023-12-06 [1] RSPM
proxy 0.4-29 2025-12-29 [1] RSPM
ps 1.9.3 2026-04-20 [1] RSPM
purrr 1.2.2 2026-04-10 [1] RSPM
R6 2.6.1 2025-02-15 [1] RSPM
rappdirs 0.3.4 2026-01-17 [1] RSPM
rbibutils 2.4.1 2026-01-21 [1] RSPM
RColorBrewer 1.1-3 2022-04-03 [1] RSPM
Rcpp 1.1.2 2026-07-05 [1] RSPM
Rdpack 2.6.6 2026-02-08 [1] RSPM
rlang 1.3.0 2026-07-05 [1] RSPM
rmarkdown 2.31 2026-03-26 [1] RSPM
rpart 4.1.27 2026-03-27 [2] CRAN (R 4.6.1)
S7 0.2.2 2026-04-22 [1] RSPM
scales 1.4.0 2025-04-24 [1] RSPM
sessioninfo 1.2.4 2026-06-04 [1] RSPM
set6 0.2.6 2026-02-23 [1] https://m~
sf 1.1-2 2026-07-23 [1] RSPM
spacefillr 0.4.0 2025-02-24 [1] RSPM
stringi 1.8.9 2026-08-04 [1] RSPM
survival 3.8-6 2026-01-16 [2] CRAN (R 4.6.1)
terra 1.9-34 2026-06-19 [1] RSPM
tf 0.5.0 2026-07-14 [1] RSPM
tibble 3.3.1 2026-01-11 [1] RSPM
tidyselect 1.2.1 2024-03-11 [1] RSPM
torch 0.17.0 2026-04-11 [1] RSPM
units 1.0-1 2026-03-11 [1] RSPM
uuid 1.2-2 2026-01-23 [1] RSPM
vctrs 0.7.3 2026-04-11 [1] RSPM
withr 3.0.3 2026-06-19 [1] RSPM
xfun 0.60 2026-07-09 [1] RSPM
yaml 2.3.12 2025-12-10 [1] RSPM
zoo 1.9-0 2026-07-31 [1] RSPM
[1] /usr/local/lib/R/site-library
[2] /usr/local/lib/R/library
* ── Packages attached to the search path.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────