library(mlr3)
library(mlr3learners)
library(mlr3tuning)
learner = lrn("classif.svm")Scope
Setting time limits is an important consideration when tuning unreliable or unstable learning algorithms and when working on shared computing resources. The mlr3 ecosystem provides several mechanisms for setting time constraints for individual learners, tuning processes, and nested resampling.
Learner
This section demonstrates how to impose time constraints using a support vector machine (SVM) as an illustrative example.
Applying timeouts to the $train() and $predict() functions is essential for managing learners that may operate indefinitely. These time constraints are set independently for both the training and prediction stages. Generally, training a learner consumes more time than prediction. Certain learners, like k-nearest neighbors, lack a distinct training phase and require a timeout only during prediction. For the SVM’s training, we set a 10-second limit.
learner$timeout = c(train = 10, predict = Inf)To effectively terminate the process if necessary, it’s important to run the training and prediction within a separate R process. The callr package is recommended for this encapsulation, as it tends to be more reliable than the evaluate package, especially for terminating externally compiled code. Note that using callr increases the runtime due to the overhead of starting an R process. Additionally, a fallback learner must be specified, such as "classif.featureless", to provide baseline predictions in case the primary learner is terminated.
learner$encapsulate(method = "callr", fallback = lrn("classif.featureless"))These time constraints are now integrated into the training, resampling, and benchmarking processes. For more information on encapsulation and fallback learners, see the mlr3book. The next section will focus on setting time limits for the entire tuning process.
Tuning
When working with high-performance computing clusters, jobs are often bound by strict time constraints. Exceeding these limits results in the job being terminated and the loss of any results generated. Therefore, it’s important to ensure that the tuning process is designed to adhere to these time constraints.
The trm("run_time") controls the duration of the tuning process. We must take into account that the terminator can only check if the time limit is reached between batches. We must therefore set the time lower than the runtime of the job. How much lower depends on the runtime or time limit of the individual learners. The last batch should be able to finish before the time limit of the cluster is reached.
terminator = trm("run_time", secs = 60)
instance = ti(
task = tsk("sonar"),
learner = learner,
resampling = rsmp("cv", folds = 3),
measures = msr("classif.ce"),
terminator = terminator
)With these settings, our tuning operation is configured to run for 60 seconds, while individual learners are set to terminate after 10 seconds. This approach ensures the tuning process is efficient and adheres to the constraints imposed by the high-performance computing cluster.
Nested Resampling
When using nested resampling, time constraints become more complex as they are applied across various levels. As before, the time limit for an individual learner during the tuning is set with $timeout. The time limit for the tuning processes in the auto tuners is controlled with the trm("run_time"). It’s important to note that once the auto tuner enters the final phase of fitting the model and making predictions on the outer test set, the time limit governed by the terminator no longer applies. Additionally, the time limit previously set on the learner is temporarily deactivated, allowing the auto tuner to complete its task uninterrupted. However, a separate time limit can be assigned to each auto tuner using $timeout. This limit encompasses not only the tuning phase but also the time required for fitting the final model and predictions on the outer test set.
The best way to show this is with an example. We set the time limit for an individual learner to 10 seconds.
learner$timeout = c(train = 10, predict = Inf)
learner$encapsulate(method = "callr", fallback = lrn("classif.featureless"))Next, we give each auto tuner 60 seconds to finish the tuning process.
terminator = trm("run_time", secs = 60)Furthermore, we impose a 120-second limit for resampling each auto tuner. This effectively divides the time allocation, with around 60 seconds for tuning and another 60 seconds for final model fitting and predictions on the outer test set.
at = auto_tuner(
tuner = tnr("random_search"),
learner = learner,
resampling = rsmp("cv", folds = 3),
measure = msr("classif.ce"),
terminator = trm("run_time", secs = 60)
)
at$timeout = c(train = 100, predict = 20)
at$encapsulate(method = "callr", fallback = lrn("classif.featureless"))In total, the entire nested resampling process is designed to be completed within 10 minutes (120 seconds multiplied by 5 folds).
rr = resample(task, at, rsmp("cv", folds = 5))Conclusion
We delved into the setting of time constraints across different levels in the mlr3 ecosystem. From individual learners to the complexities of nested resampling, we’ve seen how effectively managing time limits can significantly enhance the efficiency and reliability of machine learning workflows. By utilizing the trm("run_time") for tuning processes and setting $timeout for individual learners and auto tuners, we can ensure that our machine learning tasks are not only effective but also adhere to the practical time constraints of shared computational resources. For more information, see also the error handling section in the mlr3book.
Session Information
sessioninfo::session_info(info = "packages")═ Session info ═══════════════════════════════════════════════════════════════════════════════════════════════════════
─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
backports 1.5.1 2026-04-03 [1] RSPM
base64url 1.4 2018-05-14 [1] RSPM
batchtools 0.9.18 2025-08-20 [1] RSPM
bbotk 1.13.1 2026-09-17 [1] RSPM
bit 4.6.0 2025-03-06 [1] RSPM
bit64 4.8.6 2026-09-01 [1] RSPM
brew 1.0-10 2023-12-16 [1] RSPM
callr 3.8.0 2026-06-05 [1] RSPM
checkmate 2.3.4 2026-02-03 [1] RSPM
class 7.3-24 2026-08-03 [2] CRAN (R 4.6.2)
classInt 0.4-11 2025-01-08 [1] RSPM
cli 3.6.6 2026-04-09 [1] RSPM
codetools 0.2-20 2024-03-31 [2] CRAN (R 4.6.2)
coro 1.1.0 2024-11-05 [1] RSPM
crayon 1.5.3 2024-06-20 [1] RSPM
data.table * 1.18.6.1 2026-08-24 [1] RSPM
DBI 1.3.0 2026-02-25 [1] RSPM
dictionar6 0.1.3 2026-02-23 [1] https://m~
digest 0.6.39 2025-11-19 [1] RSPM
distr6 1.8.4 2026-02-23 [1] https://m~
dplyr 1.2.1 2026-04-03 [1] RSPM
e1071 1.7-17 2025-12-18 [1] RSPM
evaluate 1.0.5 2025-08-27 [1] RSPM
farver 2.1.2 2024-05-13 [1] RSPM
fastmap 1.2.0 2024-05-15 [1] RSPM
future 1.82.0 2026-11-02 [1] RSPM
generics 0.1.4 2025-05-09 [1] RSPM
ggplot2 4.0.3 2026-04-22 [1] RSPM
globals 0.20.0 2026-10-11 [1] RSPM
glue 1.8.1 2026-04-17 [1] RSPM
gtable 0.3.6 2024-10-25 [1] RSPM
hms 1.1.4 2025-10-17 [1] RSPM
htmltools 0.5.9 2025-12-04 [1] RSPM
htmlwidgets 1.6.4 2023-12-06 [1] RSPM
jsonlite 2.0.0 2025-03-27 [1] RSPM
KernSmooth 2.23-27 2026-08-12 [2] CRAN (R 4.6.2)
knitr 1.52 2026-09-06 [1] RSPM
lattice 0.23-1 2026-08-12 [2] CRAN (R 4.6.2)
lgr 0.5.2 2026-01-30 [1] RSPM
lifecycle 1.0.5 2026-01-08 [1] RSPM
listenv 1.1.0 2026-10-04 [1] RSPM
magrittr 2.0.5 2026-04-04 [1] RSPM
Matrix 1.7-6 2026-07-25 [2] CRAN (R 4.6.2)
matrixStats 1.5.0 2025-01-07 [1] RSPM
mgcv 1.9-4 2025-11-07 [2] CRAN (R 4.6.2)
mlr3 * 1.8.0 2026-08-21 [1] RSPM
mlr3batchmark 0.2.2 2025-09-04 [1] RSPM
mlr3benchmark 0.1.8 2026-08-21 [1] RSPM
mlr3cluster 0.7.0 2026-10-17 [1] RSPM
mlr3cmprsk 0.0.7 2026-10-18 [1] https://m~
mlr3data 0.9.0 2024-11-08 [1] RSPM
mlr3db 0.7.3 2026-11-03 [1] RSPM
mlr3extralearners 1.7.0 2026-09-15 [1] https://m~
mlr3fairness 0.4.0 2026-11-04 [1] Github (mlr-org/mlr3fairness@6946cbe)
mlr3fda 0.7.2 2026-08-21 [1] RSPM
mlr3filters 0.9.1 2026-04-23 [1] RSPM
mlr3fselect 1.7.0 2026-08-23 [1] RSPM
mlr3hyperband 1.1.1 2026-07-25 [1] RSPM
mlr3inferr 0.2.2 2026-08-21 [1] RSPM
mlr3learners * 0.16.0 2026-09-08 [1] RSPM
mlr3mbo 1.2.1 2026-07-26 [1] RSPM
mlr3measures 1.3.0 2026-04-17 [1] RSPM
mlr3misc 0.23.0 2026-08-21 [1] RSPM
mlr3oml 0.12.0 2026-01-28 [1] RSPM
mlr3pipelines 0.12.0 2026-09-05 [1] RSPM
mlr3proba 0.8.10 2026-06-05 [1] https://m~
mlr3spatial 0.7.0 2026-07-14 [1] RSPM
mlr3spatiotempcv 2.3.5 2026-08-03 [1] RSPM
mlr3torch 0.4.0 2026-10-30 [1] RSPM
mlr3tuning * 1.7.0 2026-09-02 [1] RSPM
mlr3tuningspaces 0.7.0 2026-07-25 [1] RSPM
mlr3verse 0.4.0 2026-09-15 [1] RSPM
mlr3viz 0.11.2 2026-09-23 [1] RSPM
mlr3website * 0.0.0.9000 2026-11-04 [1] Github (mlr-org/mlr3website@3087c0c)
moocore 0.3.2 2026-07-12 [1] RSPM
nlme 3.1-171 2026-09-01 [2] CRAN (R 4.6.2)
ooplah 0.2.0 2022-03-25 [1] https://m~
otel 0.2.0 2025-08-29 [1] RSPM
palmerpenguins 0.1.1 2022-08-15 [1] RSPM
paradox * 1.0.1 2024-07-09 [1] RSPM
parallelly 1.50.0 2026-10-31 [1] RSPM
param6 0.2.4 2026-02-23 [1] https://m~
pillar 1.11.1 2025-09-17 [1] RSPM
pkgconfig 2.0.3 2019-09-22 [1] RSPM
prettyunits 1.2.0 2023-09-24 [1] RSPM
processx 3.9.0 2026-04-22 [1] RSPM
progress 1.2.3 2023-12-06 [1] RSPM
proxy 0.4-29 2025-12-29 [1] RSPM
ps 1.9.3 2026-04-20 [1] RSPM
purrr 1.2.2 2026-04-10 [1] RSPM
R6 2.6.1 2025-02-15 [1] RSPM
rappdirs 0.3.4 2026-01-17 [1] RSPM
rbibutils 2.4.1 2026-01-21 [1] RSPM
RColorBrewer 1.1-3 2022-04-03 [1] RSPM
Rcpp 1.1.2 2026-07-05 [1] RSPM
Rdpack 2.6.6 2026-02-08 [1] RSPM
rlang 1.3.0 2026-07-05 [1] RSPM
rmarkdown 2.32 2026-09-01 [1] RSPM
S7 0.2.2 2026-04-22 [1] RSPM
scales 1.4.0 2025-04-24 [1] RSPM
sessioninfo 1.2.4 2026-06-04 [1] RSPM
set6 0.2.6 2026-02-23 [1] https://m~
sf 1.1-3 2026-09-11 [1] RSPM
spacefillr 0.4.0 2025-02-24 [1] RSPM
stringi 1.8.9 2026-08-04 [1] RSPM
survival 3.8-12 2026-09-09 [2] CRAN (R 4.6.2)
terra 1.9-50 2026-09-08 [1] RSPM
tf 0.5.0 2026-07-14 [1] RSPM
tibble 3.3.1 2026-01-11 [1] RSPM
tidyselect 1.2.1 2024-03-11 [1] RSPM
torch 0.17.0 2026-04-11 [1] RSPM
units 1.0-1 2026-03-11 [1] RSPM
uuid 1.2-2 2026-01-23 [1] RSPM
vctrs 0.7.3 2026-04-11 [1] RSPM
withr 3.0.3 2026-06-19 [1] RSPM
xfun 0.62 2026-10-16 [1] RSPM
yaml 2.3.12 2025-12-10 [1] RSPM
zoo 1.9-1 2026-09-25 [1] RSPM
[1] /usr/local/lib/R/site-library
[2] /usr/local/lib/R/library
* ── Packages attached to the search path.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────