# version >= 1.11 needed.
::install_github("berndbischl/ParamHelpers")
devtools::install_github("jakob-r/mlrHyperopt", dependencies = TRUE) devtools
Hyperparameter tuning with mlr is rich in options as they are multiple tuning methods:
- Simple Random Search
- Grid Search
- Iterated F-Racing (via irace)
- Sequential Model-Based Optimization (via mlrMBO)
Also the search space is easily definable and customizable for each of the 60+ learners of mlr using the ParamSets from the ParamHelpers Package.
The only drawback and shortcoming of mlr in comparison to caret in this regard is that mlr itself does not have defaults for the search spaces. This is where mlrHyperopt comes into play.
mlrHyperopt offers
- default search spaces for the most important learners in mlr,
- parameter tuning in one line of code,
- and an API to add and access custom search spaces from the mlrHyperopt Database
Installation
Tuning in one line
Tuning can be done in one line relying on the defaults. The default will automatically minimize the missclassification rate.
library(mlrHyperopt)
= hyperopt(iris.task, learner = "classif.svm")
res
res## Tune result:
## Op. pars: cost=23; gamma=0.0304
## mmce.test.mean=0.0333333
We can find out what hyperopt
did by inspecting the res
object.
Depending on the parameter space mlrHyperopt will automatically decide for a suitable tuning method:
$opt.path$par.set
res## Type len Def Constr Req Tunable Trafo
## cost numeric - 0 -15 to 15 - TRUE Y
## gamma numeric - -2 -15 to 15 - TRUE Y
$control
res## Tune control: TuneControlMBO
## Same resampling instance: TRUE
## Imputation value: 1
## Start: <NULL>
##
## Tune threshold: FALSE
## Further arguments: list()
As the search space defined in the ParamSet is only numeric, sequential Bayesian optimization was chosen. We can look into the evaluated parameter configurations and we can visualize the optimization run.
tail(as.data.frame(res$opt.path))
## cost gamma mmce.test.mean dob eol error.message exec.time
## 20 4.523485 -5.041227 0.03333333 20 NA <NA> 0.100
## 21 14.998631 -13.678416 0.03333333 21 NA <NA> 0.099
## 22 4.047901 -4.921606 0.04000000 22 NA <NA> 0.101
## 23 10.492025 -8.308426 0.05333333 23 NA <NA> 0.102
## 24 12.620900 -11.388249 0.03333333 24 NA <NA> 0.102
## 25 14.999114 -12.467912 0.05333333 25 NA <NA> 0.099
plotOptPath(res$opt.path)
## Warning: `guides(<scale> = FALSE)` is deprecated. Please use `guides(<scale> = "none")` instead.
## `guides(<scale> = FALSE)` is deprecated. Please use `guides(<scale> = "none")` instead.
## `guides(<scale> = FALSE)` is deprecated. Please use `guides(<scale> = "none")` instead.
The upper left plot shows the distribution of the tried settings in the search space and contour lines indicate where regions of good configurations are located. The lower right plot shows the value of the objective (the miss-classification rate) and how it decreases over the time. This also shows nicely that wrong settings can lead to bad results.
Using the mlrHyperopt API with mlr
If you just want to use mlrHyperopt to access the default parameter search spaces from the Often you don’t want to rely on the default procedures of mlrHyperopt and just incorporate it into your mlr-workflow. Here is one example how you can use the default search spaces for an easy benchmark:
= c("classif.xgboost", "classif.nnet")
lrns = makeLearners(lrns)
lrns = pid.task
tsk = makeResampleDesc('CV', stratify = TRUE, iters = 10)
rr = lapply(lrns, function(lrn) {
lrns.tuned if (getLearnerName(lrn) == "xgboost") {
# for xgboost we download a custom ParConfig from the Database
= downloadParConfigs(learner.name = getLearnerName(lrn))
pcs = pcs[[1]]
pc else {
} = getDefaultParConfig(learner = lrn)
pc
}= getParConfigParSet(pc)
ps # some parameters are dependend on the data (eg. the number of columns)
= evaluateParamExpressions(ps,
ps dict = mlrHyperopt::getTaskDictionary(task = tsk))
= setHyperPars(lrn, par.vals = getParConfigParVals(pc))
lrn = makeTuneControlRandom(maxit = 20)
ctrl makeTuneWrapper(learner = lrn, resampling = rr, par.set = ps,
control = ctrl)
})= benchmark(learners = c(lrns, lrns.tuned), tasks = tsk,
res resamplings = cv10)
plotBMRBoxplots(res)
As we can see we were able to improve the performance of xgboost and the nnet without any additional knowledge on what parameters we should tune. Especially for nnet improved performance is noticable.
Additional Information
Some recommended additional reads
- Vignette on getting started and also how to contribute by uploading alternative or additional ParConfigs.
- How to work with ParamSets as part of the Vignette.
- The slides of the useR 2017 Talk on mlrHyperopt.