mlr3pipelines Tutorial - German Credit

mlr3pipelines imputation filtering stacking german credit data set classification

In this use case, we continue working with the German credit dataset. We already used different Learners on it in previous posts and tried to optimize their hyperparameters. To make things interesting, we artificially introduce missing values into the dataset, perform imputation and filtering and stack Learners.

Martin Binder , Florian Pfisterer


This is the third part of a serial of use cases with the German credit dataset. The other parts of this series can be found here:

In this tutorial, we continue working with the German credit dataset. We already used different Learners on it and tried to optimize their hyperparameters. Now we will do four additional things:

  1. We preprocess the data as an integrated step of the model fitting process
  2. We tune the associated preprocessing parameters
  3. We stack multiple Learners in an ensemble model
  4. We discuss some techniques that make Learners able to tackle challenging datasets that they could not handle otherwise (we are going to outline what challenging means in particular later on)


First, load the packages we are going to use:

We initialize the random number generator with a fixed seed for reproducibility, and decrease the verbosity of the logger to keep the output clearly represented.


We again use the German credit dataset, but will restrict ourselves to the factorial features. To make things interesting or to make it a bit harder for our Learners, we introduce missing values in the dataset:

task = tsk("german_credit")
credit_full = task$data()
credit = credit_full[, sapply(credit_full, FUN = is.factor), with = FALSE]

# sample values to NA
credit = credit[, lapply(.SD, function(x) {
  x[sample(c(TRUE, NA), length(x), replace = TRUE, prob = c(.9, .1))]
credit$credit_risk = credit_full$credit_risk
task = TaskClassif$new("GermanCredit", credit, "credit_risk")

We instantiate a Resampling instance for this Task to be able to compare resampling performance:

cv10 = rsmp("cv")$instantiate(task)

We also might want to use multiple cores to reduce long run times of tuning runs.



In this use case, we will take a look at composite machine learning algorithms that may incorporate data preprocessing or the combination of multiple Learners (“ensemble methods”).

We use the mlr3pipelines package that enables us to chain PipeOps into data flow Graphs.

Available PipeOps are listed in the mlr_pipeops dictionary:

<DictionaryPipeOp> with 74 stored values
Keys: boxcox, branch, chunk, classbalancing, classifavg, classweights, colapply, collapsefactors, colroles,
  compose_crank, compose_distr, compose_probregr, copy, crankcompose, datefeatures, distrcompose, encode,
  encodeimpact, encodelmer, featureunion, filter, fixfactors, histbin, ica, imputeconstant, imputehist,
  imputelearner, imputemean, imputemedian, imputemode, imputeoor, imputesample, kernelpca, learner,
  learner_cv, missind, modelmatrix, multiplicityexply, multiplicityimply, mutate, nmf, nop, ovrsplit,
  ovrunite, pca, proxy, quantilebin, randomprojection, randomresponse, regravg, removeconstants,
  renamecolumns, replicate, scale, scalemaxabs, scalerange, select, smote, spatialsign, subsample, survavg,
  targetinvert, targetmutate, targettrafoscalerange, textvectorizer, threshold, trafopred_regrsurv,
  trafopred_survregr, trafotask_regrsurv, trafotask_survregr, tunethreshold, unbranch, vtreat, yeojohnson

Missing Value Imputation

We have just introduced missing values into our data. While some Learners can deal with missing value, many cannot. Trying to train a random forest fails because of this:

ranger = lrn("classif.ranger")
Error: Task 'GermanCredit' has missing values in column(s) 'credit_history', 'employment_duration', 'foreign_worker', 'housing', 'installment_rate', 'job', 'number_credits', 'other_debtors', 'other_installment_plans', 'people_liable', 'personal_status_sex', 'present_residence', 'property', 'purpose', 'savings', 'status', 'telephone', but learner 'classif.ranger' does not support this

We can perform imputation of missing values using a PipeOp. To find out which imputation PipeOps are available, we do the following:

[1] "imputeconstant" "imputehist"     "imputelearner"  "imputemean"     "imputemedian"   "imputemode"    
[7] "imputeoor"      "imputesample"  

We choose to impute factorial features using a new level (via PipeOpImputeOOR). Let’s use the PipeOp itself to create an imputed Task. This shows us how the PipeOp actually works:

imputer = po("imputeoor")
task_imputed = imputer$train(list(task))[[1]]
credit_risk credit_history employment_duration foreign_worker housing installment_rate job number_credits other_debtors other_installment_plans people_liable personal_status_sex present_residence property purpose savings status telephone
good all credits at this bank paid back duly >= 7 yrs .MISSING rent < 20 skilled employee/official 2-3 none none 0 to 2 male : married/widowed >= 7 yrs unknown / no property furniture/equipment ... >= 1000 DM no checking account yes (under customer name)
bad no credits taken/all credits paid back duly 1 <= ... < 4 yrs no rent 25 <= ... < 35 skilled employee/official 1 none none 0 to 2 female : non-single or male : single 1 <= ... < 4 yrs unknown / no property furniture/equipment unknown/no savings account ... < 0 DM no
good all credits at this bank paid back duly 4 <= ... < 7 yrs no rent 25 <= ... < 35 unskilled - resident 1 none none 3 or more male : married/widowed 4 <= ... < 7 yrs unknown / no property repairs unknown/no savings account ... >= 200 DM / salary for at least 1 year .MISSING
good no credits taken/all credits paid back duly 4 <= ... < 7 yrs no own 25 <= ... < 35 skilled employee/official 1 guarantor .MISSING 3 or more male : married/widowed >= 7 yrs .MISSING car (used) unknown/no savings account no checking account no
bad existing credits paid back duly till now 1 <= ... < 4 yrs .MISSING own 20 <= ... < 25 skilled employee/official 2-3 none none 3 or more male : married/widowed >= 7 yrs real estate others unknown/no savings account no checking account no
good .MISSING 1 <= ... < 4 yrs no own 25 <= ... < 35 unskilled - resident 1 none none 3 or more male : married/widowed >= 7 yrs real estate repairs ... >= 1000 DM ... >= 200 DM / salary for at least 1 year yes (under customer name)

We do not only need complete data during training but also during prediction. Using the same imputation heuristic for both is the most consistent strategy. This way the imputation strategy can, in fact, be seen as a part of the complete learner (which could be tuned).

If we used the imputed Task for Resampling, we would leak information from the test set into the training set. Therefore, it is mandatory to attach the imputation operator to the Learner itself, creating a GraphLearner:

graph_learner_ranger = as_learner(po("imputeoor") %>>% ranger)


This GraphLearner can be used for resampling – like an ordinary Learner:

rr = resample(task, learner = graph_learner_ranger, resampling = cv10)

Feature Filtering

Typically, sparse models, i.e. having models with few(er) features, are desirable. This is due to a variety of reasons, e.g., enhanced interpretability or decreased costs of acquiring data. Furthermore, sparse models may actually be associated with increased performance (especially if overfitting is anticipated). We can use feature filter to only keep features with the highest information. Filters are implemented in the mlr3filters package and listed in the following dictionary:

<DictionaryFilter> with 20 stored values
Keys: anova, auc, carscore, cmim, correlation, disr, find_correlation, importance, information_gain, jmi,
  jmim, kruskal_test, mim, mrmr, njmim, performance, permutation, relief, selected_features, variance

We apply the FilterMIM (mutual information maximization) Filter as implemented in the praznik package. This Filter allows for the selection of the top-k features of best mutual information.

filter = flt("mim")
                 status          credit_history                 savings                 purpose                property 
                 1.0000                  0.9375                  0.8750                  0.8125                  0.7500 
                housing     employment_duration other_installment_plans     personal_status_sex           other_debtors 
                 0.6875                  0.6250                  0.5625                  0.5000                  0.4375 
       installment_rate          foreign_worker                     job          number_credits               telephone 
                 0.3750                  0.3125                  0.2500                  0.1875                  0.1250 
      present_residence           people_liable 
                 0.0625                  0.0000 

Making use of this Filter, you may wonder at which costs the reduction of the feature space comes. We can investigate the trade-off between features and performance by tuning. We incorporate our filtering strategy into the pipeline using PipeOpFilter. Like before, we need to perform imputation as the Filter also relies on complete data:

fpipe = po("imputeoor") %>>% po("filter", flt("mim"), filter.nfeat = 3)
   credit_risk                              credit_history                    savings
1:        good     all credits at this bank paid back duly             ... >= 1000 DM
2:         bad no credits taken/all credits paid back duly unknown/no savings account
3:        good     all credits at this bank paid back duly unknown/no savings account
4:        good no credits taken/all credits paid back duly unknown/no savings account
5:         bad    existing credits paid back duly till now unknown/no savings account
6:        good                                    .MISSING             ... >= 1000 DM
1:                        no checking account
2:                                 ... < 0 DM
3: ... >= 200 DM / salary for at least 1 year
4:                        no checking account
5:                        no checking account
6: ... >= 200 DM / salary for at least 1 year

We can now tune over the mim.filter.nfeat parameter. It steers how many features are kept by the Filter and eventually used by the learner:

search_space = ps(
  mim.filter.nfeat = p_int(lower = 1, upper = length(task$feature_names))

The problem is one-dimensional (i.e. only one parameter is tuned). Thus, we make use of a grid search. For higher dimensions, strategies like random search are more appropriate. The tuning procedure may take some time:

instance = tune(
  method = "grid_search",
  learner = fpipe %>>% lrn("classif.ranger"),
  resampling = cv10,
  measure = msr("classif.ce"),
  search_space = search_space)

We can plot the performance against the number of features. If we do so, we see the possible trade-off between sparsity and predictive performance:

autoplot(instance, type =  "marginal")


We want to build a model that is based on the predictions of other Learners. This means that we are in the state that we need predictions already during training. This is a very specific case that is luckily handled by PipeOpLearnerCV. PipeOpLearnerCV performs cross-validation during the training phase and returns the cross-validated predictions. We use "prob" predictions because they carry more information than response prediction:

graph_stack = po("imputeoor") %>>%
    po("learner_cv", lrn("classif.ranger", predict_type = "prob")),
    po("learner_cv", lrn("classif.kknn", predict_type = "prob")))) %>>%
  po("featureunion") %>>% lrn("classif.log_reg")

We built a pretty complex Graph already. Therefore, we plot it:


We now compare the performance of the stacked learner to the performance of the individual Learners:

grid = benchmark_grid(
  task = task,
  learner = list(
    as_learner(po("imputeoor") %>>% lrn("classif.ranger")),
    as_learner(po("imputeoor") %>>% lrn("classif.kknn")),
    as_learner(po("imputeoor") %>>% lrn("classif.log_reg"))),
  resampling = cv10)

bmr = benchmark(grid)
                                                           learner_id classif.ce
1: imputeoor.classif.ranger.classif.kknn.featureunion.classif.log_reg      0.282
2:                                           imputeoor.classif.ranger      0.292
3:                                             imputeoor.classif.kknn      0.299
4:                                          imputeoor.classif.log_reg      0.283

If we train the stacked learner and look into the final Learner (the logistic regression), we can see how “important” each Learner of the stacked learner is:


stats::glm(formula = task$formula(), family = "binomial", data = data, 
    model = FALSE)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3976  -1.0061   0.5370   0.8165   1.7707  

Coefficients: (2 not defined because of singularities)
                         Estimate Std. Error z value Pr(>|z|)    
(Intercept)               -3.2008     0.3499  -9.147   <2e-16 ***
classif.ranger.prob.good   5.3460     0.5612   9.526   <2e-16 ***
classif.ranger.prob.bad        NA         NA      NA       NA    
classif.kknn.prob.good     0.7275     0.3182   2.286   0.0222 *  
classif.kknn.prob.bad          NA         NA      NA       NA    
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1221.7  on 999  degrees of freedom
Residual deviance: 1062.9  on 997  degrees of freedom
AIC: 1068.9

Number of Fisher Scoring iterations: 4

The random forest has a higher contribution.

Robustify: Preventing new Prediction Factor Levels and other Problems

We now shift the context, using the complete German credit dataset:

task = tsk("german_credit")

There is a potential practical problem for both, small data sets and data sets with covariates having many factor levels: It may occur that not all possible factor levels have been used by the Learner during training. This happens because these rare instances are simply not sampled. The prediction then may fail because the Learner does not know how to handle unseen factor levels:

task_unseen = task$clone()$filter(1:30)
learner_logreg = lrn("classif.log_reg")
Error in model.frame.default(Terms, newdata, na.action = na.action, xlev = object$xlevels): factor job has new levels unemployed/unskilled - non-resident

Not only logistic regression but also many other Learners cannot handle new levels during prediction. Thus, we use PipeOpFixFactors to prevent that. PipeOpFixFactors introduces NA values for unseen levels. This means that we may need to impute afterwards. To solve this issue we can use PipeOpImputeSample, but with affect_columns set to only factorial features.

Another observation is that all-constant features may also be a problem:

task_constant = task$clone()$filter(1:2)
learner_logreg = lrn("classif.log_reg")
Error in `contrasts<-`(`*tmp*`, value = contr.funs[1 + isOF[nn]]): contrasts can be applied only to factors with 2 or more levels

This can be fixed using PipeOpRemoveConstants.

Both, handling unseen levels and all-constant features can be handled simultaneously using the following Graph:

robustify = po("fixfactors") %>>%
  po("removeconstants") %>>%
  po("imputesample", affect_columns = selector_type(c("ordered", "factor")))


This robust learner works even in very pathological conditions:

graph_learner_robustify = as_learner(robustify %>>% learner_logreg)

<PredictionClassif> for 1000 observations:
    row_ids truth response
          1  good     good
          2   bad      bad
          3  good     good
        998  good      bad
        999   bad      bad
       1000  good      bad

Your Ideas

There are various possibilities for preprocessing with PipeOps. You can try different methods for preprocessing and training. Feel free to discover this variety by yourself! Here are only a few hints that help when working with PipeOps:


For attribution, please cite this work as

Binder & Pfisterer (2020, March 11). mlr-org: mlr3pipelines Tutorial - German Credit. Retrieved from

BibTeX citation

  author = {Binder, Martin and Pfisterer, Florian},
  title = {mlr-org: mlr3pipelines Tutorial - German Credit},
  url = {},
  year = {2020}