mlr3 - Runtime and Memory Benchmarks

Scope

This report analyzes the runtime and memory usage of the mlr3 package across different versions. It focuses on the $train() and $predict() methods of the learner and the resample() and benchmark() functions used for model evaluation. The benchmarks vary the training time of the models, the number of resampling iterations, and the size of the dataset. Additionally, the experiments are conducted in parallel to assess the impact of parallelization on both runtime and memory usage. The overhead introduced by encapsulation is examined by comparing different encapsulation methods. Furthermore, the report evaluates the object sizes of various mlr3 objects.

Given the extensive package ecosystem of mlr3, performance bottlenecks can occur at multiple stages. This report aims to help users determine whether the runtime of their workflows falls within expected ranges. If significant runtime or memory anomalies are observed, users are encouraged to report them by opening a GitHub issue.

Benchmarks are conducted on a high-performance cluster optimized for multi-core performance rather than single-core speed. Consequently, runtimes may be faster on a local machine. The benchmarks for $train() and $predict() were conducted on a different, slower cluster.

Summary of Latest mlr3 Version

The benchmarks are comprehensive; therefore, we present a summary of the results for the latest mlr3 version. The runtime of mlr3 should always be considered relative to the training and prediction time of the underlying models. For example, if the training step of a random forest model ranger::ranger() takes 100 ms and the $train() method of lrn("classif.ranger") takes 110 ms, the overhead is 10%. When the same model takes 1 second to train, the overhead introduced by mlr3 is only 1%. Instead of using real models, we simulate the training and prediction time for models by sleeping for 1, 10, 100, and 1000 ms.

We start by measuring the runtime of the $train() method of the learner. For models with a training time of 1000 and 100 ms, the overhead introduced by mlr3 is minimal. Models with a training time of 10 ms take 2 times longer to train in mlr3. For models with a training time of 1 ms, the overhead is approximately 10 times larger than the actual model training time. The overhead of $predict() is similar to $train() and the size of the dataset being predicted plays only a minor role. The $predict_newdata() method converts the data to a task and then predicts on it which doubles the overhead of the $predict() method. The recently introduced $predict_newdata_fast() method is much faster than $predict_newdata(). For models with a prediction time of 10 ms, the overhead is around 10%. For models with a prediction time of 1 ms, the overhead is around 50%.

The overhead of resample() and benchmark() is also very low for models with a training time of 1000 and 100 ms. However, for models with a training time of 10 ms, the overhead approximately doubles the runtime. In cases where the training time is only 1 ms, the overhead results in the runtime being ten times larger than the actual model training time. Running an empty R session consumes 131 MB of memory. Resampling with 10 iterations consumes approximately 164 MB, increasing to around 225 MB when performing 1,000 resampling iterations. Memory usage for benchmarking is comparable to that of resampling.

mlr3 utilizes the future package to enable parallelization over resampling iterations. However, running resample() and benchmark() in parallel introduces overhead due to the initiation of worker processes. Therefore, we compare the runtime of parallel execution with that of sequential execution. For models with a 1-second training time, using resample() and benchmark() in parallel reduces runtime. For models with a 100 ms training time, parallel execution is only advantageous when performing 100 or 1,000 resampling iterations. For models with 10 ms and 1 ms training times, sequential execution only becomes slower than parallel execution when running 1,000 resampling iterations. Memory usage increases significantly with the number of cores since each core initiates a separate R session. Utilizing 10 cores results in a total memory usage of around 1.2 GB.

Encapsulation ensures that signaled conditions (e.g., messages, warnings and errors) are intercepted and that all conditions raised during the training or prediction step are logged into the learner without interrupting the program flow. The encapsulation of the $train() method introduces a runtime overhead of ~1 seconds per model training when using the callr package. In contrast, the encapsulation of the evaluate package adds negligible overhead to the runtime.

When saving and loading mlr3 objects, their size can become significantly larger than expected. This issue is inherent to R6 objects due to data duplication during serialization and deserialization. The latest versions of mlr3 have implemented various strategies to mitigate this problem, substantially reducing the previously observed large increases in object size.

Train

The runtime and memory usage of the $train() method is measured for different mlr3 versions. The train step is performed for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with 1000 observations.

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

learner$train(task)

Model Time 1000 ms

Median runtime of $train() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $train() with models trained for 1000 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 1000 10 1016 1.0
1.0.0 1.0.1 1000 10 1016 1.0
0.23.0 1.0.1 1000 10 1016 1.0
0.22.1 1.0.1 1000 10 1016 1.0
0.22.0 1.0.1 1000 10 1016 1.0
0.21.1 1.0.1 1000 10 1015 1.0
0.21.0 1.0.1 1000 10 1015 1.0
0.20.2 1.0.1 1000 10 1015 1.0
0.20.1 1.0.1 1000 10 1015 1.0
0.20.0 1.0.0 1000 10 1019 1.0
0.19.0 0.11.1 1000 10 1016 1.0
0.18.0 0.11.1 1000 10 1016 1.0
0.17.2 0.11.1 1000 10 1016 1.0
0.17.1 0.11.1 1000 10 1016 1.0
0.17.0 0.11.1 1000 10 1016 1.0
0.16.0 0.11.1 1000 10 1018 1.0
0.15.0 0.11.1 1000 10 1019 1.0
0.14.2 0.10.0 1000 10 1018 1.0
0.13.4 0.9.0 1000 10 1016 1.0
100 Observations
1.0.1 1.0.1 1000 100 1016 1.0
1.0.0 1.0.1 1000 100 1015 1.0
0.23.0 1.0.1 1000 100 1015 1.0
0.22.1 1.0.1 1000 100 1016 1.0
0.22.0 1.0.1 1000 100 1016 1.0
0.21.1 1.0.1 1000 100 1016 1.0
0.21.0 1.0.1 1000 100 1015 1.0
0.20.2 1.0.1 1000 100 1016 1.0
0.20.1 1.0.1 1000 100 1017 1.0
0.20.0 1.0.0 1000 100 1020 1.0
0.19.0 0.11.1 1000 100 1016 1.0
0.18.0 0.11.1 1000 100 1016 1.0
0.17.2 0.11.1 1000 100 1016 1.0
0.17.1 0.11.1 1000 100 1016 1.0
0.17.0 0.11.1 1000 100 1016 1.0
0.16.0 0.11.1 1000 100 1017 1.0
0.15.0 0.11.1 1000 100 1017 1.0
0.14.2 0.10.0 1000 100 1016 1.0
0.13.4 0.9.0 1000 100 1016 1.0
1000 Observations
1.0.1 1.0.1 1000 1000 1018 1.0
1.0.0 1.0.1 1000 1000 1016 1.0
0.23.0 1.0.1 1000 1000 1015 1.0
0.22.1 1.0.1 1000 1000 1016 1.0
0.22.0 1.0.1 1000 1000 1016 1.0
0.21.1 1.0.1 1000 1000 1016 1.0
0.21.0 1.0.1 1000 1000 1015 1.0
0.20.2 1.0.1 1000 1000 1018 1.0
0.20.1 1.0.1 1000 1000 1018 1.0
0.20.0 1.0.0 1000 1000 1022 1.0
0.19.0 0.11.1 1000 1000 1016 1.0
0.18.0 0.11.1 1000 1000 1016 1.0
0.17.2 0.11.1 1000 1000 1016 1.0
0.17.1 0.11.1 1000 1000 1015 1.0
0.17.0 0.11.1 1000 1000 1017 1.0
0.16.0 0.11.1 1000 1000 1018 1.0
0.15.0 0.11.1 1000 1000 1017 1.0
0.14.2 0.10.0 1000 1000 1017 1.0
0.13.4 0.9.0 1000 1000 1016 1.0
10000 Observations
1.0.1 1.0.1 1000 10000 1018 1.0
1.0.0 1.0.1 1000 10000 1018 1.0
0.23.0 1.0.1 1000 10000 1018 1.0
0.22.1 1.0.1 1000 10000 1018 1.0
0.22.0 1.0.1 1000 10000 1017 1.0
0.21.1 1.0.1 1000 10000 1017 1.0
0.21.0 1.0.1 1000 10000 1017 1.0
0.20.2 1.0.1 1000 10000 1018 1.0
0.20.1 1.0.1 1000 10000 1018 1.0
0.20.0 1.0.0 1000 10000 1022 1.0
0.19.0 0.11.1 1000 10000 1019 1.0
0.18.0 0.11.1 1000 10000 1018 1.0
0.17.2 0.11.1 1000 10000 1018 1.0
0.17.1 0.11.1 1000 10000 1018 1.0
0.17.0 0.11.1 1000 10000 1019 1.0
0.16.0 0.11.1 1000 10000 1020 1.0
0.15.0 0.11.1 1000 10000 1020 1.0
0.14.2 0.10.0 1000 10000 1020 1.0
0.13.4 0.9.0 1000 10000 1020 1.0

Model Time 100 ms

Median runtime of $train() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $train() with models trained for 100 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 100 10 114 1.1
1.0.0 1.0.1 100 10 114 1.1
0.23.0 1.0.1 100 10 112 1.1
0.22.1 1.0.1 100 10 112 1.1
0.22.0 1.0.1 100 10 111 1.1
0.21.1 1.0.1 100 10 113 1.1
0.21.0 1.0.1 100 10 112 1.1
0.20.2 1.0.1 100 10 113 1.1
0.20.1 1.0.1 100 10 113 1.1
0.20.0 1.0.0 100 10 116 1.2
0.19.0 0.11.1 100 10 113 1.1
0.18.0 0.11.1 100 10 113 1.1
0.17.2 0.11.1 100 10 112 1.1
0.17.1 0.11.1 100 10 113 1.1
0.17.0 0.11.1 100 10 112 1.1
0.16.0 0.11.1 100 10 116 1.2
0.15.0 0.11.1 100 10 116 1.2
0.14.2 0.10.0 100 10 117 1.2
0.13.4 0.9.0 100 10 115 1.2
100 Observations
1.0.1 1.0.1 100 100 113 1.1
1.0.0 1.0.1 100 100 112 1.1
0.23.0 1.0.1 100 100 112 1.1
0.22.1 1.0.1 100 100 111 1.1
0.22.0 1.0.1 100 100 113 1.1
0.21.1 1.0.1 100 100 112 1.1
0.21.0 1.0.1 100 100 112 1.1
0.20.2 1.0.1 100 100 114 1.1
0.20.1 1.0.1 100 100 114 1.1
0.20.0 1.0.0 100 100 116 1.2
0.19.0 0.11.1 100 100 113 1.1
0.18.0 0.11.1 100 100 112 1.1
0.17.2 0.11.1 100 100 114 1.1
0.17.1 0.11.1 100 100 114 1.1
0.17.0 0.11.1 100 100 113 1.1
0.16.0 0.11.1 100 100 116 1.2
0.15.0 0.11.1 100 100 116 1.2
0.14.2 0.10.0 100 100 117 1.2
0.13.4 0.9.0 100 100 116 1.2
1000 Observations
1.0.1 1.0.1 100 1000 113 1.1
1.0.0 1.0.1 100 1000 114 1.1
0.23.0 1.0.1 100 1000 112 1.1
0.22.1 1.0.1 100 1000 112 1.1
0.22.0 1.0.1 100 1000 113 1.1
0.21.1 1.0.1 100 1000 112 1.1
0.21.0 1.0.1 100 1000 112 1.1
0.20.2 1.0.1 100 1000 113 1.1
0.20.1 1.0.1 100 1000 113 1.1
0.20.0 1.0.0 100 1000 116 1.2
0.19.0 0.11.1 100 1000 114 1.1
0.18.0 0.11.1 100 1000 112 1.1
0.17.2 0.11.1 100 1000 114 1.1
0.17.1 0.11.1 100 1000 114 1.1
0.17.0 0.11.1 100 1000 113 1.1
0.16.0 0.11.1 100 1000 116 1.2
0.15.0 0.11.1 100 1000 116 1.2
0.14.2 0.10.0 100 1000 116 1.2
0.13.4 0.9.0 100 1000 116 1.2
10000 Observations
1.0.1 1.0.1 100 10000 113 1.1
1.0.0 1.0.1 100 10000 114 1.1
0.23.0 1.0.1 100 10000 113 1.1
0.22.1 1.0.1 100 10000 113 1.1
0.22.0 1.0.1 100 10000 114 1.1
0.21.1 1.0.1 100 10000 114 1.1
0.21.0 1.0.1 100 10000 115 1.1
0.20.2 1.0.1 100 10000 116 1.2
0.20.1 1.0.1 100 10000 114 1.1
0.20.0 1.0.0 100 10000 117 1.2
0.19.0 0.11.1 100 10000 116 1.2
0.18.0 0.11.1 100 10000 114 1.1
0.17.2 0.11.1 100 10000 114 1.1
0.17.1 0.11.1 100 10000 113 1.1
0.17.0 0.11.1 100 10000 114 1.1
0.16.0 0.11.1 100 10000 118 1.2
0.15.0 0.11.1 100 10000 117 1.2
0.14.2 0.10.0 100 10000 117 1.2
0.13.4 0.9.0 100 10000 117 1.2

Model Time 10 ms

Median runtime of $train() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $train() with models trained for 10 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 10 10 21 2.1
1.0.0 1.0.1 10 10 23 2.3
0.23.0 1.0.1 10 10 20 2.0
0.22.1 1.0.1 10 10 24 2.4
0.22.0 1.0.1 10 10 23 2.3
0.21.1 1.0.1 10 10 23 2.3
0.21.0 1.0.1 10 10 23 2.3
0.20.2 1.0.1 10 10 23 2.3
0.20.1 1.0.1 10 10 23 2.3
0.20.0 1.0.0 10 10 24 2.4
0.19.0 0.11.1 10 10 23 2.3
0.18.0 0.11.1 10 10 23 2.3
0.17.2 0.11.1 10 10 21 2.1
0.17.1 0.11.1 10 10 22 2.2
0.17.0 0.11.1 10 10 21 2.1
0.16.0 0.11.1 10 10 24 2.4
0.15.0 0.11.1 10 10 24 2.4
0.14.2 0.10.0 10 10 24 2.4
0.13.4 0.9.0 10 10 24 2.4
100 Observations
1.0.1 1.0.1 10 100 21 2.1
1.0.0 1.0.1 10 100 19 1.9
0.23.0 1.0.1 10 100 20 2.0
0.22.1 1.0.1 10 100 22 2.2
0.22.0 1.0.1 10 100 22 2.2
0.21.1 1.0.1 10 100 21 2.1
0.21.0 1.0.1 10 100 20 2.0
0.20.2 1.0.1 10 100 19 1.9
0.20.1 1.0.1 10 100 20 2.0
0.20.0 1.0.0 10 100 25 2.5
0.19.0 0.11.1 10 100 23 2.3
0.18.0 0.11.1 10 100 23 2.3
0.17.2 0.11.1 10 100 23 2.3
0.17.1 0.11.1 10 100 21 2.1
0.17.0 0.11.1 10 100 24 2.4
0.16.0 0.11.1 10 100 24 2.4
0.15.0 0.11.1 10 100 24 2.4
0.14.2 0.10.0 10 100 24 2.4
0.13.4 0.9.0 10 100 24 2.4
1000 Observations
1.0.1 1.0.1 10 1000 23 2.3
1.0.0 1.0.1 10 1000 23 2.3
0.23.0 1.0.1 10 1000 19 1.9
0.22.1 1.0.1 10 1000 22 2.2
0.22.0 1.0.1 10 1000 23 2.3
0.21.1 1.0.1 10 1000 23 2.3
0.21.0 1.0.1 10 1000 24 2.4
0.20.2 1.0.1 10 1000 23 2.3
0.20.1 1.0.1 10 1000 24 2.4
0.20.0 1.0.0 10 1000 23 2.3
0.19.0 0.11.1 10 1000 20 2.0
0.18.0 0.11.1 10 1000 23 2.3
0.17.2 0.11.1 10 1000 22 2.2
0.17.1 0.11.1 10 1000 23 2.3
0.17.0 0.11.1 10 1000 22 2.2
0.16.0 0.11.1 10 1000 24 2.4
0.15.0 0.11.1 10 1000 24 2.4
0.14.2 0.10.0 10 1000 24 2.4
0.13.4 0.9.0 10 1000 24 2.4
10000 Observations
1.0.1 1.0.1 10 10000 22 2.2
1.0.0 1.0.1 10 10000 22 2.2
0.23.0 1.0.1 10 10000 22 2.2
0.22.1 1.0.1 10 10000 24 2.4
0.22.0 1.0.1 10 10000 21 2.1
0.21.1 1.0.1 10 10000 24 2.4
0.21.0 1.0.1 10 10000 23 2.3
0.20.2 1.0.1 10 10000 23 2.3
0.20.1 1.0.1 10 10000 23 2.3
0.20.0 1.0.0 10 10000 26 2.6
0.19.0 0.11.1 10 10000 24 2.4
0.18.0 0.11.1 10 10000 20 2.0
0.17.2 0.11.1 10 10000 21 2.1
0.17.1 0.11.1 10 10000 24 2.4
0.17.0 0.11.1 10 10000 24 2.4
0.16.0 0.11.1 10 10000 24 2.4
0.15.0 0.11.1 10 10000 24 2.4
0.14.2 0.10.0 10 10000 24 2.4
0.13.4 0.9.0 10 10000 24 2.4

Model Time 1 ms

Median runtime of $train() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $train() with models trained for 1 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 1 10 9 9.4
1.0.0 1.0.1 1 10 8 8.4
0.23.0 1.0.1 1 10 8 8.2
0.22.1 1.0.1 1 10 8 8.0
0.22.0 1.0.1 1 10 8 7.8
0.21.1 1.0.1 1 10 11 11
0.21.0 1.0.1 1 10 8 7.6
0.20.2 1.0.1 1 10 10 10
0.20.1 1.0.1 1 10 8 8.2
0.20.0 1.0.0 1 10 11 11
0.19.0 0.11.1 1 10 9 8.8
0.18.0 0.11.1 1 10 9 8.7
0.17.2 0.11.1 1 10 9 9.4
0.17.1 0.11.1 1 10 8 8.2
0.17.0 0.11.1 1 10 8 8.3
0.16.0 0.11.1 1 10 11 11
0.15.0 0.11.1 1 10 9 9.4
0.14.2 0.10.0 1 10 9 8.6
0.13.4 0.9.0 1 10 11 11
100 Observations
1.0.1 1.0.1 1 100 8 8.1
1.0.0 1.0.1 1 100 8 8.2
0.23.0 1.0.1 1 100 8 8.3
0.22.1 1.0.1 1 100 8 7.8
0.22.0 1.0.1 1 100 9 8.5
0.21.1 1.0.1 1 100 7 7.4
0.21.0 1.0.1 1 100 8 7.7
0.20.2 1.0.1 1 100 8 8.5
0.20.1 1.0.1 1 100 8 8.1
0.20.0 1.0.0 1 100 11 11
0.19.0 0.11.1 1 100 8 8.4
0.18.0 0.11.1 1 100 8 8.4
0.17.2 0.11.1 1 100 9 9.1
0.17.1 0.11.1 1 100 8 8.3
0.17.0 0.11.1 1 100 8 7.8
0.16.0 0.11.1 1 100 10 10
0.15.0 0.11.1 1 100 10 9.8
0.14.2 0.10.0 1 100 11 11
0.13.4 0.9.0 1 100 10 9.7
1000 Observations
1.0.1 1.0.1 1 1000 11 11
1.0.0 1.0.1 1 1000 10 9.8
0.23.0 1.0.1 1 1000 8 8.0
0.22.1 1.0.1 1 1000 8 8.1
0.22.0 1.0.1 1 1000 8 8.2
0.21.1 1.0.1 1 1000 8 8.4
0.21.0 1.0.1 1 1000 8 7.7
0.20.2 1.0.1 1 1000 9 8.6
0.20.1 1.0.1 1 1000 9 8.5
0.20.0 1.0.0 1 1000 13 13
0.19.0 0.11.1 1 1000 8 8.5
0.18.0 0.11.1 1 1000 8 8.3
0.17.2 0.11.1 1 1000 8 8.1
0.17.1 0.11.1 1 1000 8 7.9
0.17.0 0.11.1 1 1000 9 9.0
0.16.0 0.11.1 1 1000 13 13
0.15.0 0.11.1 1 1000 9 9.5
0.14.2 0.10.0 1 1000 9 9.2
0.13.4 0.9.0 1 1000 11 11
10000 Observations
1.0.1 1.0.1 1 10000 9 9.2
1.0.0 1.0.1 1 10000 12 12
0.23.0 1.0.1 1 10000 13 13
0.22.1 1.0.1 1 10000 10 9.9
0.22.0 1.0.1 1 10000 12 12
0.21.1 1.0.1 1 10000 11 11
0.21.0 1.0.1 1 10000 11 11
0.20.2 1.0.1 1 10000 10 10
0.20.1 1.0.1 1 10000 10 9.6
0.20.0 1.0.0 1 10000 13 13
0.19.0 0.11.1 1 10000 10 10
0.18.0 0.11.1 1 10000 10 9.7
0.17.2 0.11.1 1 10000 10 10
0.17.1 0.11.1 1 10000 10 9.5
0.17.0 0.11.1 1 10000 10 9.7
0.16.0 0.11.1 1 10000 10 10
0.15.0 0.11.1 1 10000 11 11
0.14.2 0.10.0 1 10000 10 10
0.13.4 0.9.0 1 10000 13 13

Predict

The runtime of the $predict() method is measured for different mlr3 versions. The predict step is performed for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with different amounts of observations (10, 100, 1000, and 10000).

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

learner$train(task)

learner$predict(task)

Model Time 1000 ms

Median runtime of $predict() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict() with models trained for 1000 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 1000 10 1016 1.0
1.0.0 1.0.1 1000 10 1016 1.0
0.23.0 1.0.1 1000 10 1017 1.0
0.22.1 1.0.1 1000 10 1017 1.0
0.22.0 1.0.1 1000 10 1016 1.0
0.21.1 1.0.1 1000 10 1016 1.0
0.21.0 1.0.1 1000 10 1017 1.0
0.20.2 1.0.1 1000 10 1012 1.0
0.20.1 1.0.1 1000 10 1012 1.0
0.20.0 1.0.0 1000 10 1016 1.0
0.19.0 0.11.1 1000 10 1012 1.0
0.18.0 0.11.1 1000 10 1011 1.0
0.17.2 0.11.1 1000 10 1012 1.0
0.17.1 0.11.1 1000 10 1012 1.0
0.17.0 0.11.1 1000 10 1011 1.0
0.16.0 0.11.1 1000 10 1013 1.0
0.15.0 0.11.1 1000 10 1012 1.0
0.14.2 0.10.0 1000 10 1014 1.0
0.13.4 0.9.0 1000 10 1013 1.0
100 Observations
1.0.1 1.0.1 1000 100 1016 1.0
1.0.0 1.0.1 1000 100 1018 1.0
0.23.0 1.0.1 1000 100 1016 1.0
0.22.1 1.0.1 1000 100 1017 1.0
0.22.0 1.0.1 1000 100 1016 1.0
0.21.1 1.0.1 1000 100 1017 1.0
0.21.0 1.0.1 1000 100 1017 1.0
0.20.2 1.0.1 1000 100 1013 1.0
0.20.1 1.0.1 1000 100 1013 1.0
0.20.0 1.0.0 1000 100 1017 1.0
0.19.0 0.11.1 1000 100 1011 1.0
0.18.0 0.11.1 1000 100 1012 1.0
0.17.2 0.11.1 1000 100 1012 1.0
0.17.1 0.11.1 1000 100 1011 1.0
0.17.0 0.11.1 1000 100 1011 1.0
0.16.0 0.11.1 1000 100 1014 1.0
0.15.0 0.11.1 1000 100 1014 1.0
0.14.2 0.10.0 1000 100 1012 1.0
0.13.4 0.9.0 1000 100 1013 1.0
1000 Observations
1.0.1 1.0.1 1000 1000 1017 1.0
1.0.0 1.0.1 1000 1000 1016 1.0
0.23.0 1.0.1 1000 1000 1016 1.0
0.22.1 1.0.1 1000 1000 1018 1.0
0.22.0 1.0.1 1000 1000 1017 1.0
0.21.1 1.0.1 1000 1000 1017 1.0
0.21.0 1.0.1 1000 1000 1019 1.0
0.20.2 1.0.1 1000 1000 1014 1.0
0.20.1 1.0.1 1000 1000 1013 1.0
0.20.0 1.0.0 1000 1000 1017 1.0
0.19.0 0.11.1 1000 1000 1010 1.0
0.18.0 0.11.1 1000 1000 1011 1.0
0.17.2 0.11.1 1000 1000 1012 1.0
0.17.1 0.11.1 1000 1000 1012 1.0
0.17.0 0.11.1 1000 1000 1011 1.0
0.16.0 0.11.1 1000 1000 1013 1.0
0.15.0 0.11.1 1000 1000 1014 1.0
0.14.2 0.10.0 1000 1000 1014 1.0
0.13.4 0.9.0 1000 1000 1013 1.0
10000 Observations
1.0.1 1.0.1 1000 10000 1018 1.0
1.0.0 1.0.1 1000 10000 1019 1.0
0.23.0 1.0.1 1000 10000 1017 1.0
0.22.1 1.0.1 1000 10000 1018 1.0
0.22.0 1.0.1 1000 10000 1017 1.0
0.21.1 1.0.1 1000 10000 1019 1.0
0.21.0 1.0.1 1000 10000 1019 1.0
0.20.2 1.0.1 1000 10000 1015 1.0
0.20.1 1.0.1 1000 10000 1016 1.0
0.20.0 1.0.0 1000 10000 1019 1.0
0.19.0 0.11.1 1000 10000 1014 1.0
0.18.0 0.11.1 1000 10000 1014 1.0
0.17.2 0.11.1 1000 10000 1015 1.0
0.17.1 0.11.1 1000 10000 1014 1.0
0.17.0 0.11.1 1000 10000 1015 1.0
0.16.0 0.11.1 1000 10000 1016 1.0
0.15.0 0.11.1 1000 10000 1016 1.0
0.14.2 0.10.0 1000 10000 1015 1.0
0.13.4 0.9.0 1000 10000 1016 1.0

Model Time 100 ms

Median runtime of $predict() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict() with models trained for 100 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 100 10 113 1.1
1.0.0 1.0.1 100 10 113 1.1
0.23.0 1.0.1 100 10 112 1.1
0.22.1 1.0.1 100 10 113 1.1
0.22.0 1.0.1 100 10 113 1.1
0.21.1 1.0.1 100 10 113 1.1
0.21.0 1.0.1 100 10 112 1.1
0.20.2 1.0.1 100 10 110 1.1
0.20.1 1.0.1 100 10 109 1.1
0.20.0 1.0.0 100 10 115 1.2
0.19.0 0.11.1 100 10 110 1.1
0.18.0 0.11.1 100 10 109 1.1
0.17.2 0.11.1 100 10 110 1.1
0.17.1 0.11.1 100 10 110 1.1
0.17.0 0.11.1 100 10 110 1.1
0.16.0 0.11.1 100 10 112 1.1
0.15.0 0.11.1 100 10 112 1.1
0.14.2 0.10.0 100 10 113 1.1
0.13.4 0.9.0 100 10 112 1.1
100 Observations
1.0.1 1.0.1 100 100 113 1.1
1.0.0 1.0.1 100 100 114 1.1
0.23.0 1.0.1 100 100 113 1.1
0.22.1 1.0.1 100 100 113 1.1
0.22.0 1.0.1 100 100 113 1.1
0.21.1 1.0.1 100 100 113 1.1
0.21.0 1.0.1 100 100 112 1.1
0.20.2 1.0.1 100 100 110 1.1
0.20.1 1.0.1 100 100 110 1.1
0.20.0 1.0.0 100 100 114 1.1
0.19.0 0.11.1 100 100 110 1.1
0.18.0 0.11.1 100 100 109 1.1
0.17.2 0.11.1 100 100 108 1.1
0.17.1 0.11.1 100 100 110 1.1
0.17.0 0.11.1 100 100 110 1.1
0.16.0 0.11.1 100 100 113 1.1
0.15.0 0.11.1 100 100 113 1.1
0.14.2 0.10.0 100 100 112 1.1
0.13.4 0.9.0 100 100 112 1.1
1000 Observations
1.0.1 1.0.1 100 1000 114 1.1
1.0.0 1.0.1 100 1000 113 1.1
0.23.0 1.0.1 100 1000 113 1.1
0.22.1 1.0.1 100 1000 113 1.1
0.22.0 1.0.1 100 1000 113 1.1
0.21.1 1.0.1 100 1000 114 1.1
0.21.0 1.0.1 100 1000 114 1.1
0.20.2 1.0.1 100 1000 110 1.1
0.20.1 1.0.1 100 1000 111 1.1
0.20.0 1.0.0 100 1000 115 1.1
0.19.0 0.11.1 100 1000 111 1.1
0.18.0 0.11.1 100 1000 110 1.1
0.17.2 0.11.1 100 1000 110 1.1
0.17.1 0.11.1 100 1000 109 1.1
0.17.0 0.11.1 100 1000 110 1.1
0.16.0 0.11.1 100 1000 113 1.1
0.15.0 0.11.1 100 1000 113 1.1
0.14.2 0.10.0 100 1000 112 1.1
0.13.4 0.9.0 100 1000 112 1.1
10000 Observations
1.0.1 1.0.1 100 10000 114 1.1
1.0.0 1.0.1 100 10000 115 1.2
0.23.0 1.0.1 100 10000 116 1.2
0.22.1 1.0.1 100 10000 116 1.2
0.22.0 1.0.1 100 10000 115 1.1
0.21.1 1.0.1 100 10000 114 1.1
0.21.0 1.0.1 100 10000 115 1.1
0.20.2 1.0.1 100 10000 112 1.1
0.20.1 1.0.1 100 10000 113 1.1
0.20.0 1.0.0 100 10000 117 1.2
0.19.0 0.11.1 100 10000 112 1.1
0.18.0 0.11.1 100 10000 112 1.1
0.17.2 0.11.1 100 10000 111 1.1
0.17.1 0.11.1 100 10000 111 1.1
0.17.0 0.11.1 100 10000 113 1.1
0.16.0 0.11.1 100 10000 115 1.1
0.15.0 0.11.1 100 10000 115 1.1
0.14.2 0.10.0 100 10000 114 1.1
0.13.4 0.9.0 100 10000 114 1.1

Model Time 10 ms

Median runtime of $predict() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict() with models trained for 10 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 10 10 22 2.2
1.0.0 1.0.1 10 10 23 2.3
0.23.0 1.0.1 10 10 23 2.3
0.22.1 1.0.1 10 10 24 2.4
0.22.0 1.0.1 10 10 21 2.1
0.21.1 1.0.1 10 10 22 2.2
0.21.0 1.0.1 10 10 24 2.4
0.20.2 1.0.1 10 10 15 1.5
0.20.1 1.0.1 10 10 15 1.5
0.20.0 1.0.0 10 10 24 2.4
0.19.0 0.11.1 10 10 16 1.6
0.18.0 0.11.1 10 10 16 1.6
0.17.2 0.11.1 10 10 15 1.5
0.17.1 0.11.1 10 10 15 1.5
0.17.0 0.11.1 10 10 15 1.5
0.16.0 0.11.1 10 10 21 2.1
0.15.0 0.11.1 10 10 20 2.0
0.14.2 0.10.0 10 10 20 2.0
0.13.4 0.9.0 10 10 21 2.1
100 Observations
1.0.1 1.0.1 10 100 24 2.4
1.0.0 1.0.1 10 100 23 2.3
0.23.0 1.0.1 10 100 23 2.3
0.22.1 1.0.1 10 100 22 2.2
0.22.0 1.0.1 10 100 24 2.4
0.21.1 1.0.1 10 100 21 2.1
0.21.0 1.0.1 10 100 22 2.2
0.20.2 1.0.1 10 100 16 1.6
0.20.1 1.0.1 10 100 16 1.6
0.20.0 1.0.0 10 100 23 2.3
0.19.0 0.11.1 10 100 16 1.6
0.18.0 0.11.1 10 100 15 1.5
0.17.2 0.11.1 10 100 15 1.5
0.17.1 0.11.1 10 100 15 1.5
0.17.0 0.11.1 10 100 15 1.5
0.16.0 0.11.1 10 100 20 2.0
0.15.0 0.11.1 10 100 20 2.0
0.14.2 0.10.0 10 100 20 2.0
0.13.4 0.9.0 10 100 21 2.1
1000 Observations
1.0.1 1.0.1 10 1000 22 2.2
1.0.0 1.0.1 10 1000 22 2.2
0.23.0 1.0.1 10 1000 23 2.3
0.22.1 1.0.1 10 1000 23 2.3
0.22.0 1.0.1 10 1000 23 2.3
0.21.1 1.0.1 10 1000 22 2.2
0.21.0 1.0.1 10 1000 23 2.3
0.20.2 1.0.1 10 1000 16 1.6
0.20.1 1.0.1 10 1000 16 1.6
0.20.0 1.0.0 10 1000 23 2.3
0.19.0 0.11.1 10 1000 16 1.6
0.18.0 0.11.1 10 1000 16 1.6
0.17.2 0.11.1 10 1000 15 1.5
0.17.1 0.11.1 10 1000 15 1.5
0.17.0 0.11.1 10 1000 15 1.5
0.16.0 0.11.1 10 1000 21 2.1
0.15.0 0.11.1 10 1000 22 2.2
0.14.2 0.10.0 10 1000 22 2.2
0.13.4 0.9.0 10 1000 22 2.2
10000 Observations
1.0.1 1.0.1 10 10000 24 2.4
1.0.0 1.0.1 10 10000 23 2.3
0.23.0 1.0.1 10 10000 24 2.4
0.22.1 1.0.1 10 10000 23 2.3
0.22.0 1.0.1 10 10000 24 2.4
0.21.1 1.0.1 10 10000 24 2.4
0.21.0 1.0.1 10 10000 23 2.3
0.20.2 1.0.1 10 10000 20 2.0
0.20.1 1.0.1 10 10000 20 2.0
0.20.0 1.0.0 10 10000 24 2.4
0.19.0 0.11.1 10 10000 19 1.9
0.18.0 0.11.1 10 10000 20 2.0
0.17.2 0.11.1 10 10000 19 1.9
0.17.1 0.11.1 10 10000 18 1.8
0.17.0 0.11.1 10 10000 19 1.9
0.16.0 0.11.1 10 10000 23 2.3
0.15.0 0.11.1 10 10000 23 2.3
0.14.2 0.10.0 10 10000 22 2.2
0.13.4 0.9.0 10 10000 22 2.2

Model Time 1 ms

Median runtime of $predict() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict() with models trained for 1 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
10 Observations
1.0.1 1.0.1 1 10 8 8.4
1.0.0 1.0.1 1 10 8 8.2
0.23.0 1.0.1 1 10 8 8.4
0.22.1 1.0.1 1 10 9 9.0
0.22.0 1.0.1 1 10 8 8.3
0.21.1 1.0.1 1 10 8 8.0
0.21.0 1.0.1 1 10 8 8.1
0.20.2 1.0.1 1 10 6 5.5
0.20.1 1.0.1 1 10 6 5.8
0.20.0 1.0.0 1 10 9 8.8
0.19.0 0.11.1 1 10 5 5.3
0.18.0 0.11.1 1 10 6 6.0
0.17.2 0.11.1 1 10 5 5.3
0.17.1 0.11.1 1 10 5 5.3
0.17.0 0.11.1 1 10 5 5.4
0.16.0 0.11.1 1 10 7 6.7
0.15.0 0.11.1 1 10 7 6.9
0.14.2 0.10.0 1 10 7 6.8
0.13.4 0.9.0 1 10 7 7.0
100 Observations
1.0.1 1.0.1 1 100 9 8.5
1.0.0 1.0.1 1 100 9 8.6
0.23.0 1.0.1 1 100 8 7.9
0.22.1 1.0.1 1 100 8 8.3
0.22.0 1.0.1 1 100 8 8.2
0.21.1 1.0.1 1 100 8 8.2
0.21.0 1.0.1 1 100 8 8.5
0.20.2 1.0.1 1 100 6 5.5
0.20.1 1.0.1 1 100 6 5.5
0.20.0 1.0.0 1 100 9 8.7
0.19.0 0.11.1 1 100 5 5.1
0.18.0 0.11.1 1 100 5 5.4
0.17.2 0.11.1 1 100 5 5.4
0.17.1 0.11.1 1 100 6 5.7
0.17.0 0.11.1 1 100 5 5.3
0.16.0 0.11.1 1 100 7 6.6
0.15.0 0.11.1 1 100 6 6.2
0.14.2 0.10.0 1 100 6 6.3
0.13.4 0.9.0 1 100 7 6.6
1000 Observations
1.0.1 1.0.1 1 1000 9 9.1
1.0.0 1.0.1 1 1000 9 9.3
0.23.0 1.0.1 1 1000 8 8.2
0.22.1 1.0.1 1 1000 9 8.5
0.22.0 1.0.1 1 1000 8 8.4
0.21.1 1.0.1 1 1000 8 8.2
0.21.0 1.0.1 1 1000 9 8.8
0.20.2 1.0.1 1 1000 6 6.1
0.20.1 1.0.1 1 1000 6 6.2
0.20.0 1.0.0 1 1000 9 8.7
0.19.0 0.11.1 1 1000 5 5.5
0.18.0 0.11.1 1 1000 5 5.4
0.17.2 0.11.1 1 1000 5 5.4
0.17.1 0.11.1 1 1000 5 5.5
0.17.0 0.11.1 1 1000 6 5.9
0.16.0 0.11.1 1 1000 7 6.5
0.15.0 0.11.1 1 1000 7 7.0
0.14.2 0.10.0 1 1000 7 6.6
0.13.4 0.9.0 1 1000 7 7.4
10000 Observations
1.0.1 1.0.1 1 10000 10 10
1.0.0 1.0.1 1 10000 11 11
0.23.0 1.0.1 1 10000 10 10
0.22.1 1.0.1 1 10000 11 11
0.22.0 1.0.1 1 10000 10 10
0.21.1 1.0.1 1 10000 10 10
0.21.0 1.0.1 1 10000 10 9.8
0.20.2 1.0.1 1 10000 8 8.2
0.20.1 1.0.1 1 10000 8 7.7
0.20.0 1.0.0 1 10000 11 11
0.19.0 0.11.1 1 10000 7 7.5
0.18.0 0.11.1 1 10000 8 8.2
0.17.2 0.11.1 1 10000 8 7.6
0.17.1 0.11.1 1 10000 9 8.9
0.17.0 0.11.1 1 10000 7 7.5
0.16.0 0.11.1 1 10000 8 8.2
0.15.0 0.11.1 1 10000 8 8.3
0.14.2 0.10.0 1 10000 8 8.1
0.13.4 0.9.0 1 10000 9 8.9

Predict Newdata

The runtime of the $predict_newdata() method is measured for different mlr3 versions. The predict step is performed for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with different amounts of observations (1, 10, 100, 1000, and 10000).

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

learner$train(task)

learner$predict_newdata(newdata)

Model Time 1000 ms

Median runtime of $predict_newdata() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata() with models trained for 1000 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.0.1 1.0.1 1000 1 1043 1.0
1.0.0 1.0.1 1000 1 1045 1.0
0.23.0 1.0.1 1000 1 1043 1.0
0.22.0 1.0.1 1000 1 1021 1.0
0.21.1 1.0.1 1000 1 1021 1.0
0.21.0 1.0.1 1000 1 1021 1.0
0.20.2 1.0.1 1000 1 1020 1.0
0.20.1 1.0.1 1000 1 1022 1.0
0.20.0 1.0.0 1000 1 1026 1.0
0.19.0 0.11.1 1000 1 1018 1.0
0.18.0 0.11.1 1000 1 1018 1.0
0.17.2 0.11.1 1000 1 1019 1.0
0.17.1 0.11.1 1000 1 1019 1.0
0.17.0 0.11.1 1000 1 1019 1.0
0.16.0 0.11.1 1000 1 1020 1.0
0.15.0 0.11.1 1000 1 1019 1.0
0.14.2 0.10.0 1000 1 1021 1.0
0.13.4 0.9.0 1000 1 1020 1.0
10 Observations
1.0.1 1.0.1 1000 10 1040 1.0
1.0.0 1.0.1 1000 10 1043 1.0
0.23.0 1.0.1 1000 10 1045 1.0
0.22.0 1.0.1 1000 10 1021 1.0
0.21.1 1.0.1 1000 10 1023 1.0
0.21.0 1.0.1 1000 10 1022 1.0
0.20.2 1.0.1 1000 10 1020 1.0
0.20.1 1.0.1 1000 10 1020 1.0
0.20.0 1.0.0 1000 10 1025 1.0
0.19.0 0.11.1 1000 10 1019 1.0
0.18.0 0.11.1 1000 10 1018 1.0
0.17.2 0.11.1 1000 10 1019 1.0
0.17.1 0.11.1 1000 10 1017 1.0
0.17.0 0.11.1 1000 10 1019 1.0
0.16.0 0.11.1 1000 10 1020 1.0
0.15.0 0.11.1 1000 10 1020 1.0
0.14.2 0.10.0 1000 10 1020 1.0
0.13.4 0.9.0 1000 10 1020 1.0
100 Observations
1.0.1 1.0.1 1000 100 1042 1.0
1.0.0 1.0.1 1000 100 1040 1.0
0.23.0 1.0.1 1000 100 1039 1.0
0.22.0 1.0.1 1000 100 1022 1.0
0.21.1 1.0.1 1000 100 1022 1.0
0.21.0 1.0.1 1000 100 1021 1.0
0.20.2 1.0.1 1000 100 1021 1.0
0.20.1 1.0.1 1000 100 1020 1.0
0.20.0 1.0.0 1000 100 1025 1.0
0.19.0 0.11.1 1000 100 1020 1.0
0.18.0 0.11.1 1000 100 1018 1.0
0.17.2 0.11.1 1000 100 1019 1.0
0.17.1 0.11.1 1000 100 1018 1.0
0.17.0 0.11.1 1000 100 1019 1.0
0.16.0 0.11.1 1000 100 1020 1.0
0.15.0 0.11.1 1000 100 1021 1.0
0.14.2 0.10.0 1000 100 1019 1.0
0.13.4 0.9.0 1000 100 1020 1.0
1000 Observations
1.0.1 1.0.1 1000 1000 1043 1.0
1.0.0 1.0.1 1000 1000 1045 1.0
0.23.0 1.0.1 1000 1000 1041 1.0
0.22.0 1.0.1 1000 1000 1023 1.0
0.21.1 1.0.1 1000 1000 1023 1.0
0.21.0 1.0.1 1000 1000 1022 1.0
0.20.2 1.0.1 1000 1000 1022 1.0
0.20.1 1.0.1 1000 1000 1021 1.0
0.20.0 1.0.0 1000 1000 1025 1.0
0.19.0 0.11.1 1000 1000 1019 1.0
0.18.0 0.11.1 1000 1000 1020 1.0
0.17.2 0.11.1 1000 1000 1020 1.0
0.17.1 0.11.1 1000 1000 1020 1.0
0.17.0 0.11.1 1000 1000 1019 1.0
0.16.0 0.11.1 1000 1000 1021 1.0
0.15.0 0.11.1 1000 1000 1021 1.0
0.14.2 0.10.0 1000 1000 1021 1.0
0.13.4 0.9.0 1000 1000 1022 1.0
10000 Observations
1.0.1 1.0.1 1000 10000 1059 1.1
1.0.0 1.0.1 1000 10000 1060 1.1
0.23.0 1.0.1 1000 10000 1067 1.1
0.22.0 1.0.1 1000 10000 1025 1.0
0.21.1 1.0.1 1000 10000 1024 1.0
0.21.0 1.0.1 1000 10000 1025 1.0
0.20.2 1.0.1 1000 10000 1024 1.0
0.20.1 1.0.1 1000 10000 1024 1.0
0.20.0 1.0.0 1000 10000 1028 1.0
0.19.0 0.11.1 1000 10000 1025 1.0
0.18.0 0.11.1 1000 10000 1022 1.0
0.17.2 0.11.1 1000 10000 1025 1.0
0.17.1 0.11.1 1000 10000 1025 1.0
0.17.0 0.11.1 1000 10000 1024 1.0
0.16.0 0.11.1 1000 10000 1026 1.0
0.15.0 0.11.1 1000 10000 1026 1.0
0.14.2 0.10.0 1000 10000 1026 1.0
0.13.4 0.9.0 1000 10000 1025 1.0

Model Time 100 ms

Median runtime of $predict_newdata() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata() with models trained for 100 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.0.1 1.0.1 100 1 136 1.4
1.0.0 1.0.1 100 1 132 1.3
0.23.0 1.0.1 100 1 135 1.4
0.22.0 1.0.1 100 1 116 1.2
0.21.1 1.0.1 100 1 115 1.1
0.21.0 1.0.1 100 1 116 1.2
0.20.2 1.0.1 100 1 115 1.1
0.20.1 1.0.1 100 1 116 1.2
0.20.0 1.0.0 100 1 118 1.2
0.19.0 0.11.1 100 1 116 1.2
0.18.0 0.11.1 100 1 115 1.2
0.17.2 0.11.1 100 1 115 1.2
0.17.1 0.11.1 100 1 115 1.2
0.17.0 0.11.1 100 1 116 1.2
0.16.0 0.11.1 100 1 120 1.2
0.15.0 0.11.1 100 1 119 1.2
0.14.2 0.10.0 100 1 120 1.2
0.13.4 0.9.0 100 1 120 1.2
10 Observations
1.0.1 1.0.1 100 10 136 1.4
1.0.0 1.0.1 100 10 135 1.4
0.23.0 1.0.1 100 10 132 1.3
0.22.0 1.0.1 100 10 117 1.2
0.21.1 1.0.1 100 10 115 1.2
0.21.0 1.0.1 100 10 116 1.2
0.20.2 1.0.1 100 10 113 1.1
0.20.1 1.0.1 100 10 115 1.1
0.20.0 1.0.0 100 10 118 1.2
0.19.0 0.11.1 100 10 115 1.2
0.18.0 0.11.1 100 10 114 1.1
0.17.2 0.11.1 100 10 115 1.2
0.17.1 0.11.1 100 10 115 1.1
0.17.0 0.11.1 100 10 115 1.1
0.16.0 0.11.1 100 10 120 1.2
0.15.0 0.11.1 100 10 119 1.2
0.14.2 0.10.0 100 10 120 1.2
0.13.4 0.9.0 100 10 120 1.2
100 Observations
1.0.1 1.0.1 100 100 132 1.3
1.0.0 1.0.1 100 100 132 1.3
0.23.0 1.0.1 100 100 133 1.3
0.22.0 1.0.1 100 100 116 1.2
0.21.1 1.0.1 100 100 115 1.2
0.21.0 1.0.1 100 100 116 1.2
0.20.2 1.0.1 100 100 115 1.2
0.20.1 1.0.1 100 100 116 1.2
0.20.0 1.0.0 100 100 119 1.2
0.19.0 0.11.1 100 100 115 1.2
0.18.0 0.11.1 100 100 114 1.1
0.17.2 0.11.1 100 100 114 1.1
0.17.1 0.11.1 100 100 115 1.1
0.17.0 0.11.1 100 100 114 1.1
0.16.0 0.11.1 100 100 120 1.2
0.15.0 0.11.1 100 100 120 1.2
0.14.2 0.10.0 100 100 120 1.2
0.13.4 0.9.0 100 100 120 1.2
1000 Observations
1.0.1 1.0.1 100 1000 135 1.3
1.0.0 1.0.1 100 1000 134 1.3
0.23.0 1.0.1 100 1000 135 1.4
0.22.0 1.0.1 100 1000 117 1.2
0.21.1 1.0.1 100 1000 114 1.1
0.21.0 1.0.1 100 1000 115 1.1
0.20.2 1.0.1 100 1000 116 1.2
0.20.1 1.0.1 100 1000 114 1.1
0.20.0 1.0.0 100 1000 119 1.2
0.19.0 0.11.1 100 1000 118 1.2
0.18.0 0.11.1 100 1000 115 1.1
0.17.2 0.11.1 100 1000 115 1.2
0.17.1 0.11.1 100 1000 116 1.2
0.17.0 0.11.1 100 1000 115 1.2
0.16.0 0.11.1 100 1000 120 1.2
0.15.0 0.11.1 100 1000 120 1.2
0.14.2 0.10.0 100 1000 120 1.2
0.13.4 0.9.0 100 1000 120 1.2
10000 Observations
1.0.1 1.0.1 100 10000 147 1.5
1.0.0 1.0.1 100 10000 145 1.4
0.23.0 1.0.1 100 10000 148 1.5
0.22.0 1.0.1 100 10000 120 1.2
0.21.1 1.0.1 100 10000 120 1.2
0.21.0 1.0.1 100 10000 121 1.2
0.20.2 1.0.1 100 10000 118 1.2
0.20.1 1.0.1 100 10000 119 1.2
0.20.0 1.0.0 100 10000 122 1.2
0.19.0 0.11.1 100 10000 118 1.2
0.18.0 0.11.1 100 10000 121 1.2
0.17.2 0.11.1 100 10000 120 1.2
0.17.1 0.11.1 100 10000 119 1.2
0.17.0 0.11.1 100 10000 118 1.2
0.16.0 0.11.1 100 10000 122 1.2
0.15.0 0.11.1 100 10000 124 1.2
0.14.2 0.10.0 100 10000 123 1.2
0.13.4 0.9.0 100 10000 124 1.2

Model Time 10 ms

Median runtime of $predict_newdata() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata() with models trained for 10 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.0.1 1.0.1 10 1 45 4.5
1.0.0 1.0.1 10 1 44 4.4
0.23.0 1.0.1 10 1 41 4.1
0.22.0 1.0.1 10 1 23 2.3
0.21.1 1.0.1 10 1 24 2.4
0.21.0 1.0.1 10 1 25 2.5
0.20.2 1.0.1 10 1 24 2.4
0.20.1 1.0.1 10 1 24 2.4
0.20.0 1.0.0 10 1 28 2.8
0.19.0 0.11.1 10 1 24 2.4
0.18.0 0.11.1 10 1 23 2.3
0.17.2 0.11.1 10 1 24 2.4
0.17.1 0.11.1 10 1 24 2.4
0.17.0 0.11.1 10 1 24 2.4
0.16.0 0.11.1 10 1 24 2.4
0.15.0 0.11.1 10 1 25 2.5
0.14.2 0.10.0 10 1 24 2.4
0.13.4 0.9.0 10 1 24 2.4
10 Observations
1.0.1 1.0.1 10 10 42 4.2
1.0.0 1.0.1 10 10 41 4.1
0.23.0 1.0.1 10 10 41 4.1
0.22.0 1.0.1 10 10 25 2.5
0.21.1 1.0.1 10 10 25 2.5
0.21.0 1.0.1 10 10 25 2.5
0.20.2 1.0.1 10 10 24 2.4
0.20.1 1.0.1 10 10 24 2.4
0.20.0 1.0.0 10 10 27 2.7
0.19.0 0.11.1 10 10 24 2.4
0.18.0 0.11.1 10 10 24 2.4
0.17.2 0.11.1 10 10 24 2.4
0.17.1 0.11.1 10 10 24 2.4
0.17.0 0.11.1 10 10 24 2.4
0.16.0 0.11.1 10 10 24 2.4
0.15.0 0.11.1 10 10 24 2.4
0.14.2 0.10.0 10 10 24 2.4
0.13.4 0.9.0 10 10 24 2.4
100 Observations
1.0.1 1.0.1 10 100 41 4.1
1.0.0 1.0.1 10 100 40 4.0
0.23.0 1.0.1 10 100 41 4.1
0.22.0 1.0.1 10 100 25 2.5
0.21.1 1.0.1 10 100 24 2.4
0.21.0 1.0.1 10 100 25 2.5
0.20.2 1.0.1 10 100 25 2.5
0.20.1 1.0.1 10 100 24 2.4
0.20.0 1.0.0 10 100 27 2.7
0.19.0 0.11.1 10 100 25 2.5
0.18.0 0.11.1 10 100 24 2.4
0.17.2 0.11.1 10 100 24 2.4
0.17.1 0.11.1 10 100 24 2.4
0.17.0 0.11.1 10 100 24 2.4
0.16.0 0.11.1 10 100 24 2.4
0.15.0 0.11.1 10 100 24 2.4
0.14.2 0.10.0 10 100 23 2.3
0.13.4 0.9.0 10 100 24 2.4
1000 Observations
1.0.1 1.0.1 10 1000 42 4.2
1.0.0 1.0.1 10 1000 44 4.4
0.23.0 1.0.1 10 1000 42 4.2
0.22.0 1.0.1 10 1000 25 2.5
0.21.1 1.0.1 10 1000 25 2.5
0.21.0 1.0.1 10 1000 25 2.5
0.20.2 1.0.1 10 1000 24 2.4
0.20.1 1.0.1 10 1000 25 2.5
0.20.0 1.0.0 10 1000 28 2.8
0.19.0 0.11.1 10 1000 25 2.5
0.18.0 0.11.1 10 1000 24 2.4
0.17.2 0.11.1 10 1000 24 2.4
0.17.1 0.11.1 10 1000 25 2.5
0.17.0 0.11.1 10 1000 24 2.4
0.16.0 0.11.1 10 1000 25 2.5
0.15.0 0.11.1 10 1000 24 2.4
0.14.2 0.10.0 10 1000 24 2.4
0.13.4 0.9.0 10 1000 24 2.4
10000 Observations
1.0.1 1.0.1 10 10000 54 5.4
1.0.0 1.0.1 10 10000 55 5.5
0.23.0 1.0.1 10 10000 57 5.7
0.22.0 1.0.1 10 10000 29 2.9
0.21.1 1.0.1 10 10000 30 3.0
0.21.0 1.0.1 10 10000 28 2.8
0.20.2 1.0.1 10 10000 29 2.9
0.20.1 1.0.1 10 10000 29 2.9
0.20.0 1.0.0 10 10000 32 3.2
0.19.0 0.11.1 10 10000 29 2.9
0.18.0 0.11.1 10 10000 27 2.7
0.17.2 0.11.1 10 10000 28 2.8
0.17.1 0.11.1 10 10000 27 2.7
0.17.0 0.11.1 10 10000 28 2.8
0.16.0 0.11.1 10 10000 31 3.1
0.15.0 0.11.1 10 10000 31 3.1
0.14.2 0.10.0 10 10000 30 3.0
0.13.4 0.9.0 10 10000 29 2.9

Model Time 1 ms

Median runtime of $predict_newdata() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata() with models trained for 1 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.0.1 1.0.1 1 1 31 31
1.0.0 1.0.1 1 1 33 33
0.23.0 1.0.1 1 1 30 30
0.22.0 1.0.1 1 1 13 13
0.21.1 1.0.1 1 1 13 13
0.21.0 1.0.1 1 1 12 12
0.20.2 1.0.1 1 1 11 11
0.20.1 1.0.1 1 1 12 12
0.20.0 1.0.0 1 1 15 15
0.19.0 0.11.1 1 1 12 12
0.18.0 0.11.1 1 1 11 11
0.17.2 0.11.1 1 1 12 12
0.17.1 0.11.1 1 1 11 11
0.17.0 0.11.1 1 1 12 12
0.16.0 0.11.1 1 1 13 13
0.15.0 0.11.1 1 1 13 13
0.14.2 0.10.0 1 1 12 12
0.13.4 0.9.0 1 1 14 14
10 Observations
1.0.1 1.0.1 1 10 30 30
1.0.0 1.0.1 1 10 30 30
0.23.0 1.0.1 1 10 29 29
0.22.0 1.0.1 1 10 12 12
0.21.1 1.0.1 1 10 13 13
0.21.0 1.0.1 1 10 13 13
0.20.2 1.0.1 1 10 12 12
0.20.1 1.0.1 1 10 13 13
0.20.0 1.0.0 1 10 15 15
0.19.0 0.11.1 1 10 12 12
0.18.0 0.11.1 1 10 12 12
0.17.2 0.11.1 1 10 11 11
0.17.1 0.11.1 1 10 11 11
0.17.0 0.11.1 1 10 11 11
0.16.0 0.11.1 1 10 12 12
0.15.0 0.11.1 1 10 12 12
0.14.2 0.10.0 1 10 12 12
0.13.4 0.9.0 1 10 12 12
100 Observations
1.0.1 1.0.1 1 100 28 28
1.0.0 1.0.1 1 100 29 29
0.23.0 1.0.1 1 100 29 29
0.22.0 1.0.1 1 100 12 12
0.21.1 1.0.1 1 100 13 13
0.21.0 1.0.1 1 100 12 12
0.20.2 1.0.1 1 100 11 11
0.20.1 1.0.1 1 100 11 11
0.20.0 1.0.0 1 100 14 14
0.19.0 0.11.1 1 100 11 11
0.18.0 0.11.1 1 100 11 11
0.17.2 0.11.1 1 100 13 13
0.17.1 0.11.1 1 100 12 12
0.17.0 0.11.1 1 100 11 11
0.16.0 0.11.1 1 100 13 13
0.15.0 0.11.1 1 100 13 13
0.14.2 0.10.0 1 100 12 12
0.13.4 0.9.0 1 100 13 13
1000 Observations
1.0.1 1.0.1 1 1000 31 31
1.0.0 1.0.1 1 1000 31 31
0.23.0 1.0.1 1 1000 29 29
0.22.0 1.0.1 1 1000 13 13
0.21.1 1.0.1 1 1000 13 13
0.21.0 1.0.1 1 1000 12 12
0.20.2 1.0.1 1 1000 12 12
0.20.1 1.0.1 1 1000 12 12
0.20.0 1.0.0 1 1000 15 15
0.19.0 0.11.1 1 1000 12 12
0.18.0 0.11.1 1 1000 12 12
0.17.2 0.11.1 1 1000 12 12
0.17.1 0.11.1 1 1000 12 12
0.17.0 0.11.1 1 1000 12 12
0.16.0 0.11.1 1 1000 13 13
0.15.0 0.11.1 1 1000 13 13
0.14.2 0.10.0 1 1000 13 13
0.13.4 0.9.0 1 1000 13 13
10000 Observations
1.0.1 1.0.1 1 10000 43 43
1.0.0 1.0.1 1 10000 44 44
0.23.0 1.0.1 1 10000 44 44
0.22.0 1.0.1 1 10000 18 18
0.21.1 1.0.1 1 10000 16 16
0.21.0 1.0.1 1 10000 17 17
0.20.2 1.0.1 1 10000 16 16
0.20.1 1.0.1 1 10000 16 16
0.20.0 1.0.0 1 10000 23 23
0.19.0 0.11.1 1 10000 18 18
0.18.0 0.11.1 1 10000 19 19
0.17.2 0.11.1 1 10000 17 17
0.17.1 0.11.1 1 10000 18 18
0.17.0 0.11.1 1 10000 17 17
0.16.0 0.11.1 1 10000 17 17
0.15.0 0.11.1 1 10000 17 17
0.14.2 0.10.0 1 10000 18 18
0.13.4 0.9.0 1 10000 18 18

Predict Newdata Fast

The runtime of the $predict_newdata_fast() method is measured for different mlr3 versions. The predict step is performed for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with different amounts of observations (1, 10, 100, 1000, and 10000).

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

learner$train(task)

learner$predict_newdata_fast(task)

Model Time 1000 ms

Median runtime of $predict_newdata_fast() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata_fast() with models trained for 1000 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.1.0 1.0.1 1000 1 1002 1.0
10 Observations
1.1.0 1.0.1 1000 10 1002 1.0
100 Observations
1.1.0 1.0.1 1000 100 1002 1.0
1000 Observations
1.1.0 1.0.1 1000 1000 1002 1.0
10000 Observations
1.1.0 1.0.1 1000 10000 1002 1.0

Model Time 100 ms

Median runtime of $predict_newdata_fast() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata_fast() with models trained for 100 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.1.0 1.0.1 100 1 101 1.0
10 Observations
1.1.0 1.0.1 100 10 101 1.0
100 Observations
1.1.0 1.0.1 100 100 101 1.0
1000 Observations
1.1.0 1.0.1 100 1000 101 1.0
10000 Observations
1.1.0 1.0.1 100 10000 101 1.0

Model Time 10 ms

Median runtime of $predict_newdata_fast() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata_fast() with models trained for 10 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.1.0 1.0.1 10 1 11 1.1
10 Observations
1.1.0 1.0.1 10 10 11 1.1
100 Observations
1.1.0 1.0.1 10 100 11 1.1
1000 Observations
1.1.0 1.0.1 10 1000 11 1.1
10000 Observations
1.1.0 1.0.1 10 10000 11 1.1

Model Time 1 ms

Median runtime of $predict_newdata_fast() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime usage of $predict_newdata_fast() with models trained for 1 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models.
mlr3 Version paradox Version Model Time [ms] Task Size Median Runtime [ms] K
1 Observation
1.1.0 1.0.1 1 1 1 1.5
10 Observations
1.1.0 1.0.1 1 10 1 1.5
100 Observations
1.1.0 1.0.1 1 100 1 1.5
1000 Observations
1.1.0 1.0.1 1 1000 1 1.5
10000 Observations
1.1.0 1.0.1 1 10000 1 1.4

Resampling

The runtime and memory usage of the resample() function is measured for different mlr3 versions. The models are trained for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with 1000 and 10,000 observations. The resampling iterations (evals) are set to 1000, 100, and 10.

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

resampling = rsmp("subsampling", repeats = evals)

resample(task, learner, resampling)

Model Time 1000 ms

Median runtime of resample() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 1000 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1000 10 10 12 11 1.2 175 203
0.14.2 0.10.0 1000 10 10 12 11 1.2 174 203
0.15.0 0.11.1 1000 10 10 13 11 1.3 174 203
0.16.0 0.11.1 1000 10 10 11 11 1.1 174 203
0.17.0 0.11.1 1000 10 10 10 10 1.0 175 203
0.17.1 0.11.1 1000 10 10 10 10 1.0 175 203
0.17.2 0.11.1 1000 10 10 10 10 1.0 175 203
0.18.0 0.11.1 1000 10 10 10 10 1.0 175 203
0.19.0 0.11.1 1000 10 10 10 10 1.0 175 203
0.20.0 1.0.0 1000 10 10 10 10 1.0 167 193
0.20.1 1.0.1 1000 10 10 10 10 1.0 165 198
0.20.2 1.0.1 1000 10 10 10 10 1.0 165 197
0.21.0 1.0.1 1000 10 10 10 10 1.0 164 202
0.21.1 1.0.1 1000 10 10 10 10 1.0 164 202
0.22.0 1.0.1 1000 10 10 10 10 1.0 163 200
0.22.1 1.0.1 1000 10 10 10 10 1.0 163 200
100 Resampling Iterations
0.13.4 0.9.0 1000 100 100 110 100 1.1 222 203
0.14.2 0.10.0 1000 100 100 110 110 1.1 224 206
0.15.0 0.11.1 1000 100 100 110 110 1.1 224 205
0.16.0 0.11.1 1000 100 100 110 100 1.1 224 206
0.17.0 0.11.1 1000 100 100 100 100 1.0 247 209
0.17.1 0.11.1 1000 100 100 100 100 1.0 207 203
0.17.2 0.11.1 1000 100 100 100 100 1.0 207 203
0.18.0 0.11.1 1000 100 100 100 100 1.0 207 203
0.19.0 0.11.1 1000 100 100 100 100 1.0 207 203
0.20.0 1.0.0 1000 100 100 100 100 1.0 241 226
0.20.1 1.0.1 1000 100 100 100 100 1.0 206 200
0.20.2 1.0.1 1000 100 100 100 100 1.0 206 200
0.21.0 1.0.1 1000 100 100 100 100 1.0 182 202
0.21.1 1.0.1 1000 100 100 100 100 1.0 182 202
0.22.0 1.0.1 1000 100 100 100 100 1.0 179 206
0.22.1 1.0.1 1000 100 100 100 100 1.0 179 204

Model Time 100 ms

Median runtime of resample() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 100 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 100 10 1 1.9 3.6 1.9 175 203
0.14.2 0.10.0 100 10 1 3.4 2.9 3.4 174 203
0.15.0 0.11.1 100 10 1 3.1 2.9 3.1 174 203
0.16.0 0.11.1 100 10 1 1.9 2.7 1.9 174 203
0.17.0 0.11.1 100 10 1 1.1 1.1 1.1 175 203
0.17.1 0.11.1 100 10 1 1.1 1.1 1.1 175 203
0.17.2 0.11.1 100 10 1 1.1 1.1 1.1 175 203
0.18.0 0.11.1 100 10 1 1.1 1.1 1.1 175 203
0.19.0 0.11.1 100 10 1 1.1 1.1 1.1 175 203
0.20.0 1.0.0 100 10 1 1.1 1.1 1.1 167 193
0.20.1 1.0.1 100 10 1 1.1 1.1 1.1 165 198
0.20.2 1.0.1 100 10 1 1.1 1.1 1.1 165 197
0.21.0 1.0.1 100 10 1 1.1 1.1 1.1 164 202
0.21.1 1.0.1 100 10 1 1.1 1.1 1.1 164 202
0.22.0 1.0.1 100 10 1 1.1 1.1 1.1 163 200
0.22.1 1.0.1 100 10 1 1.1 1.1 1.1 163 200
100 Resampling Iterations
0.13.4 0.9.0 100 100 10 31 24 3.1 222 203
0.14.2 0.10.0 100 100 10 16 27 1.6 224 206
0.15.0 0.11.1 100 100 10 20 34 2.0 224 205
0.16.0 0.11.1 100 100 10 20 47 2.0 224 206
0.17.0 0.11.1 100 100 10 11 12 1.1 247 209
0.17.1 0.11.1 100 100 10 11 11 1.1 207 203
0.17.2 0.11.1 100 100 10 11 11 1.1 207 203
0.18.0 0.11.1 100 100 10 11 11 1.1 207 203
0.19.0 0.11.1 100 100 10 11 11 1.1 207 203
0.20.0 1.0.0 100 100 10 11 11 1.1 241 226
0.20.1 1.0.1 100 100 10 11 11 1.1 206 200
0.20.2 1.0.1 100 100 10 11 11 1.1 206 200
0.21.0 1.0.1 100 100 10 11 11 1.1 182 202
0.21.1 1.0.1 100 100 10 11 11 1.1 182 202
0.22.0 1.0.1 100 100 10 11 11 1.1 179 206
0.22.1 1.0.1 100 100 10 11 11 1.1 179 204
1000 Resampling Iterations
0.13.4 0.9.0 100 1000 100 150 170 1.5 343 625
0.14.2 0.10.0 100 1000 100 140 170 1.4 315 680
0.15.0 0.11.1 100 1000 100 150 180 1.5 350 613
0.16.0 0.11.1 100 1000 100 130 160 1.3 316 564
0.17.0 0.11.1 100 1000 100 110 140 1.1 342 574
0.17.1 0.11.1 100 1000 100 110 110 1.1 266 406
0.17.2 0.11.1 100 1000 100 110 110 1.1 266 405
0.18.0 0.11.1 100 1000 100 110 110 1.1 267 408
0.19.0 0.11.1 100 1000 100 110 110 1.1 266 396
0.20.0 1.0.0 100 1000 100 110 110 1.1 312 437
0.20.1 1.0.1 100 1000 100 110 110 1.1 267 385
0.20.2 1.0.1 100 1000 100 110 110 1.1 267 385
0.21.0 1.0.1 100 1000 100 110 110 1.1 225 355
0.21.1 1.0.1 100 1000 100 110 110 1.1 225 355
0.22.0 1.0.1 100 1000 100 110 110 1.1 225 381
0.22.1 1.0.1 100 1000 100 110 110 1.1 225 382

Model Time 10 ms

Median runtime of resample() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 10 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 10 10 0.1 1.2 0.91 12 175 203
0.14.2 0.10.0 10 10 0.1 1.2 0.76 12 174 203
0.15.0 0.11.1 10 10 0.1 1.8 1.4 18 174 203
0.16.0 0.11.1 10 10 0.1 1.2 1.1 12 174 203
0.17.0 0.11.1 10 10 0.1 0.19 0.21 1.9 175 203
0.17.1 0.11.1 10 10 0.1 0.19 0.21 1.9 175 203
0.17.2 0.11.1 10 10 0.1 0.19 0.21 1.9 175 203
0.18.0 0.11.1 10 10 0.1 0.19 0.21 1.9 175 203
0.19.0 0.11.1 10 10 0.1 0.19 0.21 1.9 175 203
0.20.0 1.0.0 10 10 0.1 0.22 0.25 2.2 167 193
0.20.1 1.0.1 10 10 0.1 0.19 0.21 1.9 165 198
0.20.2 1.0.1 10 10 0.1 0.19 0.21 1.9 165 197
0.21.0 1.0.1 10 10 0.1 0.18 0.20 1.8 164 202
0.21.1 1.0.1 10 10 0.1 0.18 0.21 1.8 164 202
0.22.0 1.0.1 10 10 0.1 0.19 0.20 1.9 163 200
0.22.1 1.0.1 10 10 0.1 0.18 0.20 1.8 163 200
100 Resampling Iterations
0.13.4 0.9.0 10 100 1.0 13 12 13 222 203
0.14.2 0.10.0 10 100 1.0 8.8 9.6 8.8 224 206
0.15.0 0.11.1 10 100 1.0 19 9.2 19 224 205
0.16.0 0.11.1 10 100 1.0 12 10 12 224 206
0.17.0 0.11.1 10 100 1.0 1.9 2.4 1.9 247 209
0.17.1 0.11.1 10 100 1.0 1.8 2.1 1.8 207 203
0.17.2 0.11.1 10 100 1.0 1.8 2.1 1.8 207 203
0.18.0 0.11.1 10 100 1.0 1.8 2.0 1.8 207 203
0.19.0 0.11.1 10 100 1.0 1.8 2.0 1.8 207 203
0.20.0 1.0.0 10 100 1.0 2.2 2.4 2.2 241 226
0.20.1 1.0.1 10 100 1.0 1.8 2.1 1.8 206 200
0.20.2 1.0.1 10 100 1.0 1.8 2.0 1.8 206 200
0.21.0 1.0.1 10 100 1.0 1.7 2.0 1.7 182 202
0.21.1 1.0.1 10 100 1.0 1.8 2.0 1.8 182 202
0.22.0 1.0.1 10 100 1.0 1.7 2.0 1.7 179 206
0.22.1 1.0.1 10 100 1.0 1.8 2.0 1.8 179 204
1000 Resampling Iterations
0.13.4 0.9.0 10 1000 10.0 95 170 9.5 343 625
0.14.2 0.10.0 10 1000 10.0 86 120 8.6 315 680
0.15.0 0.11.1 10 1000 10.0 140 150 14 350 613
0.16.0 0.11.1 10 1000 10.0 110 140 11 316 564
0.17.0 0.11.1 10 1000 10.0 22 54 2.2 342 574
0.17.1 0.11.1 10 1000 10.0 18 20 1.8 266 406
0.17.2 0.11.1 10 1000 10.0 18 20 1.8 266 405
0.18.0 0.11.1 10 1000 10.0 18 20 1.8 267 408
0.19.0 0.11.1 10 1000 10.0 18 20 1.8 266 396
0.20.0 1.0.0 10 1000 10.0 22 24 2.2 312 437
0.20.1 1.0.1 10 1000 10.0 18 20 1.8 267 385
0.20.2 1.0.1 10 1000 10.0 18 20 1.8 267 385
0.21.0 1.0.1 10 1000 10.0 18 19 1.8 225 355
0.21.1 1.0.1 10 1000 10.0 18 19 1.8 225 355
0.22.0 1.0.1 10 1000 10.0 18 19 1.8 225 381
0.22.1 1.0.1 10 1000 10.0 17 19 1.7 225 382

Model Time 1 ms

Median runtime of resample() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 1 ms depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1 10 0.01 0.89 1.3 89 175 203
0.14.2 0.10.0 1 10 0.01 1.3 1.3 130 174 203
0.15.0 0.11.1 1 10 0.01 1.4 1.9 140 174 203
0.16.0 0.11.1 1 10 0.01 1.3 1.3 130 174 203
0.17.0 0.11.1 1 10 0.01 0.11 0.13 11 175 203
0.17.1 0.11.1 1 10 0.01 0.10 0.13 10 175 203
0.17.2 0.11.1 1 10 0.01 0.098 0.13 9.8 175 203
0.18.0 0.11.1 1 10 0.01 0.10 0.13 10 175 203
0.19.0 0.11.1 1 10 0.01 0.11 0.12 11 175 203
0.20.0 1.0.0 1 10 0.01 0.14 0.16 14 167 193
0.20.1 1.0.1 1 10 0.01 0.10 0.13 10 165 198
0.20.2 1.0.1 1 10 0.01 0.10 0.13 10 165 197
0.21.0 1.0.1 1 10 0.01 0.093 0.12 9.3 164 202
0.21.1 1.0.1 1 10 0.01 0.087 0.12 8.7 164 202
0.22.0 1.0.1 1 10 0.01 0.085 0.11 8.5 163 200
0.22.1 1.0.1 1 10 0.01 0.084 0.11 8.4 163 200
100 Resampling Iterations
0.13.4 0.9.0 1 100 0.10 10 13 100 222 203
0.14.2 0.10.0 1 100 0.10 12 15 120 224 206
0.15.0 0.11.1 1 100 0.10 19 17 190 224 205
0.16.0 0.11.1 1 100 0.10 11 9.5 110 224 206
0.17.0 0.11.1 1 100 0.10 0.97 1.6 9.7 247 209
0.17.1 0.11.1 1 100 0.10 0.93 1.2 9.3 207 203
0.17.2 0.11.1 1 100 0.10 0.94 1.2 9.4 207 203
0.18.0 0.11.1 1 100 0.10 0.90 1.2 9.0 207 203
0.19.0 0.11.1 1 100 0.10 0.90 1.2 9.0 207 203
0.20.0 1.0.0 1 100 0.10 1.3 1.5 13 241 226
0.20.1 1.0.1 1 100 0.10 0.92 1.2 9.2 206 200
0.20.2 1.0.1 1 100 0.10 0.91 1.2 9.1 206 200
0.21.0 1.0.1 1 100 0.10 0.89 1.1 8.9 182 202
0.21.1 1.0.1 1 100 0.10 0.87 1.1 8.7 182 202
0.22.0 1.0.1 1 100 0.10 0.81 0.96 8.1 179 206
0.22.1 1.0.1 1 100 0.10 0.84 0.97 8.4 179 204
1000 Resampling Iterations
0.13.4 0.9.0 1 1000 1.00 110 160 110 343 625
0.14.2 0.10.0 1 1000 1.00 130 110 130 315 680
0.15.0 0.11.1 1 1000 1.00 180 150 180 350 613
0.16.0 0.11.1 1 1000 1.00 87 120 87 316 564
0.17.0 0.11.1 1 1000 1.00 14 47 14 342 574
0.17.1 0.11.1 1 1000 1.00 9.7 12 9.7 266 406
0.17.2 0.11.1 1 1000 1.00 9.7 12 9.7 266 405
0.18.0 0.11.1 1 1000 1.00 9.5 12 9.5 267 408
0.19.0 0.11.1 1 1000 1.00 9.5 12 9.5 266 396
0.20.0 1.0.0 1 1000 1.00 13 16 13 312 437
0.20.1 1.0.1 1 1000 1.00 9.2 11 9.2 267 385
0.20.2 1.0.1 1 1000 1.00 9.2 11 9.2 267 385
0.21.0 1.0.1 1 1000 1.00 8.8 10 8.8 225 355
0.21.1 1.0.1 1 1000 1.00 8.6 9.8 8.6 225 355
0.22.0 1.0.1 1 1000 1.00 8.4 9.9 8.4 225 381
0.22.1 1.0.1 1 1000 1.00 8.4 10 8.4 225 382

Memory

Memory usage of resample() depending on the mlr3 version and the number of resampling iterations. Error bars represent the median absolute deviation of the memory usage.

Benchmark

The runtime and memory usage of the benchmark() function is measured for different mlr3 versions. The models are trained for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with 1000 and 10,000 observations. The resampling iterations (evals) are set to 1000, 100, and 10.

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

resampling = rsmp("subsampling", repeats = instance$evals / 5)

design = benchmark_grid(task, replicate(5, learner), resampling)

benchmark(design)

Model Time 1000 ms

Median runtime of benchmark() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 1000 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1000 10 10 11 11 1.1 170 204
0.14.2 0.10.0 1000 10 10 18 16 1.8 170 204
0.15.0 0.11.1 1000 10 10 11 12 1.1 170 204
0.16.0 0.11.1 1000 10 10 12 12 1.2 170 204
0.17.0 0.11.1 1000 10 10 10 10 1.0 171 204
0.17.1 0.11.1 1000 10 10 10 10 1.0 171 204
0.17.2 0.11.1 1000 10 10 10 10 1.0 171 204
0.18.0 0.11.1 1000 10 10 10 10 1.0 171 204
0.19.0 0.11.1 1000 10 10 10 10 1.0 171 204
0.20.0 1.0.0 1000 10 10 10 10 1.0 168 194
0.20.1 1.0.1 1000 10 10 10 10 1.0 167 199
0.20.2 1.0.1 1000 10 10 10 10 1.0 167 199
0.21.0 1.0.1 1000 10 10 10 10 1.0 166 202
0.21.1 1.0.1 1000 10 10 10 10 1.0 166 202
100 Resampling Iterations
0.13.4 0.9.0 1000 100 100 130 120 1.3 225 202
0.14.2 0.10.0 1000 100 100 150 120 1.5 225 203
0.15.0 0.11.1 1000 100 100 120 120 1.2 225 214
0.16.0 0.11.1 1000 100 100 120 120 1.2 225 215
0.17.0 0.11.1 1000 100 100 100 100 1.0 224 202
0.17.1 0.11.1 1000 100 100 100 100 1.0 209 204
0.17.2 0.11.1 1000 100 100 100 100 1.0 209 204
0.18.0 0.11.1 1000 100 100 100 100 1.0 209 204
0.19.0 0.11.1 1000 100 100 100 100 1.0 209 204
0.20.0 1.0.0 1000 100 100 100 100 1.0 242 206
0.20.1 1.0.1 1000 100 100 100 100 1.0 205 200
0.20.2 1.0.1 1000 100 100 100 100 1.0 205 201
0.21.0 1.0.1 1000 100 100 100 100 1.0 183 202
0.21.1 1.0.1 1000 100 100 100 100 1.0 183 202

Model Time 100 ms

Median runtime of benchmark() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 100 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 100 10 1 3.8 3.9 3.8 170 204
0.14.2 0.10.0 100 10 1 7.4 6.5 7.4 170 204
0.15.0 0.11.1 100 10 1 4.8 3.3 4.8 170 204
0.16.0 0.11.1 100 10 1 2.0 3.3 2.0 170 204
0.17.0 0.11.1 100 10 1 1.1 1.1 1.1 171 204
0.17.1 0.11.1 100 10 1 1.1 1.1 1.1 171 204
0.17.2 0.11.1 100 10 1 1.1 1.1 1.1 171 204
0.18.0 0.11.1 100 10 1 1.1 1.1 1.1 171 204
0.19.0 0.11.1 100 10 1 1.1 1.1 1.1 171 204
0.20.0 1.0.0 100 10 1 1.1 1.1 1.1 168 194
0.20.1 1.0.1 100 10 1 1.1 1.1 1.1 167 199
0.20.2 1.0.1 100 10 1 1.1 1.1 1.1 167 199
0.21.0 1.0.1 100 10 1 1.1 1.1 1.1 166 202
0.21.1 1.0.1 100 10 1 1.1 1.1 1.1 166 202
100 Resampling Iterations
0.13.4 0.9.0 100 100 10 42 30 4.2 225 202
0.14.2 0.10.0 100 100 10 75 44 7.5 225 203
0.15.0 0.11.1 100 100 10 48 28 4.8 225 214
0.16.0 0.11.1 100 100 10 45 37 4.5 225 215
0.17.0 0.11.1 100 100 10 11 11 1.1 224 202
0.17.1 0.11.1 100 100 10 11 11 1.1 209 204
0.17.2 0.11.1 100 100 10 11 11 1.1 209 204
0.18.0 0.11.1 100 100 10 11 11 1.1 209 204
0.19.0 0.11.1 100 100 10 11 11 1.1 209 204
0.20.0 1.0.0 100 100 10 11 11 1.1 242 206
0.20.1 1.0.1 100 100 10 11 11 1.1 205 200
0.20.2 1.0.1 100 100 10 11 11 1.1 205 201
0.21.0 1.0.1 100 100 10 11 11 1.1 183 202
0.21.1 1.0.1 100 100 10 11 11 1.1 183 202
1000 Resampling Iterations
0.13.4 0.9.0 100 1000 100 230 220 2.3 322 425
0.14.2 0.10.0 100 1000 100 250 200 2.5 316 422
0.15.0 0.11.1 100 1000 100 210 250 2.1 325 457
0.16.0 0.11.1 100 1000 100 210 160 2.1 313 463
0.17.0 0.11.1 100 1000 100 110 120 1.1 293 388
0.17.1 0.11.1 100 1000 100 110 110 1.1 268 306
0.17.2 0.11.1 100 1000 100 110 110 1.1 268 306
0.18.0 0.11.1 100 1000 100 110 110 1.1 269 305
0.19.0 0.11.1 100 1000 100 110 110 1.1 269 311
0.20.0 1.0.0 100 1000 100 110 110 1.1 310 339
0.20.1 1.0.1 100 1000 100 110 110 1.1 265 300
0.20.2 1.0.1 100 1000 100 110 110 1.1 265 305
0.21.0 1.0.1 100 1000 100 110 110 1.1 222 281
0.21.1 1.0.1 100 1000 100 110 110 1.1 222 281

Model Time 10 ms

Median runtime of benchmark() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 10 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 10 10 0.1 1.7 1.2 17 170 204
0.14.2 0.10.0 10 10 0.1 2.1 1.8 21 170 204
0.15.0 0.11.1 10 10 0.1 1.9 2.4 19 170 204
0.16.0 0.11.1 10 10 0.1 1.1 1.4 11 170 204
0.17.0 0.11.1 10 10 0.1 0.20 0.21 2.0 171 204
0.17.1 0.11.1 10 10 0.1 0.19 0.21 1.9 171 204
0.17.2 0.11.1 10 10 0.1 0.19 0.21 1.9 171 204
0.18.0 0.11.1 10 10 0.1 0.19 0.21 1.9 171 204
0.19.0 0.11.1 10 10 0.1 0.19 0.21 1.9 171 204
0.20.0 1.0.0 10 10 0.1 0.23 0.25 2.3 168 194
0.20.1 1.0.1 10 10 0.1 0.19 0.21 1.9 167 199
0.20.2 1.0.1 10 10 0.1 0.19 0.21 1.9 167 199
0.21.0 1.0.1 10 10 0.1 0.19 0.21 1.9 166 202
0.21.1 1.0.1 10 10 0.1 0.19 0.21 1.9 166 202
100 Resampling Iterations
0.13.4 0.9.0 10 100 1.0 15 11 15 225 202
0.14.2 0.10.0 10 100 1.0 20 20 20 225 203
0.15.0 0.11.1 10 100 1.0 22 24 22 225 214
0.16.0 0.11.1 10 100 1.0 22 16 22 225 215
0.17.0 0.11.1 10 100 1.0 1.8 2.1 1.8 224 202
0.17.1 0.11.1 10 100 1.0 1.8 2.0 1.8 209 204
0.17.2 0.11.1 10 100 1.0 1.8 2.0 1.8 209 204
0.18.0 0.11.1 10 100 1.0 1.8 2.0 1.8 209 204
0.19.0 0.11.1 10 100 1.0 1.8 2.0 1.8 209 204
0.20.0 1.0.0 10 100 1.0 2.2 2.3 2.2 242 206
0.20.1 1.0.1 10 100 1.0 1.8 2.0 1.8 205 200
0.20.2 1.0.1 10 100 1.0 1.8 2.0 1.8 205 201
0.21.0 1.0.1 10 100 1.0 1.8 1.9 1.8 183 202
0.21.1 1.0.1 10 100 1.0 1.7 2.0 1.7 183 202
1000 Resampling Iterations
0.13.4 0.9.0 10 1000 10.0 160 170 16 322 425
0.14.2 0.10.0 10 1000 10.0 110 170 11 316 422
0.15.0 0.11.1 10 1000 10.0 140 110 14 325 457
0.16.0 0.11.1 10 1000 10.0 150 130 15 313 463
0.17.0 0.11.1 10 1000 10.0 19 28 1.9 293 388
0.17.1 0.11.1 10 1000 10.0 18 20 1.8 268 306
0.17.2 0.11.1 10 1000 10.0 18 20 1.8 268 306
0.18.0 0.11.1 10 1000 10.0 18 20 1.8 269 305
0.19.0 0.11.1 10 1000 10.0 18 20 1.8 269 311
0.20.0 1.0.0 10 1000 10.0 22 24 2.2 310 339
0.20.1 1.0.1 10 1000 10.0 18 19 1.8 265 300
0.20.2 1.0.1 10 1000 10.0 18 20 1.8 265 305
0.21.0 1.0.1 10 1000 10.0 18 19 1.8 222 281
0.21.1 1.0.1 10 1000 10.0 18 19 1.8 222 281

Model Time 1 ms

Median runtime of benchmark() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 1 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1 10 0.01 2.1 1.4 210 170 204
0.14.2 0.10.0 1 10 0.01 1.3 1.9 130 170 204
0.15.0 0.11.1 1 10 0.01 1.9 2.8 190 170 204
0.16.0 0.11.1 1 10 0.01 1.5 2.1 150 170 204
0.17.0 0.11.1 1 10 0.01 0.11 0.13 11 171 204
0.17.1 0.11.1 1 10 0.01 0.10 0.12 10 171 204
0.17.2 0.11.1 1 10 0.01 0.10 0.13 10 171 204
0.18.0 0.11.1 1 10 0.01 0.10 0.13 10 171 204
0.19.0 0.11.1 1 10 0.01 0.10 0.13 10 171 204
0.20.0 1.0.0 1 10 0.01 0.15 0.17 15 168 194
0.20.1 1.0.1 1 10 0.01 0.10 0.13 10 167 199
0.20.2 1.0.1 1 10 0.01 0.11 0.13 11 167 199
0.21.0 1.0.1 1 10 0.01 0.10 0.12 10 166 202
0.21.1 1.0.1 1 10 0.01 0.099 0.12 9.9 166 202
100 Resampling Iterations
0.13.4 0.9.0 1 100 0.10 21 23 210 225 202
0.14.2 0.10.0 1 100 0.10 17 25 170 225 203
0.15.0 0.11.1 1 100 0.10 21 14 210 225 214
0.16.0 0.11.1 1 100 0.10 15 17 150 225 215
0.17.0 0.11.1 1 100 0.10 0.93 1.2 9.3 224 202
0.17.1 0.11.1 1 100 0.10 0.89 1.1 8.9 209 204
0.17.2 0.11.1 1 100 0.10 0.89 1.1 8.9 209 204
0.18.0 0.11.1 1 100 0.10 0.89 1.2 8.9 209 204
0.19.0 0.11.1 1 100 0.10 0.90 1.2 9.0 209 204
0.20.0 1.0.0 1 100 0.10 1.3 1.6 13 242 206
0.20.1 1.0.1 1 100 0.10 0.87 1.2 8.7 205 200
0.20.2 1.0.1 1 100 0.10 0.91 1.2 9.1 205 201
0.21.0 1.0.1 1 100 0.10 0.85 1.1 8.5 183 202
0.21.1 1.0.1 1 100 0.10 0.85 1.1 8.5 183 202
1000 Resampling Iterations
0.13.4 0.9.0 1 1000 1.00 150 210 150 322 425
0.14.2 0.10.0 1 1000 1.00 130 300 130 316 422
0.15.0 0.11.1 1 1000 1.00 260 230 260 325 457
0.16.0 0.11.1 1 1000 1.00 240 160 240 313 463
0.17.0 0.11.1 1 1000 1.00 10 19 10 293 388
0.17.1 0.11.1 1 1000 1.00 8.9 11 8.9 268 306
0.17.2 0.11.1 1 1000 1.00 8.9 11 8.9 268 306
0.18.0 0.11.1 1 1000 1.00 8.9 11 8.9 269 305
0.19.0 0.11.1 1 1000 1.00 9.1 11 9.1 269 311
0.20.0 1.0.0 1 1000 1.00 14 16 14 310 339
0.20.1 1.0.1 1 1000 1.00 9.0 11 9.0 265 300
0.20.2 1.0.1 1 1000 1.00 8.9 11 8.9 265 305
0.21.0 1.0.1 1 1000 1.00 8.7 11 8.7 222 281
0.21.1 1.0.1 1 1000 1.00 8.4 10 8.4 222 281

Resampling Parallel

The runtime and memory usage of the resample() function with future::multisession parallelization is measured for different mlr3 versions. The parallelization is conducted on 10 cores. The models are trained for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with 1000 and 10,000 observations. The resampling iterations (evals) are set to 1000, 100, and 10.

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

resampling = rsmp("subsampling", repeats = evals)

future::plan("multisession", workers = 10)
options("mlr3.exec_chunk_size" = evals / 10)

resample(task, learner, resampling)

Model Time 1000 ms

Median runtime of resample() on 10 cores with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 1000 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A K values with a red background indicates that the runtime is 3 times larger than the total training time of the models. A red median runtime indicates that the parallelized version took longer the the sequential run. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1000 10 10 2.9 11 2.7 2.9 1,147 1,254
0.14.2 0.10.0 1000 10 10 2.9 18 2.7 2.9 1,157 1,260
0.15.0 0.11.1 1000 10 10 2.9 11 2.7 2.9 1,157 1,260
0.16.0 0.11.1 1000 10 10 2.9 12 2.7 2.9 1,157 1,260
0.17.0 0.11.1 1000 10 10 2.9 10 2.8 2.9 1,147 1,254
0.17.1 0.11.1 1000 10 10 2.9 10 2.7 2.9 1,147 1,249
0.17.2 0.11.1 1000 10 10 2.9 10 2.8 2.9 1,147 1,249
0.18.0 0.11.1 1000 10 10 2.9 10 2.8 2.9 1,147 1,249
0.19.0 0.11.1 1000 10 10 2.9 10 2.8 2.9 1,147 1,249
0.20.0 1.0.0 1000 10 10 2.8 10 2.9 2.8 1,137 1,219
0.20.1 1.0.1 1000 10 10 2.8 10 2.8 2.8 1,137 1,219
0.20.2 1.0.1 1000 10 10 2.8 10 2.9 2.8 1,137 1,229
0.21.0 1.0.1 1000 10 10 2.8 10 2.8 2.8 1,137 1,229
0.21.1 1.0.1 1000 10 10 2.8 10 2.8 2.8 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 1000 100 100 12 130 12 1.2 1,249 1,638
0.14.2 0.10.0 1000 100 100 12 150 12 1.2 1,249 1,638
0.15.0 0.11.1 1000 100 100 12 120 12 1.2 1,249 1,633
0.16.0 0.11.1 1000 100 100 12 120 12 1.2 1,249 1,638
0.17.0 0.11.1 1000 100 100 12 100 12 1.2 1,249 1,633
0.17.1 0.11.1 1000 100 100 12 100 12 1.2 1,229 1,638
0.17.2 0.11.1 1000 100 100 12 100 12 1.2 1,229 1,638
0.18.0 0.11.1 1000 100 100 12 100 12 1.2 1,229 1,638
0.19.0 0.11.1 1000 100 100 12 100 12 1.2 1,229 1,638
0.20.0 1.0.0 1000 100 100 12 100 12 1.2 1,229 1,649
0.20.1 1.0.1 1000 100 100 12 100 12 1.2 1,219 1,659
0.20.2 1.0.1 1000 100 100 12 100 12 1.2 1,219 1,659
0.21.0 1.0.1 1000 100 100 12 100 12 1.2 1,167 1,669
0.21.1 1.0.1 1000 100 100 12 100 12 1.2 1,167 1,669

Model Time 100 ms

Median runtime of resample() on 10 cores with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 100 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A K values with a red background indicates that the runtime is 3 times larger than the total training time of the models. A red median runtime indicates that the parallelized version took longer the the sequential run. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 100 10 1 2.0 3.8 2.0 20 1,147 1,254
0.14.2 0.10.0 100 10 1 2.0 7.4 2.0 20 1,157 1,260
0.15.0 0.11.1 100 10 1 2.0 4.8 2.0 20 1,157 1,260
0.16.0 0.11.1 100 10 1 2.0 2.0 2.0 20 1,157 1,260
0.17.0 0.11.1 100 10 1 2.0 1.1 2.0 20 1,147 1,254
0.17.1 0.11.1 100 10 1 2.0 1.1 2.0 20 1,147 1,249
0.17.2 0.11.1 100 10 1 2.0 1.1 2.0 20 1,147 1,249
0.18.0 0.11.1 100 10 1 2.0 1.1 2.0 20 1,147 1,249
0.19.0 0.11.1 100 10 1 2.0 1.1 2.0 20 1,147 1,249
0.20.0 1.0.0 100 10 1 2.0 1.1 2.0 20 1,137 1,219
0.20.1 1.0.1 100 10 1 2.0 1.1 2.0 20 1,137 1,219
0.20.2 1.0.1 100 10 1 2.0 1.1 2.0 20 1,137 1,229
0.21.0 1.0.1 100 10 1 2.0 1.1 1.9 20 1,137 1,229
0.21.1 1.0.1 100 10 1 2.0 1.1 1.9 20 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 100 100 10 3.0 42 12 3.0 1,249 1,638
0.14.2 0.10.0 100 100 10 3.0 75 11 3.0 1,249 1,638
0.15.0 0.11.1 100 100 10 3.0 48 8.9 3.0 1,249 1,633
0.16.0 0.11.1 100 100 10 3.0 45 11 3.0 1,249 1,638
0.17.0 0.11.1 100 100 10 3.0 11 8.9 3.0 1,249 1,633
0.17.1 0.11.1 100 100 10 2.9 11 2.7 2.9 1,229 1,638
0.17.2 0.11.1 100 100 10 2.9 11 2.7 2.9 1,229 1,638
0.18.0 0.11.1 100 100 10 2.9 11 2.7 2.9 1,229 1,638
0.19.0 0.11.1 100 100 10 2.9 11 2.6 2.9 1,229 1,638
0.20.0 1.0.0 100 100 10 2.9 11 2.7 2.9 1,229 1,649
0.20.1 1.0.1 100 100 10 2.9 11 2.7 2.9 1,219 1,659
0.20.2 1.0.1 100 100 10 2.9 11 2.7 2.9 1,219 1,659
0.21.0 1.0.1 100 100 10 2.9 11 2.8 2.9 1,167 1,669
0.21.1 1.0.1 100 100 10 2.9 11 2.8 2.9 1,167 1,669
1000 Resampling Iterations
0.13.4 0.9.0 100 1000 100 18 230 260 1.8 1,751 2,330
0.14.2 0.10.0 100 1000 100 18 250 300 1.8 1,751 2,314
0.15.0 0.11.1 100 1000 100 18 210 210 1.8 1,741 2,355
0.16.0 0.11.1 100 1000 100 19 210 270 1.9 1,741 2,355
0.17.0 0.11.1 100 1000 100 17 110 210 1.7 1,751 2,324
0.17.1 0.11.1 100 1000 100 13 110 13 1.3 1,567 2,017
0.17.2 0.11.1 100 1000 100 13 110 13 1.3 1,567 2,017
0.18.0 0.11.1 100 1000 100 13 110 13 1.3 1,587 2,038
0.19.0 0.11.1 100 1000 100 13 110 13 1.3 1,567 2,017
0.20.0 1.0.0 100 1000 100 13 110 14 1.3 1,812 2,099
0.20.1 1.0.1 100 1000 100 13 110 13 1.3 1,577 2,017
0.20.2 1.0.1 100 1000 100 13 110 13 1.3 1,567 2,017
0.21.0 1.0.1 100 1000 100 13 110 13 1.3 1,362 1,956
0.21.1 1.0.1 100 1000 100 13 110 13 1.3 1,362 2,017

Model Time 10 ms

Median runtime of resample() on 10 cores with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 10 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A K values with a red background indicates that the runtime is 3 times larger than the total training time of the models. A red median runtime indicates that the parallelized version took longer the the sequential run. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 10 10 0.1 1.9 1.7 1.9 190 1,147 1,254
0.14.2 0.10.0 10 10 0.1 2.0 2.1 1.9 200 1,157 1,260
0.15.0 0.11.1 10 10 0.1 2.0 1.9 1.9 200 1,157 1,260
0.16.0 0.11.1 10 10 0.1 2.0 1.1 1.9 200 1,157 1,260
0.17.0 0.11.1 10 10 0.1 1.9 0.20 1.9 190 1,147 1,254
0.17.1 0.11.1 10 10 0.1 1.9 0.19 1.9 190 1,147 1,249
0.17.2 0.11.1 10 10 0.1 1.9 0.19 2.0 190 1,147 1,249
0.18.0 0.11.1 10 10 0.1 1.9 0.19 1.9 190 1,147 1,249
0.19.0 0.11.1 10 10 0.1 1.9 0.19 1.9 190 1,147 1,249
0.20.0 1.0.0 10 10 0.1 1.9 0.23 1.9 190 1,137 1,219
0.20.1 1.0.1 10 10 0.1 1.9 0.19 1.9 190 1,137 1,219
0.20.2 1.0.1 10 10 0.1 1.9 0.19 1.9 190 1,137 1,229
0.21.0 1.0.1 10 10 0.1 1.9 0.19 1.9 190 1,137 1,229
0.21.1 1.0.1 10 10 0.1 1.9 0.19 1.9 190 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 10 100 1.0 2.2 15 2.4 22 1,249 1,638
0.14.2 0.10.0 10 100 1.0 2.2 20 2.4 22 1,249 1,638
0.15.0 0.11.1 10 100 1.0 2.2 22 2.9 22 1,249 1,633
0.16.0 0.11.1 10 100 1.0 2.2 22 2.5 22 1,249 1,638
0.17.0 0.11.1 10 100 1.0 2.1 1.8 2.3 21 1,249 1,633
0.17.1 0.11.1 10 100 1.0 2.0 1.8 1.8 20 1,229 1,638
0.17.2 0.11.1 10 100 1.0 2.0 1.8 1.8 20 1,229 1,638
0.18.0 0.11.1 10 100 1.0 2.0 1.8 1.8 20 1,229 1,638
0.19.0 0.11.1 10 100 1.0 2.0 1.8 1.8 20 1,229 1,638
0.20.0 1.0.0 10 100 1.0 2.1 2.2 1.8 21 1,229 1,649
0.20.1 1.0.1 10 100 1.0 2.1 1.8 1.8 21 1,219 1,659
0.20.2 1.0.1 10 100 1.0 2.1 1.8 1.8 21 1,219 1,659
0.21.0 1.0.1 10 100 1.0 2.1 1.8 1.9 21 1,167 1,669
0.21.1 1.0.1 10 100 1.0 2.1 1.7 1.9 21 1,167 1,669
1000 Resampling Iterations
0.13.4 0.9.0 10 1000 10.0 9.3 160 160 9.3 1,751 2,330
0.14.2 0.10.0 10 1000 10.0 10 110 220 10 1,751 2,314
0.15.0 0.11.1 10 1000 10.0 12 140 160 12 1,741 2,355
0.16.0 0.11.1 10 1000 10.0 11 150 200 11 1,741 2,355
0.17.0 0.11.1 10 1000 10.0 7.8 19 200 7.8 1,751 2,324
0.17.1 0.11.1 10 1000 10.0 3.9 18 4.3 3.9 1,567 2,017
0.17.2 0.11.1 10 1000 10.0 3.9 18 4.2 3.9 1,567 2,017
0.18.0 0.11.1 10 1000 10.0 3.9 18 4.2 3.9 1,587 2,038
0.19.0 0.11.1 10 1000 10.0 3.9 18 4.2 3.9 1,567 2,017
0.20.0 1.0.0 10 1000 10.0 4.2 22 4.6 4.2 1,812 2,099
0.20.1 1.0.1 10 1000 10.0 3.9 18 4.2 3.9 1,577 2,017
0.20.2 1.0.1 10 1000 10.0 3.9 18 4.3 3.9 1,567 2,017
0.21.0 1.0.1 10 1000 10.0 3.9 18 4.2 3.9 1,362 1,956
0.21.1 1.0.1 10 1000 10.0 3.9 18 4.2 3.9 1,362 2,017

Model Time 1 ms

Median runtime of resample() on 10 cores with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of resample() with models trained for 1 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A K values with a red background indicates that the runtime is 3 times larger than the total training time of the models. A red median runtime indicates that the parallelized version took longer the the sequential run. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1 10 0.01 1.9 2.1 1.9 1,900 1,147 1,254
0.14.2 0.10.0 1 10 0.01 1.9 1.3 1.9 1,900 1,157 1,260
0.15.0 0.11.1 1 10 0.01 2.0 1.9 1.9 2,000 1,157 1,260
0.16.0 0.11.1 1 10 0.01 1.9 1.5 1.9 1,900 1,157 1,260
0.17.0 0.11.1 1 10 0.01 1.9 0.11 1.9 1,900 1,147 1,254
0.17.1 0.11.1 1 10 0.01 1.9 0.10 1.9 1,900 1,147 1,249
0.17.2 0.11.1 1 10 0.01 1.9 0.10 1.9 1,900 1,147 1,249
0.18.0 0.11.1 1 10 0.01 1.9 0.10 1.9 1,900 1,147 1,249
0.19.0 0.11.1 1 10 0.01 1.9 0.10 1.9 1,900 1,147 1,249
0.20.0 1.0.0 1 10 0.01 1.9 0.15 1.9 1,900 1,137 1,219
0.20.1 1.0.1 1 10 0.01 1.9 0.10 1.9 1,900 1,137 1,219
0.20.2 1.0.1 1 10 0.01 1.9 0.11 1.9 1,900 1,137 1,229
0.21.0 1.0.1 1 10 0.01 1.9 0.10 1.9 1,900 1,137 1,229
0.21.1 1.0.1 1 10 0.01 1.9 0.099 1.9 1,900 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 1 100 0.10 2.0 21 2.2 200 1,249 1,638
0.14.2 0.10.0 1 100 0.10 2.1 17 2.2 210 1,249 1,638
0.15.0 0.11.1 1 100 0.10 2.1 21 2.2 210 1,249 1,633
0.16.0 0.11.1 1 100 0.10 2.1 15 2.2 210 1,249 1,638
0.17.0 0.11.1 1 100 0.10 2.0 0.93 2.1 200 1,249 1,633
0.17.1 0.11.1 1 100 0.10 1.9 0.89 1.7 190 1,229 1,638
0.17.2 0.11.1 1 100 0.10 1.9 0.89 1.7 190 1,229 1,638
0.18.0 0.11.1 1 100 0.10 1.9 0.89 1.7 190 1,229 1,638
0.19.0 0.11.1 1 100 0.10 1.9 0.90 1.7 190 1,229 1,638
0.20.0 1.0.0 1 100 0.10 2.0 1.3 1.7 200 1,229 1,649
0.20.1 1.0.1 1 100 0.10 2.0 0.87 1.7 200 1,219 1,659
0.20.2 1.0.1 1 100 0.10 2.0 0.91 1.7 200 1,219 1,659
0.21.0 1.0.1 1 100 0.10 2.0 0.85 1.9 200 1,167 1,669
0.21.1 1.0.1 1 100 0.10 2.0 0.85 1.8 200 1,167 1,669
1000 Resampling Iterations
0.13.4 0.9.0 1 1000 1.00 8.8 150 170 88 1,751 2,330
0.14.2 0.10.0 1 1000 1.00 15 130 130 150 1,751 2,314
0.15.0 0.11.1 1 1000 1.00 11 260 140 110 1,741 2,355
0.16.0 0.11.1 1 1000 1.00 18 240 140 180 1,741 2,355
0.17.0 0.11.1 1 1000 1.00 6.9 10 140 69 1,751 2,324
0.17.1 0.11.1 1 1000 1.00 2.9 8.9 3.3 29 1,567 2,017
0.17.2 0.11.1 1 1000 1.00 2.9 8.9 3.2 29 1,567 2,017
0.18.0 0.11.1 1 1000 1.00 2.9 8.9 3.2 29 1,587 2,038
0.19.0 0.11.1 1 1000 1.00 3.0 9.1 3.2 30 1,567 2,017
0.20.0 1.0.0 1 1000 1.00 3.3 14 3.7 33 1,812 2,099
0.20.1 1.0.1 1 1000 1.00 2.9 9.0 3.3 29 1,577 2,017
0.20.2 1.0.1 1 1000 1.00 3.0 8.9 3.4 30 1,567 2,017
0.21.0 1.0.1 1 1000 1.00 2.9 8.7 3.3 29 1,362 1,956
0.21.1 1.0.1 1 1000 1.00 2.9 8.4 3.3 29 1,362 2,017

Memory

Memory usage of resample() on 10 cores depending on the mlr3 version. Error bars represent the median absolute deviation of the memory usage.

Benchmark Parallel

The runtime and memory usage of the benchmark() function with future::multisession parallelization is measured for different mlr3 versions. The parallelization is conducted on 10 cores. The models are trained for different amounts of time (1 ms, 10 ms, 100 ms, and 1000 ms) on the spam dataset with 1000 and 10,000 observations. The resampling iterations (evals) are set to 1000, 100, and 10.

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = model_time / 2,
  sleep_predict = model_time / 2)

resampling = rsmp("subsampling", repeats = instance$evals / 5)

design = benchmark_grid(task, replicate(5, learner), resampling)

future::plan("multisession", workers = 10)
options("mlr3.exec_chunk_size" = evals / 10)

benchmark(design)

Model Time 1000 ms

Median runtime of benchmark() with models trained for 1000 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 1000 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1000 10 10 2.4 11 2.4 2.4 1,147 1,249
0.14.2 0.10.0 1000 10 10 2.4 18 2.4 2.4 1,147 1,249
0.15.0 0.11.1 1000 10 10 2.4 11 2.4 2.4 1,152 1,249
0.16.0 0.11.1 1000 10 10 2.4 12 2.4 2.4 1,147 1,249
0.17.0 0.11.1 1000 10 10 2.4 10 2.4 2.4 1,147 1,244
0.17.1 0.11.1 1000 10 10 2.4 10 2.4 2.4 1,147 1,239
0.17.2 0.11.1 1000 10 10 2.4 10 2.4 2.4 1,147 1,239
0.18.0 0.11.1 1000 10 10 2.4 10 2.4 2.4 1,147 1,239
0.19.0 0.11.1 1000 10 10 2.4 10 2.4 2.4 1,152 1,249
0.20.0 1.0.0 1000 10 10 2.4 10 2.4 2.4 1,137 1,219
0.20.1 1.0.1 1000 10 10 2.4 10 2.4 2.4 1,137 1,219
0.20.2 1.0.1 1000 10 10 2.4 10 2.4 2.4 1,137 1,229
0.21.0 1.0.1 1000 10 10 2.4 10 2.4 2.4 1,137 1,229
0.21.1 1.0.1 1000 10 10 2.4 10 2.4 2.4 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 1000 100 100 12 130 12 1.2 1,229 1,618
0.14.2 0.10.0 1000 100 100 12 150 12 1.2 1,229 1,608
0.15.0 0.11.1 1000 100 100 12 120 12 1.2 1,229 1,613
0.16.0 0.11.1 1000 100 100 12 120 12 1.2 1,229 1,608
0.17.0 0.11.1 1000 100 100 11 100 12 1.1 1,229 1,597
0.17.1 0.11.1 1000 100 100 11 100 12 1.1 1,229 1,608
0.17.2 0.11.1 1000 100 100 11 100 12 1.1 1,229 1,613
0.18.0 0.11.1 1000 100 100 11 100 12 1.1 1,229 1,608
0.19.0 0.11.1 1000 100 100 11 100 12 1.1 1,229 1,608
0.20.0 1.0.0 1000 100 100 12 100 12 1.2 1,229 1,618
0.20.1 1.0.1 1000 100 100 11 100 12 1.1 1,219 1,654
0.20.2 1.0.1 1000 100 100 11 100 12 1.1 1,219 1,659
0.21.0 1.0.1 1000 100 100 11 100 11 1.1 1,167 1,649
0.21.1 1.0.1 1000 100 100 11 100 11 1.1 1,167 1,649

Model Time 100 ms

Median runtime of benchmark() with models trained for 100 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 100 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 100 10 1 1.5 3.8 1.6 15 1,147 1,249
0.14.2 0.10.0 100 10 1 1.6 7.4 1.6 16 1,147 1,249
0.15.0 0.11.1 100 10 1 1.6 4.8 1.6 16 1,152 1,249
0.16.0 0.11.1 100 10 1 1.6 2.0 1.6 16 1,147 1,249
0.17.0 0.11.1 100 10 1 1.6 1.1 1.6 16 1,147 1,244
0.17.1 0.11.1 100 10 1 1.6 1.1 1.6 16 1,147 1,239
0.17.2 0.11.1 100 10 1 1.6 1.1 1.6 16 1,147 1,239
0.18.0 0.11.1 100 10 1 1.6 1.1 1.6 16 1,147 1,239
0.19.0 0.11.1 100 10 1 1.6 1.1 1.6 16 1,152 1,249
0.20.0 1.0.0 100 10 1 1.6 1.1 1.6 16 1,137 1,219
0.20.1 1.0.1 100 10 1 1.6 1.1 1.6 16 1,137 1,219
0.20.2 1.0.1 100 10 1 1.6 1.1 1.5 16 1,137 1,229
0.21.0 1.0.1 100 10 1 1.6 1.1 1.6 16 1,137 1,229
0.21.1 1.0.1 100 10 1 1.6 1.1 1.5 16 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 100 100 10 2.5 42 11 2.5 1,229 1,618
0.14.2 0.10.0 100 100 10 2.5 75 12 2.5 1,229 1,608
0.15.0 0.11.1 100 100 10 2.5 48 21 2.5 1,229 1,613
0.16.0 0.11.1 100 100 10 2.5 45 9.8 2.5 1,229 1,608
0.17.0 0.11.1 100 100 10 2.5 11 9.2 2.5 1,229 1,597
0.17.1 0.11.1 100 100 10 2.4 11 2.5 2.4 1,229 1,608
0.17.2 0.11.1 100 100 10 2.4 11 2.5 2.4 1,229 1,613
0.18.0 0.11.1 100 100 10 2.4 11 2.5 2.4 1,229 1,608
0.19.0 0.11.1 100 100 10 2.4 11 2.5 2.4 1,229 1,608
0.20.0 1.0.0 100 100 10 2.5 11 2.5 2.5 1,229 1,618
0.20.1 1.0.1 100 100 10 2.4 11 2.5 2.4 1,219 1,654
0.20.2 1.0.1 100 100 10 2.4 11 2.5 2.4 1,219 1,659
0.21.0 1.0.1 100 100 10 2.4 11 2.4 2.4 1,167 1,649
0.21.1 1.0.1 100 100 10 2.4 11 2.4 2.4 1,167 1,649
1000 Resampling Iterations
0.13.4 0.9.0 100 1000 100 16 230 220 1.6 1,649 2,104
0.14.2 0.10.0 100 1000 100 16 250 280 1.6 1,638 2,145
0.15.0 0.11.1 100 1000 100 16 210 280 1.6 1,638 2,089
0.16.0 0.11.1 100 1000 100 15 210 150 1.5 1,628 2,130
0.17.0 0.11.1 100 1000 100 13 110 200 1.3 1,649 2,068
0.17.1 0.11.1 100 1000 100 12 110 12 1.2 1,536 1,884
0.17.2 0.11.1 100 1000 100 12 110 12 1.2 1,536 1,884
0.18.0 0.11.1 100 1000 100 12 110 12 1.2 1,546 1,884
0.19.0 0.11.1 100 1000 100 12 110 12 1.2 1,536 1,884
0.20.0 1.0.0 100 1000 100 13 110 13 1.3 1,772 2,007
0.20.1 1.0.1 100 1000 100 12 110 12 1.2 1,536 1,874
0.20.2 1.0.1 100 1000 100 12 110 12 1.2 1,526 1,874
0.21.0 1.0.1 100 1000 100 12 110 12 1.2 1,321 1,761
0.21.1 1.0.1 100 1000 100 12 110 12 1.2 1,321 1,761

Model Time 10 ms

Median runtime of benchmark() with models trained for 10 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 10 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 10 10 0.1 1.5 1.7 1.5 150 1,147 1,249
0.14.2 0.10.0 10 10 0.1 1.5 2.1 1.5 150 1,147 1,249
0.15.0 0.11.1 10 10 0.1 1.5 1.9 1.5 150 1,152 1,249
0.16.0 0.11.1 10 10 0.1 1.5 1.1 1.5 150 1,147 1,249
0.17.0 0.11.1 10 10 0.1 1.5 0.20 1.5 150 1,147 1,244
0.17.1 0.11.1 10 10 0.1 1.5 0.19 1.5 150 1,147 1,239
0.17.2 0.11.1 10 10 0.1 1.5 0.19 1.5 150 1,147 1,239
0.18.0 0.11.1 10 10 0.1 1.5 0.19 1.5 150 1,147 1,239
0.19.0 0.11.1 10 10 0.1 1.5 0.19 1.5 150 1,152 1,249
0.20.0 1.0.0 10 10 0.1 1.5 0.23 1.5 150 1,137 1,219
0.20.1 1.0.1 10 10 0.1 1.5 0.19 1.5 150 1,137 1,219
0.20.2 1.0.1 10 10 0.1 1.5 0.19 1.5 150 1,137 1,229
0.21.0 1.0.1 10 10 0.1 1.5 0.19 1.5 150 1,137 1,229
0.21.1 1.0.1 10 10 0.1 1.5 0.19 1.5 150 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 10 100 1.0 1.6 15 7.2 16 1,229 1,618
0.14.2 0.10.0 10 100 1.0 1.7 20 5.8 17 1,229 1,608
0.15.0 0.11.1 10 100 1.0 1.6 22 3.3 16 1,229 1,613
0.16.0 0.11.1 10 100 1.0 1.6 22 5.9 16 1,229 1,608
0.17.0 0.11.1 10 100 1.0 1.6 1.8 1.8 16 1,229 1,597
0.17.1 0.11.1 10 100 1.0 1.5 1.8 1.6 15 1,229 1,608
0.17.2 0.11.1 10 100 1.0 1.5 1.8 1.6 15 1,229 1,613
0.18.0 0.11.1 10 100 1.0 1.5 1.8 1.6 15 1,229 1,608
0.19.0 0.11.1 10 100 1.0 1.5 1.8 1.6 15 1,229 1,608
0.20.0 1.0.0 10 100 1.0 1.6 2.2 1.7 16 1,229 1,618
0.20.1 1.0.1 10 100 1.0 1.5 1.8 1.6 15 1,219 1,654
0.20.2 1.0.1 10 100 1.0 1.5 1.8 1.6 15 1,219 1,659
0.21.0 1.0.1 10 100 1.0 1.5 1.8 1.6 15 1,167 1,649
0.21.1 1.0.1 10 100 1.0 1.5 1.7 1.5 15 1,167 1,649
1000 Resampling Iterations
0.13.4 0.9.0 10 1000 10.0 9.4 160 130 9.4 1,649 2,104
0.14.2 0.10.0 10 1000 10.0 11 110 110 11 1,638 2,145
0.15.0 0.11.1 10 1000 10.0 14 140 110 14 1,638 2,089
0.16.0 0.11.1 10 1000 10.0 12 150 120 12 1,628 2,130
0.17.0 0.11.1 10 1000 10.0 4.4 19 91 4.4 1,649 2,068
0.17.1 0.11.1 10 1000 10.0 3.3 18 3.4 3.3 1,536 1,884
0.17.2 0.11.1 10 1000 10.0 3.3 18 3.3 3.3 1,536 1,884
0.18.0 0.11.1 10 1000 10.0 3.3 18 3.2 3.3 1,546 1,884
0.19.0 0.11.1 10 1000 10.0 3.3 18 3.2 3.3 1,536 1,884
0.20.0 1.0.0 10 1000 10.0 3.6 22 3.6 3.6 1,772 2,007
0.20.1 1.0.1 10 1000 10.0 3.3 18 3.3 3.3 1,536 1,874
0.20.2 1.0.1 10 1000 10.0 3.3 18 3.3 3.3 1,526 1,874
0.21.0 1.0.1 10 1000 10.0 3.3 18 3.3 3.3 1,321 1,761
0.21.1 1.0.1 10 1000 10.0 3.3 18 3.3 3.3 1,321 1,761

Model Time 1 ms

Median runtime of benchmark() with models trained for 1 ms depending on the mlr3 version. The dashed line indicates the total training time of the models. Error bars represent the median absolute deviation of the runtime.
Runtime and memory usage of benchmark() with models trained for 1 depending on the mlr3 version. The K factor shows how much longer the runtime is than the model training. A red background indicates that the runtime is 3 times larger than the total training time of the models. The table includes runtime and memory usage for tasks of size 1000 and 10,000.
mlr3 Version paradox Version Model Time [ms] Resampling Iterations Total Model Time [s] Median Runtime [s] Median Runtime Sequential [s] Median Runtime 10,000 [s] K Median Memory [MB] Median Memory 10,000 [s]
10 Resampling Iterations
0.13.4 0.9.0 1 10 0.01 1.5 2.1 1.5 1,500 1,147 1,249
0.14.2 0.10.0 1 10 0.01 1.5 1.3 1.5 1,500 1,147 1,249
0.15.0 0.11.1 1 10 0.01 1.5 1.9 1.5 1,500 1,152 1,249
0.16.0 0.11.1 1 10 0.01 1.5 1.5 1.5 1,500 1,147 1,249
0.17.0 0.11.1 1 10 0.01 1.5 0.11 1.5 1,500 1,147 1,244
0.17.1 0.11.1 1 10 0.01 1.4 0.10 1.5 1,400 1,147 1,239
0.17.2 0.11.1 1 10 0.01 1.5 0.10 1.5 1,500 1,147 1,239
0.18.0 0.11.1 1 10 0.01 1.5 0.10 1.5 1,500 1,147 1,239
0.19.0 0.11.1 1 10 0.01 1.5 0.10 1.5 1,500 1,152 1,249
0.20.0 1.0.0 1 10 0.01 1.5 0.15 1.5 1,500 1,137 1,219
0.20.1 1.0.1 1 10 0.01 1.5 0.10 1.5 1,500 1,137 1,219
0.20.2 1.0.1 1 10 0.01 1.5 0.11 1.5 1,500 1,137 1,229
0.21.0 1.0.1 1 10 0.01 1.5 0.10 1.5 1,500 1,137 1,229
0.21.1 1.0.1 1 10 0.01 1.5 0.099 1.5 1,500 1,137 1,229
100 Resampling Iterations
0.13.4 0.9.0 1 100 0.10 1.6 21 1.7 160 1,229 1,618
0.14.2 0.10.0 1 100 0.10 1.6 17 1.7 160 1,229 1,608
0.15.0 0.11.1 1 100 0.10 1.6 21 1.7 160 1,229 1,613
0.16.0 0.11.1 1 100 0.10 1.6 15 1.7 160 1,229 1,608
0.17.0 0.11.1 1 100 0.10 1.5 0.93 1.6 150 1,229 1,597
0.17.1 0.11.1 1 100 0.10 1.5 0.89 1.5 150 1,229 1,608
0.17.2 0.11.1 1 100 0.10 1.5 0.89 1.5 150 1,229 1,613
0.18.0 0.11.1 1 100 0.10 1.4 0.89 1.5 140 1,229 1,608
0.19.0 0.11.1 1 100 0.10 1.4 0.90 1.5 140 1,229 1,608
0.20.0 1.0.0 1 100 0.10 1.5 1.3 1.6 150 1,229 1,618
0.20.1 1.0.1 1 100 0.10 1.4 0.87 1.5 140 1,219 1,654
0.20.2 1.0.1 1 100 0.10 1.4 0.91 1.5 140 1,219 1,659
0.21.0 1.0.1 1 100 0.10 1.4 0.85 1.5 140 1,167 1,649
0.21.1 1.0.1 1 100 0.10 1.4 0.85 1.5 140 1,167 1,649
1000 Resampling Iterations
0.13.4 0.9.0 1 1000 1.00 9.4 150 130 94 1,649 2,104
0.14.2 0.10.0 1 1000 1.00 7.4 130 140 74 1,638 2,145
0.15.0 0.11.1 1 1000 1.00 6.8 260 130 68 1,638 2,089
0.16.0 0.11.1 1 1000 1.00 10 240 140 100 1,628 2,130
0.17.0 0.11.1 1 1000 1.00 3.4 10 58 34 1,649 2,068
0.17.1 0.11.1 1 1000 1.00 2.4 8.9 2.2 24 1,536 1,884
0.17.2 0.11.1 1 1000 1.00 2.4 8.9 2.3 24 1,536 1,884
0.18.0 0.11.1 1 1000 1.00 2.4 8.9 2.3 24 1,546 1,884
0.19.0 0.11.1 1 1000 1.00 2.4 9.1 2.3 24 1,536 1,884
0.20.0 1.0.0 1 1000 1.00 2.7 14 2.7 27 1,772 2,007
0.20.1 1.0.1 1 1000 1.00 2.4 9.0 2.3 24 1,536 1,874
0.20.2 1.0.1 1 1000 1.00 2.4 8.9 2.3 24 1,526 1,874
0.21.0 1.0.1 1 1000 1.00 2.4 8.7 2.3 24 1,321 1,761
0.21.1 1.0.1 1 1000 1.00 2.3 8.4 2.4 23 1,321 1,761

Memory

Memory usage of benchmark() on 10 cores depending on the mlr3 version. Error bars represent the median absolute deviation of the memory usage.

Encapsulation

The runtime and memory usage of the $train() method is measured for different encapsulation methods and mlr3 versions.

task = tsk("spam")

learner = lrn("classif.sleep",
  sleep_train = 1 / 1000 / 2,
  sleep_predict = 1 / 1000 / 2)

learner$encapsulate(method, fallback = lrn("classif.featureless"))

learner$train(task)

Runtime

Median runtime of $train() for different encapsulation methods depending on the mlr3 version.

Memory

Memory usage of $train() for different encapsulation methods depending on the mlr3 version.

Object Size

The size of different mlr3 objects is compared for different mlr3 versions. The size is measured in memory and after calling serialize() and unserialize().

Task

task = tsk("spam_1000")

Memory usage of a Task object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a Task object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.52 4.6 0.59 4.7
0.14.2 0.10.0 10 0.52 4.6 0.59 4.7
0.15.0 0.11.1 10 0.52 4.6 0.59 4.7
0.16.0 0.11.1 10 0.52 4.6 0.59 4.7
0.17.0 0.11.1 10 0.52 4.6 0.59 4.7
0.17.1 0.11.1 10 0.52 4.6 0.59 4.7
0.17.2 0.11.1 10 0.52 4.6 0.59 4.7
0.18.0 0.11.1 10 0.52 4.6 0.59 4.7
0.19.0 0.11.1 10 0.52 4.6 0.59 4.7
0.20.0 1.0.0 10 0.53 4.6 0.60 4.7
0.20.1 1.0.1 10 0.53 4.6 0.60 4.7
0.20.2 1.0.1 10 0.53 4.6 0.60 4.7
0.21.0 1.0.1 10 0.53 4.6 0.60 4.7
0.21.1 1.0.1 10 0.53 4.6 0.60 4.7
100 Resampling Iterations
0.13.4 0.9.0 100 0.52 4.6 0.59 4.7
0.14.2 0.10.0 100 0.52 4.6 0.59 4.7
0.15.0 0.11.1 100 0.52 4.6 0.59 4.7
0.16.0 0.11.1 100 0.52 4.6 0.59 4.7
0.17.0 0.11.1 100 0.52 4.6 0.59 4.7
0.17.1 0.11.1 100 0.52 4.6 0.59 4.7
0.17.2 0.11.1 100 0.52 4.6 0.59 4.7
0.18.0 0.11.1 100 0.52 4.6 0.59 4.7
0.19.0 0.11.1 100 0.52 4.6 0.59 4.7
0.20.0 1.0.0 100 0.53 4.6 0.60 4.7
0.20.1 1.0.1 100 0.53 4.6 0.60 4.7
0.20.2 1.0.1 100 0.53 4.6 0.60 4.7
0.21.0 1.0.1 100 0.53 4.6 0.60 4.7
0.21.1 1.0.1 100 0.53 4.6 0.60 4.7
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.52 4.6 0.59 4.7
0.14.2 0.10.0 1000 0.52 4.6 0.59 4.7
0.15.0 0.11.1 1000 0.52 4.6 0.59 4.7
0.16.0 0.11.1 1000 0.52 4.6 0.59 4.7
0.17.0 0.11.1 1000 0.52 4.6 0.59 4.7
0.17.1 0.11.1 1000 0.52 4.6 0.59 4.7
0.17.2 0.11.1 1000 0.52 4.6 0.59 4.7
0.18.0 0.11.1 1000 0.52 4.6 0.59 4.7
0.19.0 0.11.1 1000 0.52 4.6 0.59 4.7
0.20.0 1.0.0 1000 0.53 4.6 0.60 4.7
0.20.1 1.0.1 1000 0.53 4.6 0.60 4.7
0.20.2 1.0.1 1000 0.53 4.6 0.60 4.7
0.21.0 1.0.1 1000 0.53 4.6 0.60 4.7
0.21.1 1.0.1 1000 0.53 4.6 0.60 4.7

Learner

learner = lrn("classif.rpart")

Memory usage of a lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.23 0.23 0.42 0.42
0.14.2 0.10.0 10 0.23 0.23 0.42 0.42
0.15.0 0.11.1 10 0.23 0.23 0.42 0.42
0.16.0 0.11.1 10 0.23 0.23 0.42 0.42
0.17.0 0.11.1 10 0.23 0.23 0.42 0.42
0.17.1 0.11.1 10 0.23 0.23 0.42 0.42
0.17.2 0.11.1 10 0.23 0.23 0.42 0.42
0.18.0 0.11.1 10 0.23 0.23 0.42 0.42
0.19.0 0.11.1 10 0.23 0.23 0.42 0.42
0.20.0 1.0.0 10 0.081 0.081 0.12 0.12
0.20.1 1.0.1 10 0.082 0.082 0.12 0.12
0.20.2 1.0.1 10 0.082 0.082 0.12 0.12
0.21.0 1.0.1 10 0.084 0.084 0.13 0.13
0.21.1 1.0.1 10 0.084 0.084 0.13 0.13
100 Resampling Iterations
0.13.4 0.9.0 100 0.23 0.23 0.42 0.42
0.14.2 0.10.0 100 0.23 0.23 0.42 0.42
0.15.0 0.11.1 100 0.23 0.23 0.42 0.42
0.16.0 0.11.1 100 0.23 0.23 0.42 0.42
0.17.0 0.11.1 100 0.23 0.23 0.42 0.42
0.17.1 0.11.1 100 0.23 0.23 0.42 0.42
0.17.2 0.11.1 100 0.23 0.23 0.42 0.42
0.18.0 0.11.1 100 0.23 0.23 0.42 0.42
0.19.0 0.11.1 100 0.23 0.23 0.42 0.42
0.20.0 1.0.0 100 0.081 0.081 0.12 0.12
0.20.1 1.0.1 100 0.082 0.082 0.12 0.12
0.20.2 1.0.1 100 0.082 0.082 0.12 0.12
0.21.0 1.0.1 100 0.084 0.084 0.13 0.13
0.21.1 1.0.1 100 0.084 0.084 0.13 0.13
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.23 0.23 0.42 0.42
0.14.2 0.10.0 1000 0.23 0.23 0.42 0.42
0.15.0 0.11.1 1000 0.23 0.23 0.42 0.42
0.16.0 0.11.1 1000 0.23 0.23 0.42 0.42
0.17.0 0.11.1 1000 0.23 0.23 0.42 0.42
0.17.1 0.11.1 1000 0.23 0.23 0.42 0.42
0.17.2 0.11.1 1000 0.23 0.23 0.42 0.42
0.18.0 0.11.1 1000 0.23 0.23 0.42 0.42
0.19.0 0.11.1 1000 0.23 0.23 0.42 0.42
0.20.0 1.0.0 1000 0.081 0.081 0.12 0.12
0.20.1 1.0.1 1000 0.082 0.082 0.12 0.12
0.20.2 1.0.1 1000 0.082 0.082 0.12 0.12
0.21.0 1.0.1 1000 0.084 0.084 0.13 0.13
0.21.1 1.0.1 1000 0.084 0.084 0.13 0.13
learner$train(task)

Memory usage of a trained lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a trained lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.39 0.50 0.64 0.74
0.14.2 0.10.0 10 0.39 0.50 0.64 0.74
0.15.0 0.11.1 10 0.39 0.50 0.64 0.75
0.16.0 0.11.1 10 0.39 0.50 0.64 0.75
0.17.0 0.11.1 10 0.39 0.50 0.64 0.75
0.17.1 0.11.1 10 0.39 0.50 0.64 0.75
0.17.2 0.11.1 10 0.39 0.50 0.64 0.75
0.18.0 0.11.1 10 0.39 0.50 0.64 0.75
0.19.0 0.11.1 10 0.39 0.50 0.64 0.75
0.20.0 1.0.0 10 0.25 0.35 0.35 0.46
0.20.1 1.0.1 10 0.25 0.36 0.35 0.46
0.20.2 1.0.1 10 0.25 0.36 0.35 0.46
0.21.0 1.0.1 10 0.25 0.36 0.36 0.46
0.21.1 1.0.1 10 0.25 0.36 0.36 0.46
100 Resampling Iterations
0.13.4 0.9.0 100 0.39 0.50 0.64 0.74
0.14.2 0.10.0 100 0.39 0.50 0.64 0.74
0.15.0 0.11.1 100 0.39 0.50 0.64 0.75
0.16.0 0.11.1 100 0.39 0.50 0.64 0.75
0.17.0 0.11.1 100 0.39 0.50 0.64 0.75
0.17.1 0.11.1 100 0.39 0.50 0.64 0.75
0.17.2 0.11.1 100 0.39 0.50 0.64 0.75
0.18.0 0.11.1 100 0.39 0.50 0.64 0.75
0.19.0 0.11.1 100 0.39 0.50 0.64 0.75
0.20.0 1.0.0 100 0.25 0.35 0.35 0.46
0.20.1 1.0.1 100 0.25 0.36 0.35 0.46
0.20.2 1.0.1 100 0.25 0.36 0.35 0.46
0.21.0 1.0.1 100 0.25 0.36 0.36 0.46
0.21.1 1.0.1 100 0.25 0.36 0.36 0.46
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.39 0.50 0.64 0.74
0.14.2 0.10.0 1000 0.39 0.50 0.64 0.74
0.15.0 0.11.1 1000 0.39 0.50 0.64 0.75
0.16.0 0.11.1 1000 0.39 0.50 0.64 0.75
0.17.0 0.11.1 1000 0.39 0.50 0.64 0.75
0.17.1 0.11.1 1000 0.39 0.50 0.64 0.75
0.17.2 0.11.1 1000 0.39 0.50 0.64 0.75
0.18.0 0.11.1 1000 0.39 0.50 0.64 0.75
0.19.0 0.11.1 1000 0.39 0.50 0.64 0.75
0.20.0 1.0.0 1000 0.25 0.35 0.35 0.46
0.20.1 1.0.1 1000 0.25 0.36 0.35 0.46
0.20.2 1.0.1 1000 0.25 0.36 0.35 0.46
0.21.0 1.0.1 1000 0.25 0.36 0.36 0.46
0.21.1 1.0.1 1000 0.25 0.36 0.36 0.46
learner$model

Memory usage of a model in a lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a model in a lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.097 0.17 0.098 0.17
0.14.2 0.10.0 10 0.097 0.17 0.098 0.17
0.15.0 0.11.1 10 0.097 0.17 0.098 0.17
0.16.0 0.11.1 10 0.097 0.17 0.098 0.17
0.17.0 0.11.1 10 0.097 0.17 0.098 0.17
0.17.1 0.11.1 10 0.097 0.17 0.098 0.17
0.17.2 0.11.1 10 0.097 0.17 0.098 0.17
0.18.0 0.11.1 10 0.097 0.17 0.098 0.17
0.19.0 0.11.1 10 0.097 0.17 0.098 0.17
0.20.0 1.0.0 10 0.097 0.17 0.098 0.17
0.20.1 1.0.1 10 0.097 0.17 0.098 0.17
0.20.2 1.0.1 10 0.097 0.17 0.098 0.17
0.21.0 1.0.1 10 0.097 0.17 0.098 0.17
0.21.1 1.0.1 10 0.097 0.17 0.098 0.17
100 Resampling Iterations
0.13.4 0.9.0 100 0.097 0.17 0.098 0.17
0.14.2 0.10.0 100 0.097 0.17 0.098 0.17
0.15.0 0.11.1 100 0.097 0.17 0.098 0.17
0.16.0 0.11.1 100 0.097 0.17 0.098 0.17
0.17.0 0.11.1 100 0.097 0.17 0.098 0.17
0.17.1 0.11.1 100 0.097 0.17 0.098 0.17
0.17.2 0.11.1 100 0.097 0.17 0.098 0.17
0.18.0 0.11.1 100 0.097 0.17 0.098 0.17
0.19.0 0.11.1 100 0.097 0.17 0.098 0.17
0.20.0 1.0.0 100 0.097 0.17 0.098 0.17
0.20.1 1.0.1 100 0.097 0.17 0.098 0.17
0.20.2 1.0.1 100 0.097 0.17 0.098 0.17
0.21.0 1.0.1 100 0.097 0.17 0.098 0.17
0.21.1 1.0.1 100 0.097 0.17 0.098 0.17
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.097 0.17 0.098 0.17
0.14.2 0.10.0 1000 0.097 0.17 0.098 0.17
0.15.0 0.11.1 1000 0.097 0.17 0.098 0.17
0.16.0 0.11.1 1000 0.097 0.17 0.098 0.17
0.17.0 0.11.1 1000 0.097 0.17 0.098 0.17
0.17.1 0.11.1 1000 0.097 0.17 0.098 0.17
0.17.2 0.11.1 1000 0.097 0.17 0.098 0.17
0.18.0 0.11.1 1000 0.097 0.17 0.098 0.17
0.19.0 0.11.1 1000 0.097 0.17 0.098 0.17
0.20.0 1.0.0 1000 0.097 0.17 0.098 0.17
0.20.1 1.0.1 1000 0.097 0.17 0.098 0.17
0.20.2 1.0.1 1000 0.097 0.17 0.098 0.17
0.21.0 1.0.1 1000 0.097 0.17 0.098 0.17
0.21.1 1.0.1 1000 0.097 0.17 0.098 0.17
learner$param_set

Memory usage of the ParamSet of a lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of the ParamSet of a lrn("classif.rpart")object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.20 0.20 0.36 0.36
0.14.2 0.10.0 10 0.20 0.20 0.36 0.36
0.15.0 0.11.1 10 0.20 0.20 0.36 0.36
0.16.0 0.11.1 10 0.20 0.20 0.36 0.36
0.17.0 0.11.1 10 0.20 0.20 0.36 0.36
0.17.1 0.11.1 10 0.20 0.20 0.36 0.36
0.17.2 0.11.1 10 0.20 0.20 0.36 0.36
0.18.0 0.11.1 10 0.20 0.20 0.36 0.36
0.19.0 0.11.1 10 0.20 0.20 0.36 0.36
0.20.0 1.0.0 10 0.051 0.051 0.061 0.061
0.20.1 1.0.1 10 0.052 0.052 0.062 0.062
0.20.2 1.0.1 10 0.052 0.052 0.062 0.062
0.21.0 1.0.1 10 0.052 0.052 0.062 0.062
0.21.1 1.0.1 10 0.052 0.052 0.062 0.062
100 Resampling Iterations
0.13.4 0.9.0 100 0.20 0.20 0.36 0.36
0.14.2 0.10.0 100 0.20 0.20 0.36 0.36
0.15.0 0.11.1 100 0.20 0.20 0.36 0.36
0.16.0 0.11.1 100 0.20 0.20 0.36 0.36
0.17.0 0.11.1 100 0.20 0.20 0.36 0.36
0.17.1 0.11.1 100 0.20 0.20 0.36 0.36
0.17.2 0.11.1 100 0.20 0.20 0.36 0.36
0.18.0 0.11.1 100 0.20 0.20 0.36 0.36
0.19.0 0.11.1 100 0.20 0.20 0.36 0.36
0.20.0 1.0.0 100 0.051 0.051 0.061 0.061
0.20.1 1.0.1 100 0.052 0.052 0.062 0.062
0.20.2 1.0.1 100 0.052 0.052 0.062 0.062
0.21.0 1.0.1 100 0.052 0.052 0.062 0.062
0.21.1 1.0.1 100 0.052 0.052 0.062 0.062
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.20 0.20 0.36 0.36
0.14.2 0.10.0 1000 0.20 0.20 0.36 0.36
0.15.0 0.11.1 1000 0.20 0.20 0.36 0.36
0.16.0 0.11.1 1000 0.20 0.20 0.36 0.36
0.17.0 0.11.1 1000 0.20 0.20 0.36 0.36
0.17.1 0.11.1 1000 0.20 0.20 0.36 0.36
0.17.2 0.11.1 1000 0.20 0.20 0.36 0.36
0.18.0 0.11.1 1000 0.20 0.20 0.36 0.36
0.19.0 0.11.1 1000 0.20 0.20 0.36 0.36
0.20.0 1.0.0 1000 0.051 0.051 0.061 0.061
0.20.1 1.0.1 1000 0.052 0.052 0.062 0.062
0.20.2 1.0.1 1000 0.052 0.052 0.062 0.062
0.21.0 1.0.1 1000 0.052 0.052 0.062 0.062
0.21.1 1.0.1 1000 0.052 0.052 0.062 0.062

Prediction

pred = learner$predict(task)

lobstr::obj_size() seems to misreport the size of pred$data$response. Thus the objects size appears smaller after unserializing.

Memory usage of a Prediction object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a Prediction object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.092 0.78 0.034 0.14
0.14.2 0.10.0 10 0.092 0.78 0.034 0.14
0.15.0 0.11.1 10 0.092 0.78 0.034 0.14
0.16.0 0.11.1 10 0.092 0.78 0.034 0.14
0.17.0 0.11.1 10 0.093 0.78 0.034 0.14
0.17.1 0.11.1 10 0.093 0.78 0.034 0.14
0.17.2 0.11.1 10 0.093 0.78 0.034 0.14
0.18.0 0.11.1 10 0.093 0.78 0.034 0.14
0.19.0 0.11.1 10 0.093 0.78 0.034 0.14
0.20.0 1.0.0 10 0.093 0.78 0.034 0.14
0.20.1 1.0.1 10 0.093 0.78 0.034 0.14
0.20.2 1.0.1 10 0.093 0.78 0.034 0.14
0.21.0 1.0.1 10 0.093 0.78 0.035 0.14
0.21.1 1.0.1 10 0.093 0.78 0.035 0.14
100 Resampling Iterations
0.13.4 0.9.0 100 0.092 0.78 0.034 0.14
0.14.2 0.10.0 100 0.092 0.78 0.034 0.14
0.15.0 0.11.1 100 0.092 0.78 0.034 0.14
0.16.0 0.11.1 100 0.092 0.78 0.034 0.14
0.17.0 0.11.1 100 0.093 0.78 0.034 0.14
0.17.1 0.11.1 100 0.093 0.78 0.034 0.14
0.17.2 0.11.1 100 0.093 0.78 0.034 0.14
0.18.0 0.11.1 100 0.093 0.78 0.034 0.14
0.19.0 0.11.1 100 0.093 0.78 0.034 0.14
0.20.0 1.0.0 100 0.093 0.78 0.034 0.14
0.20.1 1.0.1 100 0.093 0.78 0.034 0.14
0.20.2 1.0.1 100 0.093 0.78 0.034 0.14
0.21.0 1.0.1 100 0.093 0.78 0.035 0.14
0.21.1 1.0.1 100 0.093 0.78 0.035 0.14
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.092 0.78 0.034 0.14
0.14.2 0.10.0 1000 0.092 0.78 0.034 0.14
0.15.0 0.11.1 1000 0.092 0.78 0.034 0.14
0.16.0 0.11.1 1000 0.092 0.78 0.034 0.14
0.17.0 0.11.1 1000 0.093 0.78 0.034 0.14
0.17.1 0.11.1 1000 0.093 0.78 0.034 0.14
0.17.2 0.11.1 1000 0.093 0.78 0.034 0.14
0.18.0 0.11.1 1000 0.093 0.78 0.034 0.14
0.19.0 0.11.1 1000 0.093 0.78 0.034 0.14
0.20.0 1.0.0 1000 0.093 0.78 0.034 0.14
0.20.1 1.0.1 1000 0.093 0.78 0.034 0.14
0.20.2 1.0.1 1000 0.093 0.78 0.034 0.14
0.21.0 1.0.1 1000 0.093 0.78 0.035 0.14
0.21.1 1.0.1 1000 0.093 0.78 0.035 0.14

Resampling

resampling = rsmp("subsampling", repeats = evals)

Memory usage of a Resampling object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a Resampling object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.16 0.16 0.27 0.27
0.14.2 0.10.0 10 0.16 0.16 0.27 0.27
0.15.0 0.11.1 10 0.16 0.16 0.27 0.27
0.16.0 0.11.1 10 0.16 0.16 0.27 0.27
0.17.0 0.11.1 10 0.16 0.16 0.27 0.27
0.17.1 0.11.1 10 0.16 0.16 0.27 0.27
0.17.2 0.11.1 10 0.16 0.16 0.27 0.27
0.18.0 0.11.1 10 0.16 0.16 0.27 0.27
0.19.0 0.11.1 10 0.16 0.16 0.27 0.27
0.20.0 1.0.0 10 0.064 0.064 0.080 0.080
0.20.1 1.0.1 10 0.064 0.064 0.081 0.081
0.20.2 1.0.1 10 0.064 0.064 0.081 0.081
0.21.0 1.0.1 10 0.065 0.065 0.081 0.081
0.21.1 1.0.1 10 0.065 0.065 0.081 0.081
100 Resampling Iterations
0.13.4 0.9.0 100 0.16 0.16 0.27 0.27
0.14.2 0.10.0 100 0.16 0.16 0.27 0.27
0.15.0 0.11.1 100 0.16 0.16 0.27 0.27
0.16.0 0.11.1 100 0.16 0.16 0.27 0.27
0.17.0 0.11.1 100 0.16 0.16 0.27 0.27
0.17.1 0.11.1 100 0.16 0.16 0.27 0.27
0.17.2 0.11.1 100 0.16 0.16 0.27 0.27
0.18.0 0.11.1 100 0.16 0.16 0.27 0.27
0.19.0 0.11.1 100 0.16 0.16 0.27 0.27
0.20.0 1.0.0 100 0.064 0.064 0.080 0.080
0.20.1 1.0.1 100 0.064 0.064 0.081 0.081
0.20.2 1.0.1 100 0.064 0.064 0.081 0.081
0.21.0 1.0.1 100 0.065 0.065 0.081 0.081
0.21.1 1.0.1 100 0.065 0.065 0.081 0.081
1000 Resampling Iterations
0.13.4 0.9.0 1000 0.16 0.16 0.27 0.27
0.14.2 0.10.0 1000 0.16 0.16 0.27 0.27
0.15.0 0.11.1 1000 0.16 0.16 0.27 0.27
0.16.0 0.11.1 1000 0.16 0.16 0.27 0.27
0.17.0 0.11.1 1000 0.16 0.16 0.27 0.27
0.17.1 0.11.1 1000 0.16 0.16 0.27 0.27
0.17.2 0.11.1 1000 0.16 0.16 0.27 0.27
0.18.0 0.11.1 1000 0.16 0.16 0.27 0.27
0.19.0 0.11.1 1000 0.16 0.16 0.27 0.27
0.20.0 1.0.0 1000 0.064 0.064 0.080 0.080
0.20.1 1.0.1 1000 0.064 0.064 0.081 0.081
0.20.2 1.0.1 1000 0.064 0.064 0.081 0.081
0.21.0 1.0.1 1000 0.065 0.065 0.081 0.081
0.21.1 1.0.1 1000 0.065 0.065 0.081 0.081
resampling$instantiate(task)

Memory usage of a instantiated Resampling object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a instantiated Resampling object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 0.19 0.47 0.30 0.58
0.14.2 0.10.0 10 0.19 0.47 0.30 0.58
0.15.0 0.11.1 10 0.19 0.47 0.30 0.58
0.16.0 0.11.1 10 0.19 0.47 0.30 0.58
0.17.0 0.11.1 10 0.19 0.47 0.30 0.58
0.17.1 0.11.1 10 0.19 0.47 0.31 0.58
0.17.2 0.11.1 10 0.19 0.47 0.31 0.58
0.18.0 0.11.1 10 0.19 0.47 0.31 0.58
0.19.0 0.11.1 10 0.19 0.47 0.31 0.58
0.20.0 1.0.0 10 0.096 0.37 0.11 0.39
0.20.1 1.0.1 10 0.096 0.37 0.11 0.39
0.20.2 1.0.1 10 0.096 0.37 0.11 0.39
0.21.0 1.0.1 10 0.096 0.37 0.11 0.39
0.21.1 1.0.1 10 0.096 0.37 0.11 0.39
100 Resampling Iterations
0.13.4 0.9.0 100 0.44 2.9 0.55 3.0
0.14.2 0.10.0 100 0.44 2.9 0.55 3.0
0.15.0 0.11.1 100 0.44 2.9 0.55 3.0
0.16.0 0.11.1 100 0.44 2.9 0.55 3.0
0.17.0 0.11.1 100 0.44 2.9 0.55 3.0
0.17.1 0.11.1 100 0.44 2.9 0.55 3.0
0.17.2 0.11.1 100 0.44 2.9 0.55 3.0
0.18.0 0.11.1 100 0.44 2.9 0.55 3.0
0.19.0 0.11.1 100 0.44 2.9 0.55 3.0
0.20.0 1.0.0 100 0.34 2.8 0.36 2.8
0.20.1 1.0.1 100 0.34 2.8 0.36 2.8
0.20.2 1.0.1 100 0.34 2.8 0.36 2.8
0.21.0 1.0.1 100 0.34 2.8 0.36 2.8
0.21.1 1.0.1 100 0.34 2.8 0.36 2.8
1000 Resampling Iterations
0.13.4 0.9.0 1000 2.9 27 3.0 27
0.14.2 0.10.0 1000 2.9 27 3.0 27
0.15.0 0.11.1 1000 2.9 27 3.0 27
0.16.0 0.11.1 1000 2.9 27 3.0 27
0.17.0 0.11.1 1000 2.9 27 3.0 27
0.17.1 0.11.1 1000 2.9 27 3.0 27
0.17.2 0.11.1 1000 2.9 27 3.0 27
0.18.0 0.11.1 1000 2.9 27 3.0 27
0.19.0 0.11.1 1000 2.9 27 3.0 27
0.20.0 1.0.0 1000 2.8 27 2.8 27
0.20.1 1.0.1 1000 2.8 27 2.8 27
0.20.2 1.0.1 1000 2.8 27 2.8 27
0.21.0 1.0.1 1000 2.8 27 2.8 27
0.21.1 1.0.1 1000 2.8 27 2.8 27

Resample Result

rr = resample(task, learner, resampling)

Memory usage of a ResampleResult object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a ResampleResult object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 1.0 5.7 1.5 6.3
0.14.2 0.10.0 10 1.0 5.7 1.5 6.3
0.15.0 0.11.1 10 1.0 5.7 1.5 6.3
0.16.0 0.11.1 10 1.0 5.7 1.5 6.3
0.17.0 0.11.1 10 1.0 5.7 1.5 6.3
0.17.1 0.11.1 10 0.98 6.1 1.4 6.2
0.17.2 0.11.1 10 0.98 6.1 1.4 6.2
0.18.0 0.11.1 10 0.98 6.1 1.4 6.2
0.19.0 0.11.1 10 0.99 6.1 1.4 6.2
0.20.0 1.0.0 10 0.83 5.9 0.94 5.7
0.20.1 1.0.1 10 0.83 5.9 0.94 5.7
0.20.2 1.0.1 10 0.83 5.9 0.94 5.7
0.21.0 1.0.1 10 0.84 5.9 0.95 5.7
0.21.1 1.0.1 10 0.84 5.9 0.95 5.7
100 Resampling Iterations
0.13.4 0.9.0 100 2.9 13 3.3 14
0.14.2 0.10.0 100 2.9 13 3.3 14
0.15.0 0.11.1 100 2.9 13 3.3 14
0.16.0 0.11.1 100 2.9 13 3.3 14
0.17.0 0.11.1 100 2.9 13 3.3 14
0.17.1 0.11.1 100 2.2 15 2.4 13
0.17.2 0.11.1 100 2.2 15 2.4 13
0.18.0 0.11.1 100 2.2 15 2.4 13
0.19.0 0.11.1 100 2.2 15 2.4 13
0.20.0 1.0.0 100 2.1 15 1.9 12
0.20.1 1.0.1 100 2.1 15 1.9 12
0.20.2 1.0.1 100 2.1 15 1.9 12
0.21.0 1.0.1 100 2.1 15 1.9 12
0.21.1 1.0.1 100 2.1 15 1.9 12
1000 Resampling Iterations
0.13.4 0.9.0 1000 21 85 22 86
0.14.2 0.10.0 1000 21 85 22 86
0.15.0 0.11.1 1000 21 85 22 86
0.16.0 0.11.1 1000 21 85 22 86
0.17.0 0.11.1 1000 21 85 22 86
0.17.1 0.11.1 1000 14 100 12 76
0.17.2 0.11.1 1000 14 100 12 76
0.18.0 0.11.1 1000 14 100 12 76
0.19.0 0.11.1 1000 15 100 12 76
0.20.0 1.0.0 1000 14 100 12 76
0.20.1 1.0.1 1000 14 100 12 76
0.20.2 1.0.1 1000 14 100 12 76
0.21.0 1.0.1 1000 14 100 12 76
0.21.1 1.0.1 1000 14 100 12 76
rr = resample(task, learner, resampling, store_models = TRUE)

Memory usage of a ResamplingResult object with models depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a ResamplingResult object with models depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 1.7 7.7 2.4 8.5
0.14.2 0.10.0 10 1.7 7.7 2.4 8.5
0.15.0 0.11.1 10 1.7 7.7 2.4 8.5
0.16.0 0.11.1 10 1.7 7.7 2.4 8.5
0.17.0 0.11.1 10 1.7 7.7 2.4 8.5
0.17.1 0.11.1 10 1.6 7.9 2.3 8.4
0.17.2 0.11.1 10 1.6 7.9 2.3 8.4
0.18.0 0.11.1 10 1.6 7.9 2.3 8.4
0.19.0 0.11.1 10 1.6 7.9 2.3 8.4
0.20.0 1.0.0 10 1.5 7.7 1.8 7.9
0.20.1 1.0.1 10 1.5 7.7 1.8 7.9
0.20.2 1.0.1 10 1.5 7.7 1.8 7.9
0.21.0 1.0.1 10 1.5 7.7 1.8 7.9
0.21.1 1.0.1 10 1.5 7.7 1.8 7.9
100 Resampling Iterations
0.13.4 0.9.0 100 8.7 29 12 32
0.14.2 0.10.0 100 8.7 29 12 32
0.15.0 0.11.1 100 8.7 29 12 32
0.16.0 0.11.1 100 8.7 29 12 32
0.17.0 0.11.1 100 8.7 29 12 32
0.17.1 0.11.1 100 8.0 30 11 31
0.17.2 0.11.1 100 8.0 30 11 31
0.18.0 0.11.1 100 8.0 30 11 31
0.19.0 0.11.1 100 8.0 30 11 31
0.20.0 1.0.0 100 7.9 30 10 30
0.20.1 1.0.1 100 7.9 30 10 30
0.20.2 1.0.1 100 7.9 30 10 30
0.21.0 1.0.1 100 7.9 30 10 30
0.21.1 1.0.1 100 7.9 30 10 30
1000 Resampling Iterations
0.13.4 0.9.0 1000 79 240 110 270
0.14.2 0.10.0 1000 79 240 110 270
0.15.0 0.11.1 1000 79 240 110 270
0.16.0 0.11.1 1000 79 240 110 270
0.17.0 0.11.1 1000 79 240 110 270
0.17.1 0.11.1 1000 72 260 96 260
0.17.2 0.11.1 1000 72 260 96 260
0.18.0 0.11.1 1000 72 260 96 260
0.19.0 0.11.1 1000 72 260 96 260
0.20.0 1.0.0 1000 72 260 96 260
0.20.1 1.0.1 1000 72 260 96 260
0.20.2 1.0.1 1000 72 260 96 260
0.21.0 1.0.1 1000 72 260 96 260
0.21.1 1.0.1 1000 72 260 96 260

Benchmark Result

bmr = benchmark(task, learner, resampling)

Memory usage of a BenchmarkResult object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a BenchmarkResult object depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 1.1 5.6 1.6 6.2
0.14.2 0.10.0 10 1.1 5.6 1.6 6.2
0.15.0 0.11.1 10 1.1 5.6 1.6 6.2
0.16.0 0.11.1 10 1.1 5.6 1.6 6.2
0.17.0 0.11.1 10 1.1 5.6 1.6 6.2
0.17.1 0.11.1 10 1.0 5.9 1.5 6.1
0.17.2 0.11.1 10 1.0 5.9 1.5 6.1
0.18.0 0.11.1 10 1.0 5.9 1.5 6.1
0.19.0 0.11.1 10 1.0 5.9 1.5 6.1
0.20.0 1.0.0 10 0.80 5.7 0.91 5.5
0.20.1 1.0.1 10 0.80 5.7 0.92 5.5
0.20.2 1.0.1 10 0.80 5.7 0.92 5.5
0.21.0 1.0.1 10 0.80 5.7 0.92 5.5
0.21.1 1.0.1 10 0.80 5.7 0.92 5.5
100 Resampling Iterations
0.13.4 0.9.0 100 2.6 11 3.2 11
0.14.2 0.10.0 100 2.6 11 3.2 11
0.15.0 0.11.1 100 2.6 11 3.2 11
0.16.0 0.11.1 100 2.6 11 3.2 11
0.17.0 0.11.1 100 2.6 11 3.2 11
0.17.1 0.11.1 100 1.9 13 2.3 11
0.17.2 0.11.1 100 1.9 13 2.3 11
0.18.0 0.11.1 100 1.9 13 2.3 11
0.19.0 0.11.1 100 2.0 13 2.3 11
0.20.0 1.0.0 100 1.7 12 1.7 9.9
0.20.1 1.0.1 100 1.7 12 1.7 9.9
0.20.2 1.0.1 100 1.7 12 1.7 9.9
0.21.0 1.0.1 100 1.7 12 1.7 10
0.21.1 1.0.1 100 1.7 12 1.7 10
1000 Resampling Iterations
0.13.4 0.9.0 1000 18 63 19 64
0.14.2 0.10.0 1000 18 63 19 64
0.15.0 0.11.1 1000 18 63 19 64
0.16.0 0.11.1 1000 18 63 19 64
0.17.0 0.11.1 1000 18 63 19 64
0.17.1 0.11.1 1000 11 80 9.9 55
0.17.2 0.11.1 1000 11 80 9.9 55
0.18.0 0.11.1 1000 11 80 9.9 55
0.19.0 0.11.1 1000 11 80 10 55
0.20.0 1.0.0 1000 11 80 9.4 54
0.20.1 1.0.1 1000 11 80 9.4 54
0.20.2 1.0.1 1000 11 80 9.4 54
0.21.0 1.0.1 1000 11 80 9.4 54
0.21.1 1.0.1 1000 11 80 9.4 54
bmr = benchmark(task, learner, resampling, store_models = TRUE)

Memory usage of a BenchmarkResult object with models depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
Memory usage of a BenchmarkResult object with models depending on the mlr3 version. The size is measured in memory and after calling serialize() and unserialize().
mlr3 Version paradox Version Resampling Iterations In Memory [MB] In Memory 10,000 [MB] Serialized [MB] Serialized 10,000 [MB]
10 Resampling Iterations
0.13.4 0.9.0 10 1.7 7.5 2.5 8.4
0.14.2 0.10.0 10 1.7 7.5 2.5 8.4
0.15.0 0.11.1 10 1.7 7.5 2.5 8.4
0.16.0 0.11.1 10 1.7 7.5 2.5 8.4
0.17.0 0.11.1 10 1.7 7.5 2.5 8.4
0.17.1 0.11.1 10 1.6 7.0 2.3 7.4
0.17.2 0.11.1 10 1.6 7.0 2.3 7.4
0.18.0 0.11.1 10 1.6 7.0 2.3 7.4
0.19.0 0.11.1 10 1.6 7.0 2.3 7.4
0.20.0 1.0.0 10 1.4 6.8 1.7 6.8
0.20.1 1.0.1 10 1.4 6.8 1.7 6.8
0.20.2 1.0.1 10 1.4 6.8 1.7 6.8
0.21.0 1.0.1 10 1.4 6.8 1.7 6.8
0.21.1 1.0.1 10 1.4 6.8 1.7 6.8
100 Resampling Iterations
0.13.4 0.9.0 100 8.5 27 12 30
0.14.2 0.10.0 100 8.5 27 12 30
0.15.0 0.11.1 100 8.5 27 12 30
0.16.0 0.11.1 100 8.5 27 12 30
0.17.0 0.11.1 100 8.5 27 12 30
0.17.1 0.11.1 100 7.1 23 10 23
0.17.2 0.11.1 100 7.1 23 10 23
0.18.0 0.11.1 100 7.1 23 10 23
0.19.0 0.11.1 100 7.1 23 10 23
0.20.0 1.0.0 100 6.9 22 9.6 23
0.20.1 1.0.1 100 6.9 22 9.6 23
0.20.2 1.0.1 100 6.9 22 9.6 23
0.21.0 1.0.1 100 6.9 22 9.7 23
0.21.1 1.0.1 100 6.9 22 9.7 23
1000 Resampling Iterations
0.13.4 0.9.0 1000 76 220 100 240
0.14.2 0.10.0 1000 76 220 100 240
0.15.0 0.11.1 1000 76 220 100 240
0.16.0 0.11.1 1000 76 220 100 240
0.17.0 0.11.1 1000 76 220 100 240
0.17.1 0.11.1 1000 62 180 89 180
0.17.2 0.11.1 1000 62 180 89 180
0.18.0 0.11.1 1000 62 180 89 180
0.19.0 0.11.1 1000 62 180 89 180
0.20.0 1.0.0 1000 62 180 88 180
0.20.1 1.0.1 1000 62 180 88 180
0.20.2 1.0.1 1000 62 180 88 180
0.21.0 1.0.1 1000 62 180 88 180
0.21.1 1.0.1 1000 62 180 88 180

Version Histroy

This section provides an overview of the changes in the mlr3 and paradox packages regarding memory usage and runtime.

mlr3

mlr3 0.21.0

  • Optimize the runtime of fixing factor levels.
  • Optimize the runtime of setting row roles.
  • Optimize the runtime of marshalling.
  • Optimize the runtime of Task$col_info.

mlr3 0.18.0

  • Skip unnecessary clone of learner’s state in resample().

mlr3 0.17.1

  • Remove data_prototype when resampling from learner$state to reduce memory consumption.
  • Optimize runtime of resample() and benchmark() by reducing the number of hashing operations.
  • Reduce number of threads used by data.table and BLAS to 1 when running resample() or benchmark() in parallel.

mlr3 0.17.0

  • Speed up resampling by removing unnecessary calls to packageVersion().
  • The design of benchmark() can now include parameter settings.

paradox

paradox 1.0.1

  • Performance improvements.

paradox 1.0.0

  • Removed Param objects. ParamSet now uses a data.table internally; individual parameters are more like Domain objects now.