paradox - Runtime Benchmarks

Scope

This report analyzes the runtime and memory usage of the paradox package across different versions. The benchmarks vary the type and the size of the parameter space.

Given the extensive package ecosystem of mlr3, performance bottlenecks can occur at multiple stages. This report aims to help users determine whether the runtime of their workflows falls within expected ranges. If significant runtime or memory anomalies are observed, users are encouraged to report them by opening a GitHub issue.

Benchmarks are conducted on a high-performance cluster optimized for multi-core performance rather than single-core speed. Consequently, runtimes may be faster on a local machine.

Summary of Latest paradox Version

The benchmarks are comprehensive; therefore, we present a summary of the most important functions in the latest paradox version. We start by measuring the runtime of the ps() function when initializing a parameter space in the size of 5, 50 and 500 parameters. The runtime increases from 2 to 10 to 100 ms but is independent of the type of the parameters. Getting values with the $get_values() method takes around 100 µs. The $set_values() method takes a half millisecond for 5 values and 14 ms for 500 values.

Initialize

The runtime of the initialize() method is measured for different paradox versions.

param_set = ps(
  a = p_dbl(lower = 0, upper = 1),
  b = p_dbl(lower = 0, upper = 1),
  c = p_dbl(lower = 0, upper = 1),
  d = p_dbl(lower = 0, upper = 1),
  e = p_dbl(lower = 0, upper = 1)
)

Runtime of the ps() function by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	2.4
1.0.1	categorical	5	2.1
1.1.0	mixed	5	2.0
1.0.1	mixed	5	1.9
1.1.0	numeric	5	1.6
1.0.1	numeric	5	1.9
50 Parameters
1.1.0	categorical	50	11
1.0.1	categorical	50	11
1.1.0	mixed	50	11
1.0.1	mixed	50	9.9
1.1.0	numeric	50	8.6
1.0.1	numeric	50	9.5
500 Parameters
1.1.0	categorical	500	110
1.0.1	categorical	500	110
1.1.0	mixed	500	99
1.0.1	mixed	500	89
1.1.0	numeric	500	87
1.0.1	numeric	500	74

Ids

The runtime usage of the param_set$ids() method is measured for different paradox versions.

param_set$ids()

Runtime of the $ids() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.037
1.0.1	categorical	5	0.039
1.1.0	mixed	5	0.036
1.0.1	mixed	5	0.038
1.1.0	numeric	5	0.038
1.0.1	numeric	5	0.035
50 Parameters
1.1.0	categorical	50	0.036
1.0.1	categorical	50	0.034
1.1.0	mixed	50	0.038
1.0.1	mixed	50	0.034
1.1.0	numeric	50	0.038
1.0.1	numeric	50	0.038
500 Parameters
1.1.0	categorical	500	0.038
1.0.1	categorical	500	0.034
1.1.0	mixed	500	0.038
1.0.1	mixed	500	0.034
1.1.0	numeric	500	0.035
1.0.1	numeric	500	0.034

Subset the parameter space by class.

param_set$ids(class = "numeric")

Runtime of the $ids() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	1.1
1.0.1	categorical	5	1.1
1.1.0	mixed	5	1.1
1.0.1	mixed	5	1.3
1.1.0	numeric	5	1.1
1.0.1	numeric	5	1.1
50 Parameters
1.1.0	categorical	50	1.2
1.0.1	categorical	50	1.1
1.1.0	mixed	50	1.1
1.0.1	mixed	50	1.1
1.1.0	numeric	50	1.2
1.0.1	numeric	50	1.0
500 Parameters
1.1.0	categorical	500	1.1
1.0.1	categorical	500	1.1
1.1.0	mixed	500	1.1
1.0.1	mixed	500	1.1
1.1.0	numeric	500	1.1
1.0.1	numeric	500	1.1

Subset the parameter space by tags.

param_set$ids(tags = "a")

Runtime of the $ids() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.044
1.0.1	categorical	5	0.042
1.1.0	mixed	5	0.042
1.0.1	mixed	5	0.037
1.1.0	numeric	5	0.040
1.0.1	numeric	5	0.038
50 Parameters
1.1.0	categorical	50	0.044
1.0.1	categorical	50	0.039
1.1.0	mixed	50	0.044
1.0.1	mixed	50	0.044
1.1.0	numeric	50	0.063
1.0.1	numeric	50	0.044
500 Parameters
1.1.0	categorical	500	0.067
1.0.1	categorical	500	0.061
1.1.0	mixed	500	0.059
1.0.1	mixed	500	0.059
1.1.0	numeric	500	0.060
1.0.1	numeric	500	0.058

Subset the parameter space by tags and class.

param_set$ids(tags = "a", class = "numeric")

Runtime of the $get_values() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	2.2
1.0.1	categorical	5	2.1
1.1.0	mixed	5	2.1
1.0.1	mixed	5	2.1
1.1.0	numeric	5	2.1
1.0.1	numeric	5	2.2
50 Parameters
1.1.0	categorical	50	2.1
1.0.1	categorical	50	2.1
1.1.0	mixed	50	2.1
1.0.1	mixed	50	2.1
1.1.0	numeric	50	2.1
1.0.1	numeric	50	2.1
500 Parameters
1.1.0	categorical	500	2.2
1.0.1	categorical	500	2.3
1.1.0	mixed	500	2.3
1.0.1	mixed	500	2.4
1.1.0	numeric	500	2.3
1.0.1	numeric	500	2.3

Get Values

The runtime of the $get_values() method is measured for different paradox versions.

param_set$get_values()

Runtime of the $get_values() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.11
1.0.1	categorical	5	0.11
1.1.0	mixed	5	0.11
1.0.1	mixed	5	0.11
1.1.0	numeric	5	0.10
1.0.1	numeric	5	0.10
50 Parameters
1.1.0	categorical	50	0.11
1.0.1	categorical	50	0.11
1.1.0	mixed	50	0.11
1.0.1	mixed	50	0.10
1.1.0	numeric	50	0.11
1.0.1	numeric	50	0.11
500 Parameters
1.1.0	categorical	500	0.12
1.0.1	categorical	500	0.14
1.1.0	mixed	500	0.14
1.0.1	mixed	500	0.14
1.1.0	numeric	500	0.15
1.0.1	numeric	500	0.15

Subset the parameter space by class.

param_set$get_values(class = "numeric")

Runtime of the $get_values() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	1.2
1.0.1	categorical	5	1.2
1.1.0	mixed	5	1.2
1.0.1	mixed	5	1.2
1.1.0	numeric	5	1.2
1.0.1	numeric	5	1.2
50 Parameters
1.1.0	categorical	50	1.2
1.0.1	categorical	50	1.2
1.1.0	mixed	50	1.2
1.0.1	mixed	50	1.1
1.1.0	numeric	50	1.2
1.0.1	numeric	50	1.2
500 Parameters
1.1.0	categorical	500	1.2
1.0.1	categorical	500	1.2
1.1.0	mixed	500	1.3
1.0.1	mixed	500	1.3
1.1.0	numeric	500	1.3
1.0.1	numeric	500	1.2

Subset the parameter space by tags.

param_set$get_values(tags = "a")

Runtime of the $get_values() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.11
1.0.1	categorical	5	0.11
1.1.0	mixed	5	0.11
1.0.1	mixed	5	0.11
1.1.0	numeric	5	0.11
1.0.1	numeric	5	0.10
50 Parameters
1.1.0	categorical	50	0.12
1.0.1	categorical	50	0.11
1.1.0	mixed	50	0.11
1.0.1	mixed	50	0.12
1.1.0	numeric	50	0.12
1.0.1	numeric	50	0.12
500 Parameters
1.1.0	categorical	500	0.15
1.0.1	categorical	500	0.15
1.1.0	mixed	500	0.15
1.0.1	mixed	500	0.14
1.1.0	numeric	500	0.15
1.0.1	numeric	500	0.15

Subset the parameter space by tags and class.

param_set$get_values(tags = "a", class = "numeric")

Runtime of the $get_values() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	2.3
1.0.1	categorical	5	2.3
1.1.0	mixed	5	2.3
1.0.1	mixed	5	2.3
1.1.0	numeric	5	2.3
1.0.1	numeric	5	2.2
50 Parameters
1.1.0	categorical	50	2.1
1.0.1	categorical	50	2.1
1.1.0	mixed	50	2.3
1.0.1	mixed	50	2.3
1.1.0	numeric	50	2.3
1.0.1	numeric	50	2.2
500 Parameters
1.1.0	categorical	500	2.4
1.0.1	categorical	500	2.5
1.1.0	mixed	500	2.5
1.0.1	mixed	500	2.5
1.1.0	numeric	500	2.3
1.0.1	numeric	500	2.3

Set Values

The runtime of the $set_values() method is measured for different paradox versions.

param_set$set_values(a = 1)

Runtime of the $set_values() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.56
1.0.1	categorical	5	0.51
1.1.0	mixed	5	0.77
1.0.1	mixed	5	0.73
1.1.0	numeric	5	0.61
1.0.1	numeric	5	0.59
50 Parameters
1.1.0	categorical	50	1.7
1.0.1	categorical	50	1.8
1.1.0	mixed	50	2.0
1.0.1	mixed	50	1.9
1.1.0	numeric	50	1.8
1.0.1	numeric	50	1.8
500 Parameters
1.1.0	categorical	500	14
1.0.1	categorical	500	13
1.1.0	mixed	500	14
1.0.1	mixed	500	14
1.1.0	numeric	500	14
1.0.1	numeric	500	13

Lower and Upper

The runtime of getting the lower and upper bounds of the parameter space is measured for different paradox versions.

param_set$lower

Runtime of the $lower field by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.015
1.0.1	categorical	5	0.015
1.1.0	mixed	5	0.015
1.0.1	mixed	5	0.015
1.1.0	numeric	5	0.016
1.0.1	numeric	5	0.015
50 Parameters
1.1.0	categorical	50	0.016
1.0.1	categorical	50	0.016
1.1.0	mixed	50	0.015
1.0.1	mixed	50	0.015
1.1.0	numeric	50	0.016
1.0.1	numeric	50	0.015
500 Parameters
1.1.0	categorical	500	0.015
1.0.1	categorical	500	0.015
1.1.0	mixed	500	0.015
1.0.1	mixed	500	0.015
1.1.0	numeric	500	0.015
1.0.1	numeric	500	0.015

Runtime of $upper().

param_set$upper

Runtime of the $upper field by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.015
1.0.1	categorical	5	0.015
1.1.0	mixed	5	0.016
1.0.1	mixed	5	0.015
1.1.0	numeric	5	0.015
1.0.1	numeric	5	0.015
50 Parameters
1.1.0	categorical	50	0.016
1.0.1	categorical	50	0.015
1.1.0	mixed	50	0.015
1.0.1	mixed	50	0.015
1.1.0	numeric	50	0.016
1.0.1	numeric	50	0.015
500 Parameters
1.1.0	categorical	500	0.015
1.0.1	categorical	500	0.015
1.1.0	mixed	500	0.015
1.0.1	mixed	500	0.015
1.1.0	numeric	500	0.015
1.0.1	numeric	500	0.015

Levels

The runtime of getting the levels of the parameter space is measured for different paradox versions.

param_set$levels

Runtime of the $levels field by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.016
1.0.1	categorical	5	0.015
1.1.0	mixed	5	0.015
1.0.1	mixed	5	0.015
1.1.0	numeric	5	0.015
1.0.1	numeric	5	0.015
50 Parameters
1.1.0	categorical	50	0.016
1.0.1	categorical	50	0.016
1.1.0	mixed	50	0.016
1.0.1	mixed	50	0.016
1.1.0	numeric	50	0.016
1.0.1	numeric	50	0.016
500 Parameters
1.1.0	categorical	500	0.015
1.0.1	categorical	500	0.015
1.1.0	mixed	500	0.015
1.0.1	mixed	500	0.020
1.1.0	numeric	500	0.015
1.0.1	numeric	500	0.016

Class

The runtime of getting the classes of the parameter space is measured for different paradox versions.

param_set$class

Runtime of the $class field by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	0.015
1.0.1	categorical	5	0.015
1.1.0	mixed	5	0.015
1.0.1	mixed	5	0.015
1.1.0	numeric	5	0.015
1.0.1	numeric	5	0.015
50 Parameters
1.1.0	categorical	50	0.016
1.0.1	categorical	50	0.015
1.1.0	mixed	50	0.015
1.0.1	mixed	50	0.015
1.1.0	numeric	50	0.015
1.0.1	numeric	50	0.016
500 Parameters
1.1.0	categorical	500	0.015
1.0.1	categorical	500	0.015
1.1.0	mixed	500	0.015
1.0.1	mixed	500	0.015
1.1.0	numeric	500	0.016
1.0.1	numeric	500	0.017

Subspaces

The runtime of subsetting the parameter space into subspaces of size 1 is measured.

param_set$subspaces()

Runtime of the $subspaces() method by paradox version, space size, parameter type and paradox version. The table includes runtimes for spaces of size 5, 50 and 500 parameters and for numeric, categorical and mixed parameters.

paradox	Type	Size	Runtime, ms
5 Parameters
1.1.0	categorical	5	14
1.0.1	categorical	5	14
1.1.0	mixed	5	16
1.0.1	mixed	5	13
1.1.0	numeric	5	13
1.0.1	numeric	5	15
50 Parameters
1.1.0	categorical	50	140
1.0.1	categorical	50	140
1.1.0	mixed	50	150
1.0.1	mixed	50	140
1.1.0	numeric	50	130
1.0.1	numeric	50	140
500 Parameters
1.1.0	categorical	500	1600
1.0.1	categorical	500	1600
1.1.0	mixed	500	1700
1.0.1	mixed	500	1500
1.1.0	numeric	500	1700
1.0.1	numeric	500	1500