The mlr3 ecosystem has a new package mlr3inferr that allows to construct confidence intervals for the generalization error.