data("Boston", package = "MASS")
head(Boston)
#> crim zn indus chas nox rm age dis rad tax ptratio black lstat medv
#> 1 0.00632 18 2.31 0 0.538 6.575 65.2 4.0900 1 296 15.3 396.90 4.98 24.0
#> 2 0.02731 0 7.07 0 0.469 6.421 78.9 4.9671 2 242 17.8 396.90 9.14 21.6
#> 3 0.02729 0 7.07 0 0.469 7.185 61.1 4.9671 2 242 17.8 392.83 4.03 34.7
#> 4 0.03237 0 2.18 0 0.458 6.998 45.8 6.0622 3 222 18.7 394.63 2.94 33.4
#> 5 0.06905 0 2.18 0 0.458 7.147 54.2 6.0622 3 222 18.7 396.90 5.33 36.2
#> 6 0.02985 0 2.18 0 0.458 6.430 58.7 6.0622 3 222 18.7 394.12 5.21 28.7
Machine learning models repeatedly outperform interpretable, parametric models like the linear regression model. The gains in performance have a price: The models operate as black boxes which are not interpretable.
Fortunately, there are many methods that can make machine learning models interpretable. The R package iml
provides tools for analysing any black box machine learning model:
- Feature importance: Which were the most important features?
- Feature effects: How does a feature influence the prediction? (Partial dependence plots and individual conditional expectation curves)
- Explanations for single predictions: How did the feature values of a single data point affect its prediction? (LIME and Shapley value)
- Surrogate trees: Can we approximate the underlying black box model with a short decision tree?
- The iml package works for any classification and regression machine learning model: random forests, linear models, neural networks, xgboost, etc.
This blog post shows you how to use the iml
package to analyse machine learning models. While the mlr
package makes it super easy to train machine learning models, the iml
package makes it easy to extract insights about the learned black box machine learning models.
If you want to learn more about the technical details of all the methods, read the Interpretable Machine Learning book.
Let’s explore the iml
-toolbox for interpreting an mlr
machine learning model with concrete examples!
Data: Boston Housing
We’ll use the MASS::Boston
dataset to demonstrate the abilities of the iml package. This dataset contains median house values from Boston neighbourhoods.
Fitting the machine learning model
First we train a randomForest to predict the Boston median housing value:
library("mlr")
data("Boston", package = "MASS")
# create an mlr task and model
= makeRegrTask(data = Boston, target = "medv")
tsk = makeLearner("regr.randomForest", ntree = 100)
lrn = train(lrn, tsk) mod
Using the iml Predictor container
We create a Predictor
object, that holds the model and the data. The iml
package uses R6 classes: New objects can be created by calling Predictor$new()
. Predictor
works best with mlr models (WrappedModel
-class), but it is also possible to use models from other packages.
library("iml")
= Boston[which(names(Boston) != "medv")]
X = Predictor$new(mod, data = X, y = Boston$medv) predictor
Feature importance
We can measure how important each feature was for the predictions with FeatureImp
. The feature importance measure works by shuffling each feature and measuring how much the performance drops. For this regression task we choose to measure the loss in performance with the mean absolute error (‘mae’); another choice would be the mean squared error (‘mse’).
Once we created a new object of FeatureImp
, the importance is automatically computed. We can call the plot()
function of the object or look at the results in a data.frame.
= FeatureImp$new(predictor, loss = "mae")
imp plot(imp)
$results
imp#> feature importance.05 importance importance.95 permutation.error
#> 1 lstat 4.321882 4.553394 4.637634 4.4285295
#> 2 rm 3.382867 3.429617 3.636302 3.3355693
#> 3 nox 1.787123 1.827256 1.832842 1.7771480
#> 4 crim 1.727133 1.732542 1.739232 1.6850317
#> 5 dis 1.661721 1.674545 1.692135 1.6286255
#> 6 ptratio 1.601922 1.636526 1.659229 1.5916490
#> 7 indus 1.443171 1.503451 1.518952 1.4622228
#> 8 tax 1.369524 1.386416 1.415495 1.3483973
#> 9 age 1.370219 1.383529 1.407349 1.3455898
#> 10 black 1.262825 1.272092 1.281437 1.2372085
#> 11 rad 1.105651 1.119705 1.124811 1.0890002
#> 12 zn 1.063475 1.067127 1.070519 1.0378637
#> 13 chas 1.016268 1.017959 1.026367 0.9900441
Partial dependence
Besides learning which features were important, we are interested in how the features influence the predicted outcome. The Partial
class implements partial dependence plots and individual conditional expectation curves. Each individual line represents the predictions