Skip to content

documentation: Clarify extraction of learner-specific outputs (e.g., coefficients, importance) after tuning with AutoTuner #891

@vstorme

Description

@vstorme

Hi,

As a user of mlr3 and mlr3learners, I’d like to suggest a documentation improvement for workflows that involve tuning models with AutoTuner.
While the tuning and resampling pipelines work very well, I found it difficult to access learner-specific outputs such as:

  • coefficients from regr.glmnet (Lasso/Ridge)
  • feature importance from regr.xgboost or regr.ranger

After tuning a learner using AutoTuner, I expected that I could extract model-specific details from at$model, but this either fails silently (e.g., coef() returns NULL) or throws errors (importance() unavailable). It took a lot of debugging to realize that the cleanest solution is to:

  • Extract the best hyperparameters from at$archive$best()
  • Reconstruct a new learner manually with fixed parameters
  • Train it directly on the task
  • Then access coef() or importance() safely from the learner-specific backend

I believe this pattern should be clearly documented in the mlr3book, perhaps under:

  • “Model Inspection” or
  • “Post-tuning workflow”

This would save users significant time, especially those expecting a caret- or tidymodels-like experience where coefficients or importances are easily accessible post-fit.

Thanks for all your great work — mlr3 is an incredibly powerful and well-designed framework, and I hope this suggestion helps smooth the learning curve a bit for others.
This is now my workflow for a lasso model:

`# 1/ Split into train/test (randomly, 80/20)
set.seed(1358)
splits = partition(tsk_mtcars, ratio = 0.8)

2/ define tuner

tnr_random_search = tnr("random_search")

3/ define terminator

trm_evals100 = trm("evals", n_evals=100)

4/ define learner

lrn_lasso = lrn("regr.glmnet",
alpha = 1, # Lasso
lambda = to_tune(p_dbl(lower = 1e-4, upper = 1, logscale = TRUE))
)

5/ Define inner and outer CV for nested resampling

outer_cv8 = rsmp("cv", folds = 8)
inner_cv5 = rsmp("cv", folds = 5)

6/ Define performance measure

msr_mse = msr("regr.mse")

7/ Wrap learner in AutoTuner

at = auto_tuner(
tuner = tnr_random_search,
terminator = trm_evals100,
learner = lrn_lasso,
resampling = inner_cv5,
measure = msr_mse
)

8/ Nested CV (on training set only):

first clone the task and then filter it (the resample function has no row_ids option)

tsk_mtcars_train <- tsk_mtcars$clone()$filter(splits$train)

set.seed(1358)
rr_lasso = resample(tsk_mtcars_train, at, outer_cv8, store_models = TRUE)

9/ Extract the generalization error and the standard error

rr_lasso$aggregate(msr_mse)
scores = rr_lasso$score(msr("regr.mse"))
mean_mse = mean(scores$regr.mse)
sd_mse = sd(scores$regr.mse)
se_mse = sd_mse / sqrt(dim(scores)[[1]])

10/ Inspect tuned hyperparameters for each fold (lambda is on the log scale)

extract_inner_tuning_results(rr_lasso)[, .(iteration, lambda, regr.mse)]

11/ Retrain best model on training data

set.seed(1358)
at$train(tsk_mtcars, row_ids = splits$train)
data.table(at$archive$data)[, .(lambda, regr.mse, runtime_learners)]

best hyperparam lambda:

lambda_log = at$archive$best()$lambda
as.data.table(at$archive)

12/ Predict on held-out test set

predict_test = at$predict(tsk_mtcars, row_ids = splits$test)
rmse_test = round(predict_test$score(msr_mse),2)

convert to data frame

tsk_mtcars$truth(splits$test)
pred.dt <- as.data.table(predict_test)
pred.df <- as.data.frame(pred.dt) #didn't work when used directly on R6 object

13/ plot truth vs predicted (response)

autoplot(predict_test)

14/ model coefficients of final model:

this requires to reconstruct the learner with the best parameters found in previous steps

the following does not return a glmnet object:

no_glmnet_model = at$model
class(no_glmnet_model)

"auto_tuner_model" "list"

14.1 extract best parameters after tuning:

best_params = at$archive$best()$x_domain[[1]]

14.2 Rebuild the final model with the best parameters

lrn_lasso_final = lrn("regr.glmnet",
alpha = 1,
lambda = best_params$lambda
)

14.3 train on the train data set

lrn_lasso_final$train(tsk_mtcars, row_ids = splits$train)
names(lrn_lasso_final)

14.4 extract coefficients

glmnet_model = lrn_lasso_final$model
class(glmnet_model)

"elnet" "glmnet"

coefs = coef(glmnet_model, s = best_params$lambda)
as.matrix(coefs)[as.matrix(coefs) != 0, , drop = FALSE]`

best regards,
Veronique

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions