Introduction

The tune package helps optimize the modeling process. Users can tag arguments in recipes and model objects for optimization. The search routines in tune can discover these arguments and evaluate candidate values until a combination with good performance is found.

As an example, let’s model the Ames housing data:

library(tidymodels)

data(ames)

set.seed(4595)
data_split <- ames %>%
  mutate(Sale_Price = log10(Sale_Price)) %>%
  initial_split(strata = Sale_Price)
ames_train <- training(data_split)
ames_test  <- testing(data_split)

For simplicity, the sale price of a house will be modeled as a function of its geo-location. These predictors appear to have nonlinear relationships with the outcome:

ames_train %>% 
  dplyr::select(Sale_Price, Longitude, Latitude) %>% 
  tidyr::pivot_longer(cols = c(Longitude, Latitude), 
                      names_to = "predictor", values_to = "value") %>% 
  ggplot(aes(x = value, Sale_Price)) + 
  geom_point(alpha = .2) + 
  geom_smooth(se = FALSE) + 
  facet_wrap(~ predictor, scales = "free_x")
#> `geom_smooth()` using method = 'gam' and formula 'y ~ s(x, bs = "cs")'

These two predictors could be modeled using natural splines in conjunction with a linear model. The amount of “wiggliness” in these splines is determined by the degrees of freedom. An appropriate value of this parameter cannot be analytically determined from the data, so it is a tuning parameter (a.k.a. a hyper-parameter). A common approach is to use resampling to estimate model performance over different values of these parameters and use these results to set reasonable values.

We can tag these parameters for optimization using the tune() function:

ames_rec <- 
  recipe(Sale_Price ~ Gr_Liv_Area + Longitude + Latitude, data = ames_train) %>% 
  step_log(Gr_Liv_Area, base = 10) %>% 
  step_ns(Longitude, Latitude, deg_free = tune())

The package can detect these values and optimize them.

However, based on the plot above, the potential amount of non-linearity between the sale price and the predictors might be different. For example, longitude might require more flexibility than latitude. The recipe above would constrain the nonlinearity of the predictors to be the same. We can probably do better than that.

To accomplish this, individual step_ns() terms can be added to the recipe for each predictor. However, we want these to be identifiable; using the same syntax as above, we can’t tell the difference between the two deg_free parameters.

tune() has an option to provide a text annotation so that each tuning parameter has a unique identifier:

ames_rec <- 
  recipe(Sale_Price ~ Gr_Liv_Area + Longitude + Latitude, data = ames_train) %>% 
  step_log(Gr_Liv_Area, base = 10) %>% 
  step_ns(Longitude, deg_free = tune("long df")) %>% 
  step_ns(Latitude,  deg_free = tune("lat df"))

The function dials::parameters() that can detect and collect the parameters that have been flagged for tuning:

parameters(ames_rec)
#> Collection of 2 parameters for tuning
#> 
#>  identifier     type    object
#>     long df deg_free nparam[+]
#>      lat df deg_free nparam[+]

The dials package has default ranges for many parameters. The generic parameter function for deg_free has a fairly small range:

deg_free()
#> Degrees of Freedom (quantitative)
#> Range: [1, 5]

but there is a dials function that is more appropriate for splines:

spline_degree()
#> Piecewise Polynomial Degree (quantitative)
#> Range: [1, 10]

The parameter objects can be easily changed using the update() function:

ames_param <- 
  ames_rec %>% 
  parameters() %>% 
  update(
    `long df` = spline_degree(), 
    `lat df` = spline_degree()
  )
ames_param
#> Collection of 2 parameters for tuning
#> 
#>  identifier     type    object
#>     long df deg_free nparam[+]
#>      lat df deg_free nparam[+]

Model Optimization

Instead of a linear regression, a nonlinear model might provide good performance. A K-nearest-neighbor fit will also be optimized. For this example, the number of neighbors and the distance weighting function will be optimized:

# requires the kknn package
knn_mod <- 
  nearest_neighbor(neighbors = tune(), weight_func = tune()) %>% 
  set_engine("kknn") %>% 
  set_mode("regression")

The easiest approach to optimize the pre-processing and model parameters is to bundle these objects into a workflow:

library(workflows)
knn_wflow <- 
  workflow() %>% 
  add_model(knn_mod) %>% 
  add_recipe(ames_rec)

From this, the parameter set can be used to modify the range and values of parameters being optimized3:

knn_param <- 
  knn_wflow %>% 
  parameters() %>% 
    update(
    `long df` = spline_degree(c(2, 18)), 
    `lat df` = spline_degree(c(2, 18)),
    neighbors = neighbors(c(3, 50)),
    weight_func = weight_func(values = c("rectangular", "inv", "gaussian", "triangular"))
  )

This parameter collection can be passed to the grid functions via the param_info arguments.

Instead of using grid search, an iterative method called Bayesian optimization can be used. This takes an initial set of results and tries to predict the next tuning parameters to evaluate.

Although no grid is required, the process requires a few additional pieces of information:

  • A description of the search space. At a minimum, the would consist of ranges for numeric values and a list of values for categorical tuning parameters.

  • An acquisition function that helps score potential tuning parameter values.

  • A model for analyzing and making predictions of the best tuning parameter values. A Gaussian Process model is typical and used here.

The code to conduct the search is:

ctrl <- control_bayes(verbose = TRUE)
set.seed(8154)
knn_search <- tune_bayes(knn_wflow, resamples = cv_splits, initial = 5, iter = 20,
                         param_info = knn_param, control = ctrl)
#> 
#> ❯  Generating a set of 5 initial parameter results
#> ✓ Initialization complete
#> 
#> Optimizing rmse using the expected improvement
#> 
#> ── Iteration 1 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08684 (@iter 0)
#> i Gaussian process model
#> ! The Gaussian process model is being fit using 7 features but only has 5
#>   data points to do so. This may cause errors or a poor model fit.
#> ✓ Gaussian process model
#> i Generating 4779 candidates
#> i Predicted candidates
#> i neighbors=3, weight_func=triangular, long df=2, lat df=5
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08983 (+/-0.00218)
#> 
#> ── Iteration 2 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08684 (@iter 0)
#> i Gaussian process model
#> ! The Gaussian process model is being fit using 7 features but only has 6
#>   data points to do so. This may cause errors or a poor model fit.
#> ✓ Gaussian process model
#> i Generating 4753 candidates
#> i Predicted candidates
#> i neighbors=3, weight_func=triangular, long df=14, lat df=11
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.09125 (+/-0.00172)
#> 
#> ── Iteration 3 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08684 (@iter 0)
#> i Gaussian process model
#> ! The Gaussian process model is being fit using 7 features but only has 7
#>   data points to do so. This may cause errors or a poor model fit.
#> ✓ Gaussian process model
#> i Generating 4766 candidates
#> i Predicted candidates
#> i neighbors=25, weight_func=triangular, long df=5, lat df=18
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.09308 (+/-0.00281)
#> 
#> ── Iteration 4 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08684 (@iter 0)
#> i Gaussian process model
#> ! The Gaussian process model is being fit using 7 features but only has 8
#>   data points to do so. This may cause errors or a poor model fit.
#> ✓ Gaussian process model
#> i Generating 4764 candidates
#> i Predicted candidates
#> i neighbors=50, weight_func=triangular, long df=15, lat df=15
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.09795 (+/-0.00278)
#> 
#> ── Iteration 5 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08684 (@iter 0)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4757 candidates
#> i Predicted candidates
#> i neighbors=7, weight_func=triangular, long df=2, lat df=17
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.09045 (+/-0.0024)
#> 
#> ── Iteration 6 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08684 (@iter 0)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4749 candidates
#> i Predicted candidates
#> i neighbors=3, weight_func=inv, long df=10, lat df=5
#> i Estimating performance
#> ✓ Estimating performance
#> ♥ Newest results:    rmse=0.08615 (+/-0.00229)
#> 
#> ── Iteration 7 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08615 (@iter 6)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4772 candidates
#> i Predicted candidates
#> i neighbors=14, weight_func=inv, long df=10, lat df=2
#> i Estimating performance
#> ✓ Estimating performance
#> ♥ Newest results:    rmse=0.08614 (+/-0.0026)
#> 
#> ── Iteration 8 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08614 (@iter 7)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4782 candidates
#> i Predicted candidates
#> i neighbors=7, weight_func=inv, long df=9, lat df=5
#> i Estimating performance
#> ✓ Estimating performance
#> ♥ Newest results:    rmse=0.08329 (+/-0.00238)
#> 
#> ── Iteration 9 ─────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08329 (@iter 8)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4766 candidates
#> i Predicted candidates
#> i neighbors=30, weight_func=gaussian, long df=9, lat df=3
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08938 (+/-0.00232)
#> 
#> ── Iteration 10 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08329 (@iter 8)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4780 candidates
#> i Predicted candidates
#> i neighbors=7, weight_func=inv, long df=8, lat df=12
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08492 (+/-0.00259)
#> 
#> ── Iteration 11 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08329 (@iter 8)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4798 candidates
#> i Predicted candidates
#> i neighbors=3, weight_func=inv, long df=9, lat df=18
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.09111 (+/-0.0023)
#> 
#> ── Iteration 12 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08329 (@iter 8)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4775 candidates
#> i Predicted candidates
#> i neighbors=8, weight_func=triangular, long df=8, lat df=5
#> i Estimating performance
#> ✓ Estimating performance
#> ♥ Newest results:    rmse=0.08231 (+/-0.00239)
#> 
#> ── Iteration 13 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4775 candidates
#> i Predicted candidates
#> i neighbors=9, weight_func=inv, long df=7, lat df=6
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08314 (+/-0.00265)
#> 
#> ── Iteration 14 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4764 candidates
#> i Predicted candidates
#> i neighbors=8, weight_func=gaussian, long df=8, lat df=5
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08374 (+/-0.00251)
#> 
#> ── Iteration 15 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4772 candidates
#> i Predicted candidates
#> i neighbors=12, weight_func=inv, long df=8, lat df=6
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08382 (+/-0.00254)
#> 
#> ── Iteration 16 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4739 candidates
#> i Predicted candidates
#> i neighbors=9, weight_func=triangular, long df=9, lat df=6
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08305 (+/-0.00247)
#> 
#> ── Iteration 17 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4779 candidates
#> i Predicted candidates
#> i neighbors=9, weight_func=triangular, long df=8, lat df=2
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08467 (+/-0.00273)
#> 
#> ── Iteration 18 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4750 candidates
#> i Predicted candidates
#> i neighbors=6, weight_func=triangular, long df=8, lat df=7
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08341 (+/-0.00229)
#> 
#> ── Iteration 19 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4757 candidates
#> i Predicted candidates
#> i neighbors=8, weight_func=triangular, long df=5, lat df=10
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08386 (+/-0.00227)
#> 
#> ── Iteration 20 ────────────────────────────────────────────────────────────────
#> 
#> i Current best:      rmse=0.08231 (@iter 12)
#> i Gaussian process model
#> ✓ Gaussian process model
#> i Generating 4780 candidates
#> i Predicted candidates
#> i neighbors=10, weight_func=triangular, long df=8, lat df=5
#> i Estimating performance
#> ✓ Estimating performance
#> ⓧ Newest results:    rmse=0.08241 (+/-0.00249)

Visually, the performance gain was:

autoplot(knn_search, type = "performance", metric = "rmse")

The best results here were:

collect_metrics(knn_search) %>% 
  dplyr::filter(.metric == "rmse") %>% 
  arrange(mean)
#> # A tibble: 25 x 11
#>    neighbors weight_func `long df` `lat df` .metric .estimator   mean     n
#>        <int> <chr>           <int>    <int> <chr>   <chr>       <dbl> <int>
#>  1         8 triangular          8        5 rmse    standard   0.0823    10
#>  2        10 triangular          8        5 rmse    standard   0.0824    10
#>  3         9 triangular          9        6 rmse    standard   0.0831    10
#>  4         9 inv                 7        6 rmse    standard   0.0831    10
#>  5         7 inv                 9        5 rmse    standard   0.0833    10
#>  6         6 triangular          8        7 rmse    standard   0.0834    10
#>  7         8 gaussian            8        5 rmse    standard   0.0837    10
#>  8        12 inv                 8        6 rmse    standard   0.0838    10
#>  9         8 triangular          5       10 rmse    standard   0.0839    10
#> 10         9 triangular          8        2 rmse    standard   0.0847    10
#> # … with 15 more rows, and 3 more variables: std_err <dbl>, .config <chr>,
#> #   .iter <int>

With this intrinsically nonlinear model there is less reliance on the nonlinear terms created by the recipe.


  1. A simple R model formula could have been used here, such as Sale_Price ~ log10(Gr_Liv_Area) + Longitude + Latitude. A recipe is not required.↩︎

  2. tune has default measures of performance that it uses if none are specified. Here the RMSE and R2 are estimated. This can be changed using the metrics option.↩︎

  3. One of the tuning parameters (weight_func) is categorical and, by default, has 10 unique values. The model used to predict new test parameters is a Gaussian process model, and this can become slow to fit when the number of tuning parameters is large or when a categorical parameter generates many dummy variables. We’ve reduced the number of categories for this parameter to speed things up a bit.↩︎