随机预测中的超参数整定

Question

我试图在Boston数据集上使用随机森林算法，借助sklearn的RandomForestRegressor来预测房价

迭代1：使用具有默认超参数的模型

#1. import the class/model
from sklearn.ensemble import RandomForestRegressor
#2. Instantiate the estimator
RFReg = RandomForestRegressor(random_state = 1, n_jobs = -1) 
#3. Fit the model with data aka model training
RFReg.fit(X_train, y_train)

#4. Predict the response for a new observation
y_pred = RFReg.predict(X_test)


y_pred_train = RFReg.predict(X_train)

迭代1的结果

^{pr2}$

迭代2：我使用RandomizedSearchCV来获得超参数的最佳值

from sklearn.ensemble import RandomForestRegressor
RFReg = RandomForestRegressor(n_estimators = 500, random_state = 1, n_jobs = -1) 

param_grid = { 
    'max_features' : ["auto", "sqrt", "log2"],
    'min_samples_split' : np.linspace(0.1, 1.0, 10),
     'max_depth' : [x for x in range(1,20)]


from sklearn.model_selection import RandomizedSearchCV
CV_rfc = RandomizedSearchCV(estimator=RFReg, param_distributions =param_grid, n_jobs = -1, cv= 10, n_iter = 50)
CV_rfc.fit(X_train, y_train)

所以我得到了如下的最佳超参数

CV_rfc.best_params_
#{'min_samples_split': 0.1, 'max_features': 'auto', 'max_depth': 18}
CV_rfc.best_score_
#0.8021713812777814

所以我训练了一个新的模型，其超参数如下所示

#1. import the class/model
from sklearn.ensemble import RandomForestRegressor
#2. Instantiate the estimator
RFReg = RandomForestRegressor(n_estimators = 500, random_state = 1, n_jobs = -1, min_samples_split = 0.1, max_features = 'auto', max_depth = 18) 
#3. Fit the model with data aka model training
RFReg.fit(X_train, y_train)

#4. Predict the response for a new observation
y_pred = RFReg.predict(X_test)


y_pred_train = RFReg.predict(X_train)

迭代2的结果

{'RMSE Test': 3.2836794902147926, 'RMSE Train': 2.71230367772569}

迭代3：我使用GridSearchCV来获得超参数的最佳值

from sklearn.ensemble import RandomForestRegressor
RFReg = RandomForestRegressor(n_estimators = 500, random_state = 1, n_jobs = -1) 

param_grid = { 
    'max_features' : ["auto", "sqrt", "log2"],
    'min_samples_split' : np.linspace(0.1, 1.0, 10),
     'max_depth' : [x for x in range(1,20)]

}

from sklearn.model_selection import GridSearchCV
CV_rfc = GridSearchCV(estimator=RFReg, param_grid=param_grid, cv= 10, n_jobs = -1)
CV_rfc.fit(X_train, y_train)

所以我得到了如下的最佳超参数

CV_rfc.best_params_
#{'max_depth': 12, 'max_features': 'auto', 'min_samples_split': 0.1}
CV_rfc.best_score_
#0.8021820114800677

迭代3的结果

{'RMSE Test': 3.283690568225705, 'RMSE Train': 2.712331014201783}

我的函数求值RMSE

def model_evaluate(y_train, y_test, y_pred, y_pred_train):
    metrics = {}
    #RMSE Test
    rmse_test = np.sqrt(mean_squared_error(y_test, y_pred))
    #RMSE Train
    rmse_train = np.sqrt(mean_squared_error(y_train, y_pred_train))

    metrics = {
              'RMSE Test': rmse_test,
              'RMSE Train': rmse_train}

    return metrics 

所以在3次迭代之后，我有以下问题

1. 为什么在我使用RandomSearchCV和GridSearchCV时，调谐的模型的结果比带默认参数的模型差。理想情况下，当使用交叉验证进行调整时，模型应该会给出良好的结果
2. 我知道交叉验证只会发生在param_grid中的值的组合。可能有一些值是好的，但不包括在我的param_grid中。那么我该如何处理这种情况呢
3. 我如何决定我应该为max_features、min_samples_split、max_depth或机器学习模型中的任何超参数来提高其准确性（这样我至少可以得到比默认超参数模型更好的优化模型）的值

Answer 1

Why are the results of tuned model worst than the model with default parameters even when I am using RandomSearchCV and GridSearchCV. Ideally the model should give good results when tuned with cross-validation

你的第二个问题回答了你的第一个问题，但我试图在波士顿数据集上重现你的结果，我得到了{'test_rmse':3.987, 'train_rmse':1.442}和默认参数，{'test_rmse':3.98, 'train_rmse':3.426}是随机搜索的“优化”参数，而{}是网格搜索。然后我使用hyperopt和以下参数空间

 {'max_depth': hp.choice('max_depth', range(1, 100)),
    'max_features': hp.choice('max_features', range(1, x_train.shape[1])),
    'min_samples_split': hp.uniform('min_samples_split', 0.1, 1)}

在大约200次测试后，结果是这样的， 所以我把这个空间扩大到'min_samples_split', 0.01, 1，这使我得到了{'test_rmse':3.278, 'train_rmse':1.716}的最佳结果，min_samples_split等于0.01。根据文献资料，min_samples_split的公式是ceil(min_samples_split * n_samples)，在我们的例子中给出了{}=34，对于这样一个小的数据集来说可能是很大的。在

I know that cross-validation will take place only for the combination of values present in param_grid.There could be values which are good but not included in my param_grid. So how do I deal with this kind of situation

How do I decide what range of values I should try for max_features, min_samples_split, max_depth or for that matter any hyper-parameters in a machine learning model to increase its accuracy.(So that I can atleast get a better tuned model than the model with default hyper-parameters)

你不可能事先知道这一点，所以你必须对每个算法进行研究，看看通常会搜索到什么样的参数空间（这方面的好来源是kaggle，例如googlekaggle kernel random forest），合并它们，考虑您的数据集特性，并使用某种Bayesian Optimization算法（有multiple existing libraries算法）对其进行优化，该算法尝试为新的参数值进行最佳选择。在