scikitlearn SelectPercentile TFIDF数据功能缩减

Question

我使用scikit learn中的各种机制来创建训练数据集和由文本特性组成的测试集的tf-idf表示。两个数据集都经过预处理以使用相同的词汇表，因此特征和特征的数量是相同的。我可以在训练数据上创建一个模型，并在测试数据上评估它的性能。我想知道如果我使用SelectPercentile来减少转换后训练集中的特征的数量，那么如何在测试集中识别相同的特征以用于预测？在

trainDenseData = trainTransformedData.toarray()
testDenseData = testTransformedData.toarray()

if ( useFeatureReduction== True):
    reducedTrainData = SelectPercentile(f_regression,percentile=10).fit_transform(trainDenseData,trainYarray)

clf.fit(reducedTrainData, trainYarray)


# apply feature reduction to the test data

Answer 1

您应该存储SelectPercentile对象，并使用它来transform测试数据：

select = SelectPercentile(f_regression,percentile=10)
reducedTrainData = select.fit_transform(trainDenseData,trainYarray)
reducedTestData = select.transform(testDenseData)

Answer 2

请参阅下面的代码和注释。在

import numpy as np

from sklearn.datasets import make_classification
from sklearn import feature_selection

# Build a classification task using 3 informative features
X, y = make_classification(n_samples=1000,
                           n_features=10,
                           n_informative=3,
                           n_redundant=0,
                           n_repeated=0,
                           n_classes=2,
                           random_state=0,
                           shuffle=False)

sp = feature_selection.SelectPercentile(feature_selection.f_regression, percentile=30)
sp.fit_transform(X[:-1], y[:-1])  #here, training are the first 9 data vectors, and the last one is the test set
idx = np.arange(0, X.shape[1])  #create an index array
features_to_keep = idx[sp.get_support() == True]  #get index positions of kept features

x_fs = X[:,features_to_keep] #prune X data vectors
x_test_fs = x_fs[-1] #take your last data vector (the test set) pruned values
print x_test_fs #these are your pruned test set values