sklearn聚类：计算TFIDFweigthed d d上的轮廓系数

Question

我想像scikit学习示例silhouette_analysis那样计算剪影得分。在

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer(use_idf=True)
sampleText = []
sampleText.append("Some text for document clustering")
tfidf_matrix = tfidf_vectorizer.fit_transform(sampleText)

如何转换tfidf_矩阵以执行以下操作：

^{pr2}$

Answer 1

tf-idf是多维的，必须减少到二维。这可以通过将tf-idf减少到方差最大的两个特性来实现。我用PCA来减少tf idf。完整的例子：

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer(use_idf=True)
sampleText = []
sampleText.append("Some text for document clustering")
tfidf_matrix = tfidf_vectorizer.fit_transform(sampleText)
X = tfidf_vectorizer.fit_transform(jobDescriptions).todense()

from sklearn.decomposition import PCA
pca = PCA(n_components=2).fit(X)
data2D = pca.transform(X)  

import matplotlib.cm as cm
from sklearn.metrics import silhouette_samples, silhouette_score
import matplotlib.pyplot as plt


for num_clusters in range(2,6):
# Create a subplot with 1 row and 2 columns
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.set_size_inches(18, 7)

# The 1st subplot is the silhouette plot
# The silhouette coefficient can range from -1, 1 but in this example all
# lie within [-0.1, 1]
ax1.set_xlim([-0.1, 1])
# The (n_clusters+1)*10 is for inserting blank space between silhouette
# plots of individual clusters, to demarcate them clearly.
ax1.set_ylim([0, len(data2D) + (num_clusters + 1) * 10])

km = KMeans(n_clusters=num_clusters,
            n_init=10,                        # number of iterations with different seeds
            random_state=1                    # fixes the seed 
           )

cluster_labels = km.fit_predict(data2D)

# The silhouette_score gives the average value for all the samples.
# This gives a perspective into the density and separation of the formed
# clusters
 silhouette_avg = silhouette_score(data2D, cluster_labels)