1
私はずっと苦労しましたが、scikit-learnパイプラインでFeatureUnionのテキスト機能と一緒に余分な機能を使う方法を理解できませんでした。
私はモデルとテストのデータとしての文のリストを訓練するための文章とそのラベルのリストを持っています。それから、私は袋の言葉に余分な特徴（各文章の長さのような）を加えようとします。このために、長さのリストを返し、列車のリストと同じ数の要素を持つカスタムLengthTransformerを書きました。
次に、私はと組み合わせてFeatureUnionを使用していますが、動作しません。FeatureUnionを使ってscikit-learnパイプラインの単語に余分な機能を追加
は、私がこれまでに思い付いてきたことはこれです：
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn import preprocessing
class LengthTransformer(BaseEstimator, TransformerMixin):
def fit(self, X, y=None):
return self
def transform(self, X):
return [len(x) for x in X]
X_train = ["new york is a hell of a town",
"new york was originally dutch",
"the big apple is great",
"new york is also called the big apple",
"nyc is nice",
"people abbreviate new york city as nyc",
"the capital of great britain is london",
"london is in the uk",
"london is in england",
"london is in great britain",
"it rains a lot in london",
"london hosts the british museum",
"new york is great and so is london",
"i like london better than new york"]
y_train_text = [["new york"], ["new york"], ["new york"], ["new york"], ["new york"],
["new york"], ["london"], ["london"], ["london"], ["london"],
["london"], ["london"], ["london", "new york"], ["new york", "london"]]
X_test = ['nice day in nyc',
'welcome to london',
'london is rainy',
'it is raining in britian',
'it is raining in britian and the big apple',
'it is raining in britian and nyc',
'hello welcome to new york. enjoy it here and london too']
lb = preprocessing.MultiLabelBinarizer()
Y = lb.fit_transform(y_train_text)
classifier = Pipeline([
('feats', FeatureUnion([
('tfidf', TfidfVectorizer()),
('len', LengthTransformer())
])),
('clf', OneVsRestClassifier(LinearSVC()))
])
classifier.fit(X_train, Y)
predicted = classifier.predict(X_test)
all_labels = lb.inverse_transform(predicted)
for item, labels in zip(X_test, all_labels):
print('{} => {}'.format(item, ', '.join(labels)))