Pythonでテキスト分類パイプラインのPMMLを生成

Question

テキスト分類パイプラインのPMML（jpmml-sklearnを使用）を生成しようとしています。コードの最後の行 - sklearn2pmml（Textpipeline、 "TextMiningClassifier.pmml"、with_repr = True） - クラッシュします。

from sklearn.datasets import fetch_20newsgroups 
from sklearn.feature_extraction.text import CountVectorizer 
from sklearn.feature_extraction.text import TfidfTransformer 
from sklearn.linear_model import SGDClassifier 
from sklearn2pmml import PMMLPipeline 

categories = [ 
'alt.atheism', 
'talk.religion.misc', 
] 

print("Loading 20 newsgroups dataset for categories:") 
print(categories) 
data = fetch_20newsgroups(subset='train', categories=categories) 
print("%d documents" % len(data.filenames)) 
print("%d categories" % len(data.target_names)) 

Textpipeline = PMMLPipeline([ 
('vect', CountVectorizer()), 
('tfidf', TfidfTransformer()), 
('clf', SGDClassifier()), 
]) 

Textpipeline.fit(data.data, data.target) 

from sklearn2pmml import sklearn2pmml 

sklearn2pmml(Textpipeline, "TextMiningClassifier.pmml", with_repr = True) 

sklearn2pmml（）は入力としてTextpipelineを取ることができません。このコードは他のパイプライン（ここでは例：https://github.com/jpmml/sklearn2pmml）では正常に機能しますが、上記のテキスト分類パイプラインでは機能しません。だから私の質問は：どのように私はテキスト分類の問題のPMMLを生成するのですか？

エラーは、私が取得：

Jun 15, 2017 12:48:00 PM org.jpmml.sklearn.Main run 
INFO: Parsing PKL.. 
Jun 15, 2017 12:48:01 PM org.jpmml.sklearn.Main run 
INFO: Parsed PKL in 489 ms. 
Jun 15, 2017 12:48:01 PM org.jpmml.sklearn.Main run 
INFO: Converting.. 
Jun 15, 2017 12:48:01 PM sklearn2pmml.PMMLPipeline encodePMML 
WARNING: The 'target_field' attribute is not set. Assuming y as the name of the target field 
Jun 15, 2017 12:48:01 PM sklearn2pmml.PMMLPipeline initFeatures 
WARNING: The 'active_fields' attribute is not set. Assuming [x1] as the names of active fields 
Jun 15, 2017 12:48:01 PM org.jpmml.sklearn.Main run 
SEVERE: Failed to convert 
java.lang.IllegalArgumentException: The tokenizer object (null) is not Splitter 
at sklearn.feature_extraction.text.CountVectorizer.getTokenizer(CountVectorizer.java:263) 
at sklearn.feature_extraction.text.CountVectorizer.encodeDefineFunction(CountVectorizer.java:164) 
at sklearn.feature_extraction.text.CountVectorizer.encodeFeatures(CountVectorizer.java:124) 
at sklearn.pipeline.Pipeline.encodeFeatures(Pipeline.java:93) 
at sklearn2pmml.PMMLPipeline.encodePMML(PMMLPipeline.java:122) 
at org.jpmml.sklearn.Main.run(Main.java:144) 
at org.jpmml.sklearn.Main.main(Main.java:93) 

Exception in thread "main" java.lang.IllegalArgumentException: The tokenizer object (null) is not Splitter 
at sklearn.feature_extraction.text.CountVectorizer.getTokenizer(CountVectorizer.java:263) 
at sklearn.feature_extraction.text.CountVectorizer.encodeDefineFunction(CountVectorizer.java:164) 
at sklearn.feature_extraction.text.CountVectorizer.encodeFeatures(CountVectorizer.java:124) 
at sklearn.pipeline.Pipeline.encodeFeatures(Pipeline.java:93) 
at sklearn2pmml.PMMLPipeline.encodePMML(PMMLPipeline.java:122) 
at org.jpmml.sklearn.Main.run(Main.java:144) 
at org.jpmml.sklearn.Main.main(Main.java:93) 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "C:\Data\Anaconda2\lib\site-packages\sklearn2pmml\__init__.py", line 142, in sklearn2pmml 
raise RuntimeError("The JPMML-SkLearn conversion application has failed. The Java process should have printed more information about the failure into its standard output and/or error streams") 
RuntimeError: The JPMML-SkLearn conversion application has failed. The Java process should have printed more information about the failure into its standard output and/or error streams 

Answer 1

あなたはPMML互換のテキストトークン化機能を使用する必要があります。デフォルトの実装では、クラスsklearn2pmml.feature_extraction.text.Splitter次のとおりです。

from sklearn.feature_extraction.text import TfidfVectorizer 
from sklearn2pmml.feature_extraction.text import Splitter 
vectorizer = TfidfVectorizer(analyzer = "word", token_pattern = None, tokenizer = Splitter()) 

詳細や参照がJPMMLメーリングリストで利用できます。https://groups.google.com/forum/#!topic/jpmml/wi-0rxzUn1o