0
私はSpark 2.1.0 Kmeans - Clusteringアルゴリズムを使用していました。Spark 2.1.0 - SparkML要件が失敗しました
public class ClusteringTest {
public static void main(String[] args) {
SparkSession session = SparkSession.builder()
.appName("Clustering Test")
.config("spark.master", "local")
.getOrCreate();
session.sparkContext().setLogLevel("ERROR");
List<Row> rawDataTraining = Arrays.asList(
RowFactory.create(1.0,Vectors.dense(1.0, 1.0, 1.0).toSparse()),
RowFactory.create(1.0,Vectors.dense(2.0, 2.0, 2.0).toSparse()),
RowFactory.create(1.0,Vectors.dense(3.0, 3.0, 3.0).toSparse()),
RowFactory.create(2.0,Vectors.dense(6.0, 6.0, 6.0).toSparse()),
RowFactory.create(2.0,Vectors.dense(7.0, 7.0, 7.0).toSparse()),
RowFactory.create(2.0,Vectors.dense(8.0, 8.0,8.0).toSparse()),
//...
StructType schema = new StructType(new StructField[]{
new StructField("label", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("features", new VectorUDT(), false, Metadata.empty())
});
Dataset<Row> myRawData = session.createDataFrame(rawDataTraining, schema);
Dataset<Row>[] splits = myRawData.randomSplit(new double[]{0.75, 0.25});
Dataset<Row> trainingData = splits[0];
Dataset<Row> testData = splits[1];
//Train Kmeans
KMeans kMeans = new KMeans().setK(3).setSeed(100);
KMeansModel kMeansModel = kMeans.fit(trainingData);
Dataset<Row> predictions = kMeansModel.transform(testData);
predictions.show(false);
MulticlassClassificationEvaluator evaluator = new MulticlassClassificationEvaluator().setLabelCol("label")
.setPredictionCol("prediction")
.setMetricName("accuracy");
double accuracy = evaluator.evaluate(predictions);
System.out.println("accuracy" + accuracy);
}
}
コンソール出力は次のとおり
+-----+----------------------------+----------+
|label|features |prediction|
+-----+----------------------------+----------+
|2.0 |(3,[0,1,2],[7.0,7.0,7.0]) |2 |
|3.0 |(3,[0,1,2],[11.0,11.0,11.0])|2 |
|3.0 |(3,[0,1,2],[12.0,12.0,12.0])|1 |
|3.0 |(3,[0,1,2],[13.0,13.0,13.0])|1 |
+-----+----------------------------+----------+
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Column prediction must be of type DoubleType but was actually IntegerType.
at scala.Predef$.require(Predef.scala:233)
at org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:42)
at org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator.evaluate(MulticlassClassificationEvaluator.scala:75)
at ClusteringTest.main(ClusteringTest.java:84)
Process finished with exit code 1
uが見ることができるように、予測結果は整数です。しかし、MulticlassClassificationEvalutorを使用するには、これらの予測結果をDoubleに変換する必要があります。どうしたらいいですか?