spark.createDatasetのヌル

Question

私はテストデータにいくつかのヌル値が必要であるという単体テストを書いています。 タプルにヌルをまっすぐ入れてみましたが、また、オプションを使って試しました。それはうまくいかなかった。ここで

は私のコードです：

import sparkSession.implicits._ 
// Data set with null for even values 
val sampleData = sparkSession.createDataset(Seq(
    (1, Some("Yes"), None), 
    (2, None, None), 
    (3, Some("Okay"), None), 
    (4, None, None))) 
    .toDF("id", "title", "value") 

スタックトレース：あなたがしている場合はnull.asInstanceOf[String]：

None.type (of class scala.reflect.internal.Types$UniqueSingleType) 
scala.MatchError: None.type (of class scala.reflect.internal.Types$UniqueSingleType) 
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor(ScalaReflection.scala:472) 
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:596) 
    at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:587) 
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252) 
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:252) 
    at scala.collection.immutable.List.foreach(List.scala:381) 
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:252) 
    at scala.collection.immutable.List.flatMap(List.scala:344) 
    at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor(ScalaReflection.scala:587) 
    at org.apache.spark.sql.catalyst.ScalaReflection$.serializerFor(ScalaReflection.scala:425) 
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:71) 
    at org.apache.spark.sql.Encoders$.product(Encoders.scala:275) 
    at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:49) 

Answer 1

あなたが代わりにNone

scala> val maybeString = None: Option[String] 
maybeString: Option[String] = None 

scala> val sampleData = spark.createDataset(Seq(
    | (1, Some("Yes"), maybeString), 
    | (2, maybeString, maybeString), 
    | (3, Some("Okay"), maybeString), 
    | (4, maybeString, maybeString))).toDF("id", "title", "value") 
sampleData: org.apache.spark.sql.DataFrame = [id: int, title: string ... 1 more field] 

scala> sampleData.show 
+---+-----+-----+ 
| id|title|value| 
+---+-----+-----+ 
| 1| Yes| null| 
| 2| null| null| 
| 3| Okay| null| 
| 4| null| null| 
+---+-----+-----+ 

Answer 2

か、使用できるのNone: Option[String]を使用する必要があります文字列を扱うだけです

val df1 = sc.parallelize(Seq((1, "Yes", null.asInstanceOf[String]), 
    | (2, null.asInstanceOf[String], null.asInstanceOf[String]), 
    | (3, "Okay", null.asInstanceOf[String]), 
    | (4, null.asInstanceOf[String], null.asInstanceOf[String]))).toDF("id", "title", "value") 
df1: org.apache.spark.sql.DataFrame = [id: int, title: string, value: string] 

scala> df1.show 
+---+-----+-----+ 
| id|title|value| 
+---+-----+-----+ 
| 1| Yes| null| 
| 2| null| null| 
| 3| Okay| null| 
| 4| null| null| 
+---+-----+-----+