列単位でデータをグループ化し、グループごとの観測数を計算する方法

Question

このDataFrameをdfに、id,typeおよびactivityの3つの列で表示する方法を教えてください。

val myData = (Seq(("aa1", "GROUP_A", "10"),("aa1","GROUP_A", "12"),("aa2","GROUP_A", "hy"),("aa2", "GROUP_B", "14"), 
       ("aa3","GROUP_B", "11"),("aa3","GROUP_B","12"),("aa2", "GROUP_3", "12")) 

val df = sc.parallelize(myData).toDF() 

私はtypeによってデータをグループ化する必要があるし、各idのための活動の数を計算します。これは期待された結果である：

type  id count 
GROUP_A aa1 2 
GROUP_A aa2 1 
GROUP_B aa3 3 
GROUP_B aa2 1 

これは私が試したものです：

df.groupBy("type","id").count().sort("count").show() 

それが正しい結果を与えるものではありませんが。

Answer 1

私は最小限のサンプル・データを変更し、それが私の作品：

//yours 
val myData = (Seq(("aa1", "GROUP_A", "10"),("aa1","GROUP_A", "12"),("aa2","GROUP_A", "hy"),("aa2", "GROUP_B", "14"),("aa3","GROUP_B", "11"),("aa3","GROUP_B","12"),("aa2", "GROUP_3", "12")) 

//mine 
//removed the (at the beginning 
//changed GROUP_3 to GROUP_B 
//other minor changes so that the resultant group by will look like you desired 
val myData = Seq(("aa1", "GROUP_A", "10"),("aa1","GROUP_A", "12"),("aa2","GROUP_A", "12"),("aa3", "GROUP_B", "14"),("aa3","GROUP_B", "11"),("aa3","GROUP_B","12"),("aa2", "GROUP_B", "12")) 


//yours 
val df = sc.parallelize(myData).toDF() 
//mine 
//added in column names 

val df = sc.parallelize(myData).toDF("id","type","count") 

df.groupBy("type","id").count.show 
+-------+---+-----+ 
| type| id|count| 
+-------+---+-----+ 
|GROUP_A|aa1| 2| 
|GROUP_A|aa2| 1| 
|GROUP_B|aa2| 1| 
|GROUP_B|aa3| 3| 
+-------+---+-----+ 

は、私が逃した何かがありますか？

Answer 2

dataframeを作成してgrouped dataにカウントすると、column namesを定義できます。これは簡単です

import sqlContext.implicits._ 

val myData = Seq(("aa1", "GROUP_A", "10"), 
    ("aa1","GROUP_A", "12"), 
    ("aa2","GROUP_A", "hy"), 
    ("aa2", "GROUP_B", "14"), 
    ("aa3","GROUP_B", "11"), 
    ("aa3","GROUP_B","12"), 
    ("aa3", "GROUP_B", "12")) 

val df = sc.parallelize(myData).toDF("id", "type", "activity") 
df.groupBy("type","id").count().sort("count").show()