ScalaとApache Sparkで2つのDataFramesを結合するには？

Question

は2つのデータフレーム（スカラ、Apacheのスパーク1.6.1）

1）マッチがあり

  MatchID | Player1 | Player2 
     -------------------------------- 
       1 | John Wayne | John Doe 
       2 | Ive Fish | San Simon 

2）個人データの 'BirthYear' が新しいデータフレームを作成することができますどのように

   Player  | BirthYear 
       -------------------------------- 
       John Wayne | 1986 
       Ive Fish | 1990 
       San Simon | 1974 
       john Doe | 1995 

両方のプレーヤーのために

  MatchID | Player1 | Player2 | BYear_P1 |BYear_P2 | Diff 
     ------------------------------------------------------------- 
       1 | John Wayne | John Doe | 1986 | 1995 | 9 
       2 | Ive Fish | San Simon | 1990 | 1974 | 16 

？

は私が

val df = MatchesDF.join(PersonalDF, MatchesDF("Player1") === PersonalDF("Player")) 

が第2のプレーヤー

val resDf = df.join(PersonalDF, df("Player2") === PersonalDF("Player")) 

のために再び参加しようとしたが、それは非常に時間のかかる作業です。

ScalaとApache Sparkで別の方法がありますか？

Answer 1

これは、より良い実行する必要があります。

case class Match(matchId: Int, player1: String, player2: String) 
case class Player(name: String, birthYear: Int) 

val matches = Seq(
    Match(1, "John Wayne", "John Doe"), 
    Match(2, "Ive Fish", "San Simon") 
) 

val players = Seq(
    Player("John Wayne", 1986), 
    Player("Ive Fish", 1990), 
    Player("San Simon", 1974), 
    Player("John Doe", 1995) 
) 

val matchesDf = sqlContext.createDataFrame(matches) 
val playersDf = sqlContext.createDataFrame(players) 

matchesDf.registerTempTable("matches") 
playersDf.registerTempTable("players") 

sqlContext.sql(
    "select matchId, player1, player2, p1.birthYear, p2.birthYear, abs(p1.birthYear-p2.birthYear) " + 
    "from matches m inner join players p1 inner join players p2 " + 
    "where m.player1 = p1.name and m.player2 = p2.name").show() 

+-------+----------+---------+---------+---------+---+ 
|matchId| player1| player2|birthYear|birthYear|_c5| 
+-------+----------+---------+---------+---------+---+ 
|  1|John Wayne| John Doe|  1986|  1995| 9| 
|  2| Ive Fish|San Simon|  1990|  1974| 16| 
+-------+----------+---------+---------+---------+---+ 

私はScalaのDSLで3つの表の結合表現する方法を見つけることができませんでした。

Answer 2

val df = left.join(right, Seq("name")) 
display(df) 

Answer 3

これは、スパークのデータフレーム機能を使用してソリューションです：

import sqlContext.implicits._ 
import org.apache.spark.sql.Row 
import org.apache.spark.sql.functions.abs 

val matches = sqlContext.sparkContext.parallelize(Row(1, "John Wayne", "John Doe"), Row(2, "Ive Fish", "San Simon"))) 

val players = sqlContext.sparkContext.parallelize(Seq(
    Row("John Wayne", 1986), 
    Row("Ive Fish", 1990), 
    Row("San Simon", 1974), 
    Row("John Doe", 1995) 
)) 

val matchesDf = sqlContext.createDataFrame(matches, StructType(Seq(
    StructField("matchId", IntegerType, nullable = false), 
    StructField("player1", StringType, nullable = false), 
    StructField("player2", StringType, nullable = false))) 
).as('matches) 

val playersDf = sqlContext.createDataFrame(players, StructType(Seq(
    StructField("player", StringType, nullable = false), 
    StructField("birthYear", IntegerType, nullable = false) 
))).as('players) 

matchesDf 
    .join(playersDf, $"matches.player1" === $"players.player") 
    .select($"matches.matchId" as "matchId", $"matches.player1" as "player1", $"matches.player2" as "player2", $"players.birthYear" as "player1BirthYear") 
    .join(playersDf, $"player2" === $"players.player") 
    .select($"matchId" as "MatchID", $"player1" as "Player1", $"player2" as "Player2", $"player1BirthYear" as "BYear_P1", $"players.birthYear" as "BYear_P2") 
    .withColumn("Diff", abs('BYear_P2.minus('BYear_P1))) 
    .show() 

+-------+----------+---------+--------+--------+----+ 
|MatchID| Player1| Player2|BYear_P1|BYear_P2|Diff| 
+-------+----------+---------+--------+--------+----+ 
|  1|John Wayne| John Doe| 1986| 1995| 9| 
|  2| Ive Fish|San Simon| 1990| 1974| 16| 
+-------+----------+---------+--------+--------+----+