sparkデータフレームの列をsparkデータフレーム（Pysparkを使用）に追加するにはどうすればよいですか？

Question

私は2つのスパークデータフレームを持っています。あるスパークデータフレームから別のスパークデータフレームにカラムを追加したいと思います。

私のコードは次のとおりです。ここで

new = df.withColumn("prob", tr_df.prob) 

が、私は私のデータフレームPROBとして名前DFにtr_dfである列結果2を追加します。私は、この検索が、何も私のために働いていないと私はerror--

を取得しています

AnalysisException: u'resolved attribute(s) prob#579 missing from q1_n_words#388L,prediction#510,res1#390,q2_n_words#389L,tfidf_word_match#384,Average#379,prob#385,probability#485,Cosine#381,word_m#383,rawPrediction#461,features#438,res2#391,question1#373,Jaccard#382,test_id#372L,raw_pred#377,question2#374,q2len#376,Common#378L,result2#387,q1len#375,result1#386,Percentage#380 in operator !Project [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#579 AS prob#634, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391, features#438, rawPrediction#461, probability#485, prediction#510];;\n!Project [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#579 AS prob#634, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391, features#438, rawPrediction#461, probability#485, prediction#510]\n+- Project [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#385, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391, features#438, rawPrediction#461, probability#485, UDF(rawPrediction#461) AS prediction#510]\n +- Project [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#385, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391, features#438, rawPrediction#461, UDF(rawPrediction#461) AS probability#485]\n  +- Project [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#385, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391, features#438, UDF(features#438) AS rawPrediction#461]\n   +- Project [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#385, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391, UDF(struct(q1len#375, q2len#376, cast(q1_n_words#388L as double) AS q1_n_words_double_VectorAssembler_4158baa8e5b4f3aced2b#435, cast(q2_n_words#389L as double) AS q2_n_words_double_VectorAssembler_4158baa8e5b4f3aced2b#436, cast(Common#378L as double) AS Common_double_VectorAssembler_4158baa8e5b4f3aced2b#437, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, prob#385, raw_pred#377, res1#390, res2#391)) AS features#438]\n   +- LogicalRDD [test_id#372L, question1#373, question2#374, q1len#375, q2len#376, raw_pred#377, Common#378L, Average#379, Percentage#380, Cosine#381, Jaccard#382, word_m#383, tfidf_word_match#384, prob#385, result1#386, result2#387, q1_n_words#388L, q2_n_words#389L, res1#390, res2#391]\n' 

tr_dfスキーマ -

tr_df.printSchema() 
root 
|-- prob: float (nullable = true) 

DFスキーマ -

df.printSchema() 
root 
|-- test_id: long (nullable = true) 

助けてください！前もって感謝します。

Answer 1

エラーメッセージが明確にあなたがあなたが以下のように同じ何かを設定することができ、あなたのスパーク構成

に=真spark.sql.crossJoin.enabledを設定する必要が述べたよう：次に取得または

val sparkConf = new SparkConf().setAppName("Test") 
sparkConf.set("spark.sql.crossJoin.enabled", "true") 

を

...あなたの参加を行う次に、このSparkConf

に

val sparkSession = SparkSession.builder().config(sparkConf).getOrCreate() 

を渡すことによってSparkSessionを作成

ソース：How to enable Cartesian join in Spark 2.0?