spark 1.6ハイブコンテキストsetConf問題

Question

ハイブコンテキストでパーティションテーブルにデータをロードするsqlを実行する際に問題が発生しましたが、私はdynamic partition = trueを設定しましたが、依然として問題が発生しています。

---

SQL： insert overwrite table target_table PARTITION (column1,column2) select * , deletion_flag ,'2018-12-23' as date_feed from source_table

---

ハイブsetconf： -

hiveContext.setConf("hive.exec.dynamic.partition","true") 
    hiveContext.setConf("hive.exec.max.dynamic.partitions","2048") 
    hiveContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict") 

---

エラー：

org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(org.apache.hadoop.fs.Path, java.lang.String, java.util.Map, boolean, int, boolean, boolean, boolean

---

Mavenの依存関係： -

 <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-hive_2.10</artifactId> 
     <version>1.6.0</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.hive</groupId> 
     <artifactId>hive-exec</artifactId> 
     <version>1.1.0</version> 
    </dependency> 

おかげ

Answer 1

私は、ClouderaのレポからすべてのMavenの依存関係を取得した後の問題を解決しました。

<dependencies> 
    <!-- Scala and Spark dependencies --> 

    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.10</artifactId> 
     <version>1.6.0-cdh5.9.2</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.10</artifactId> 
     <version>1.6.0-cdh5.9.2</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-hive_2.10</artifactId> 
     <version>1.6.0-cdh5.9.2</version> 
    </dependency> 
    <!-- https://mvnrepository.com/artifact/org.apache.hive/hive-exec --> 
    <dependency> 
     <groupId>org.apache.hive</groupId> 
     <artifactId>hive-exec</artifactId> 
     <version>1.1.0-cdh5.9.2</version> 
    </dependency> 
    <dependency> 
     <groupId>org.scalatest</groupId> 
     <artifactId>scalatest_2.10</artifactId> 
     <version>3.0.0-SNAP4</version> 
    </dependency> 
    <dependency> 
     <groupId>junit</groupId> 
     <artifactId>junit</artifactId> 
     <version>4.11</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-mllib_2.10</artifactId> 
     <version>1.4.1</version> 
    </dependency> 
    <dependency> 
     <groupId>commons-dbcp</groupId> 
     <artifactId>commons-dbcp</artifactId> 
     <version>1.2.2</version> 
    </dependency> 
    <dependency> 
     <groupId>com.databricks</groupId> 
     <artifactId>spark-csv_2.10</artifactId> 
     <version>1.4.0</version> 
    </dependency> 
    <dependency> 
     <groupId>com.databricks</groupId> 
     <artifactId>spark-xml_2.10</artifactId> 
     <version>0.2.0</version> 
    </dependency> 
    <dependency> 
     <groupId>com.amazonaws</groupId> 
     <artifactId>aws-java-sdk</artifactId> 
     <version>1.0.12</version> 
    </dependency> 
    <dependency> 
     <groupId>com.amazonaws</groupId> 
     <artifactId>aws-java-sdk-s3</artifactId> 
     <version>1.11.172</version> 
    </dependency> 
    <dependency> 
     <groupId>com.github.scopt</groupId> 
     <artifactId>scopt_2.10</artifactId> 
     <version>3.2.0</version> 
    </dependency> 
    <dependency> 
     <groupId>javax.mail</groupId> 
     <artifactId>mail</artifactId> 
     <version>1.4</version> 
    </dependency> 
</dependencies> 
<repositories> 
    <repository> 
     <id>maven-hadoop</id> 
     <name>Hadoop Releases</name> 
     <url>https://repository.cloudera.com/content/repositories/releases/</url> 
    </repository> 
    <repository> 
     <id>cloudera-repos</id> 
     <name>Cloudera Repos</name> 
     <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> 
    </repository> 
</repositories>