0
私のプロジェクトはmaven + intellijです。 そして私はWindowsシステムで開発する。 まず私は上のクラスSQLContextを取得し、2.12.2.For最新のScalaのlibrayバージョンを使用し、私は火花jarをインポートする必要があります。 
警告：複数バージョンのスカラーライブラリが検出されましたか？
その後私は、私は、このスパークjarファイルを使用する場合と言われました、私はScalaのバージョンを下げて、だから私は、これが得たmvn_clean_install.Iとき、私は今、libirayを削除し、2.10に変更....しかし：
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.5.0 requires scala version: 2.11.7
[WARNING] com.typesafe.akka:akka-remote_2.11:2.3.11 requires scala version: 2.11.7
[WARNING] com.typesafe.akka:akka-actor_2.11:2.3.11 requires scala version: 2.11.7
[WARNING] com.typesafe.akka:akka-slf4j_2.11:2.3.11 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:1.6.1 requires scala version: 2.11.7
[WARNING] org.json4s:json4s-jackson_2.11:3.2.10 requires scala version: 2.11.7
[WARNING] org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.7
[WARNING] org.json4s:json4s-ast_2.11:3.2.10 requires scala version: 2.11.7
[WARNING] org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] E:\...\src\main\scala:-1: info: compiling
[INFO] Compiling 4 source files to E:\...\target\classes at 1491813951772
[ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:2: error: object sql is not a member of package org.apache.spark
[ERROR] import org.apache.spark.sql.SQLContext
[ERROR] ^
[ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:15: error: not found: type SQLContext
[ERROR] val sqlContext = new SQLContext(sc)
[ERROR] ^
[ERROR] E:\...\mapreduce\ConvertToParquetFormat.scala:24: error: value toDF is not a member of org.apache.spark.rdd.RDD[....qubole.mapreduce.ConvertToParquetFormat.OmnitureHit]
[ERROR] possible cause: maybe a semicolon is missing before `value toDF'?
[ERROR] .toDF().write.parquet ("file:///C:/Users/Desktop/456")
[ERROR] ^
[ERROR] three errors found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:00 min
[INFO] Finished at: 2017-04-10T16:45:58+08:00
[INFO] Final Memory: 33M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project packages-omniture-qubole-mapreduce: wrap: org.apache.commons.exec.ExecuteException: Process exited with an er
ror: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
E:\github\mia\packages-omniture-qubole-mapreduce>
ので、私はScalaの10バージョンを削除し、それが私に言いました私は1つのスカラーライブラリを持っていません。スカラーを追加します2.11。いくつかの瓶に2.11バージョンが必要なように見えるので、Multiple versions of scala libraries detected?
しかし、SQLContextという単語をctrl + left_cilckすると、ページに行くことができますが、それは匿名です。
： が、これは私の左の依存リストである
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company.www</groupId>
<artifactId>packages-omniture-qubole-mapreduce</artifactId>
<version>1.0-SNAPSHOT</version>
<parent>
<groupId>com.company.www.platform</groupId>
<artifactId>platform-parent-spark</artifactId>
<version>0.1.41</version>
</parent>
<properties>
<spark.mapreduce.mainclass>com.company.www.packages.omniture.qubole.mapreduce.SampleMapReduceJob</spark.mapreduce.mainclass>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.major.minor.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.company.www.commons</groupId>
<artifactId>commons-spark</artifactId>
<version>[1.0.17, ]</version>
</dependency>
<dependency>
<groupId>com.company.www</groupId>
<artifactId>exp-user-interaction-messages-v1</artifactId>
<version>[1.4,]</version>
</dependency>
<dependency>
<groupId>org.scalaj</groupId>
<artifactId>scalaj-http_${scala.major.minor.version}</artifactId>
<version>1.1.4</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3</version>
</dependency>
<dependency>
<groupId>org.parboiled</groupId>
<artifactId>parboiled-java</artifactId>
<version>1.0.2</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>packages-omniture-qubole-mapreduce</finalName>
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
<resource>log4j.properties</resource>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${spark.mapreduce.mainclass}</mainClass>
</transformer>
</transformers>
<relocations>
<relocation>
<pattern>org.eclipse.jetty</pattern>
<shadedPattern>org.spark-project.jetty</shadedPattern>
<includes>
<include>org.eclipse.jetty.**</include>
</includes>
</relocation>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>org.spark-project.guava</shadedPattern>
<excludes>
<exclude>com/google/common/base/Absent*</exclude>
<exclude>com/google/common/base/Function</exclude>
<exclude>com/google/common/base/Optional*</exclude>
<exclude>com/google/common/base/Present*</exclude>
<exclude>com/google/common/base/Supplier</exclude>
</excludes>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
POMファイルを投稿してください。 –
私はpomを追加しました、plsは私を助けます.. – daxue