Cassandraのスパークコネクタ - NoSuchMethodError：scala.runtime.ObjectRef.zero（）Lscala /ランタイム/ objectrefに

Question

I am trying to connect spark with cassandra database but I got the error mentioned below. I think there should some mismatch with the versions.

コード：

SparkConf conf = new SparkConf().setAppName("kafka-sandbox").setMaster("local[2]"); 
    conf.set("spark.cassandra.connection.host", "192.168.34.1");//connection for cassandra database 
    JavaSparkContext sc = new JavaSparkContext(conf); 
    CassandraConnector connector = CassandraConnector.apply(sc.getConf()); 
    final Session session = connector.openSession();//error in this line 
    final PreparedStatement prepared = session.prepare("INSERT INTO spark_test5.messages JSON?"); 

---

error: 


    Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.ObjectRef.zero()Lscala/runtime/ObjectRef; 
     at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala) 
     at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:149) 
     at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$3.apply(CassandraConnector.scala:149) 
     at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31) 
     at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56) 
     at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:82) 

---

pom.xml: 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>SparkPoc</groupId> 
    <artifactId>Spark-Poc</artifactId> 
    <version>0.0.1-SNAPSHOT</version> 
    <dependencies> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.10</artifactId> 
     <version>2.0.0</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.10</artifactId> 
     <version>2.0.1</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming-kafka-0-8_2.10</artifactId> 
     <version>2.0.0</version> 
    </dependency> 
    <dependency> 
     <groupId>com.datastax.spark</groupId> 
     <artifactId>spark-cassandra-connector_2.11</artifactId> 
     <version>2.0.0-M3</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.11</artifactId> 
     <version>2.0.1</version> 
    </dependency> 
    </dependencies> 
<build> 
    <plugins> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-compiler-plugin</artifactId> 
     <version>3.3</version> 
     <configuration> 
      <source>1.8</source> 
      <target>1.8</target> 
     </configuration> 
    </plugin> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-assembly-plugin</artifactId> 
     <version>2.4.1</version> 
     <configuration> 
      <!-- get all project dependencies --> 
      <descriptorRefs> 
        <descriptorRef>jar-with-dependencies</descriptorRef> 
      </descriptorRefs> 
      <!-- MainClass in mainfest make a executable jar --> 
      <archive> 
        <manifest> 
          <mainClass>com.nwf.Consumer</mainClass> 
        </manifest> 
      </archive> 
     </configuration> 
     <executions> 
      <execution> 
        <id>make-assembly</id> 
        <!-- bind to the packaging phase --> 
        <phase>package</phase> 
        <goals> 
          <goal>single</goal> 
        </goals> 
      </execution> 
    </executions> 
    </plugin> 
    </plugins> 
</build> 
</project> 

スパークバージョン：バージョン2.0.0

Scalaバージョン：version 2.11.8

Answer 1

：

• 火花streaming_ 2.10

は

v2.10のを参照してください。

• spark-core_ 2.10
• 火花ストリーミング・カフカ-0-8_ 2.10
• 火花カサンドラ・connector_ 2.11
• 火花SQL_ 2.11

すべての依存関係は、同じScalaのバージョンを持っている必要があります。すべてを_2.11に変更してください。

Answer 2

scalar.runtime.VolatileObjectRefのゼロ（）はScala 2.11で導入されました。おそらく、Scala 2.11用にコンパイルされ、Scala 2.10ランタイムで実行されているライブラリがあります。 https://github.com/scala/scala/blob/2.10.x/src/library/scala/runtime/VolatileObjectRef.java V2.11：あなたのpom.xmlあなたが別の依存関係のためのScalaのバージョンを混在によるとhttps://github.com/scala/scala/blob/2.11.x/src/library/scala/runtime/VolatileObjectRef.java

Answer 3

In my pom.xml I changed sacala version from 2.10 to 2.11. 
Given below is the updated pom.xml 


---------- 
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>SparkPoc</groupId> 
    <artifactId>Spark-Poc</artifactId> 
    <version>0.0.1-SNAPSHOT</version> 
    <dependencies> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.11</artifactId> 
     <version>2.0.0</version> 
     <scope>provided</scope> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.11</artifactId> 
     <version>2.0.1</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming-kafka-0-8_2.11</artifactId> 
     <version>2.0.0</version> 
    </dependency> 
    <dependency> 
     <groupId>com.datastax.spark</groupId> 
     <artifactId>spark-cassandra-connector_2.11</artifactId> 
     <version>2.0.0-M3</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.11</artifactId> 
     <version>2.0.1</version> 
    </dependency> 
    </dependencies> 
<build> 
    <plugins> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-compiler-plugin</artifactId> 
     <version>3.3</version> 
     <configuration> 
      <source>1.8</source> 
      <target>1.8</target> 
     </configuration> 
    </plugin> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-assembly-plugin</artifactId> 
     <version>2.4.1</version> 
     <configuration> 
      <!-- get all project dependencies --> 
      <descriptorRefs> 
        <descriptorRef>jar-with-dependencies</descriptorRef> 
      </descriptorRefs> 
      <!-- MainClass in mainfest make a executable jar --> 
      <archive> 
        <manifest> 
          <mainClass>com.nwf.Consumer</mainClass> 
        </manifest> 
      </archive> 
     </configuration> 
     <executions> 
      <execution> 
        <id>make-assembly</id> 
        <!-- bind to the packaging phase --> 
        <phase>package</phase> 
        <goals> 
          <goal>single</goal> 
        </goals> 
      </execution> 
    </executions> 
    </plugin> 
    </plugins> 
</build> 
</project>