スパークjava.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;ローカルクラスに互換性がありません

Question

私はSparkアプリケーションをリモートクラスタ上で実行しようとしていますが、シリアル化エラーが発生しています。 ScalaとSparkのバージョンは同じです。私はこの時点で立ち往生しています。

クラスタ上で火花シェル-version：

root@a913008dd071:/usr/local/spark-2.1.1# ./bin/spark-submit --version 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.1.1 
     /_/ 

Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131 
Branch 
Compiled by user jenkins on 2017-04-25T23:51:10Z 
Revision 
Url 
Type --help for more information. 

build.sbt

import sbt.ExclusionRule 

name := "hxfa" 
version := "1.0" 
scalaVersion := "2.11.8" 

val elasticVersion = "5.4.1" 

resolvers += "Spark Packages" at "https://dl.bintray.com/spark-packages/maven/" 
resolvers += "Additional spark packages" at "https://dl.bintray.com/sbcd90/org.apache.spark" 
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases" 
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/" 
resolvers += "Spring Plugins" at "http://repo.spring.io/plugins-release/" 

/* Dependencies */ 
libraryDependencies ++= Seq(
    // Framework and configuration 
    "org.springframework.boot" % "spring-boot-starter-web" % "1.5.4.RELEASE", 
    "org.hibernate" % "hibernate-validator" % "5.2.4.Final", 

    /* Serializations */ 
    "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7", 
    "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7", 
    "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7", 
    "com.esotericsoftware" % "kryo" % "4.0.0", 


    // Spark and utilities 
    "org.apache.spark" %% "spark-core" % "2.1.0", 
    "org.apache.spark" %% "spark-sql" % "2.1.0" , 
    "org.apache.spark" %% "spark-mllib" % "2.1.0" , 
    "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11", 


    // Spark connectors 
    "org.elasticsearch" % "elasticsearch-spark-20_2.11" % elasticVersion, 
    "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0", 


    //JDBC 
    "mysql" % "mysql-connector-java" % "5.1.35", 

    // HBase 
    "org.apache.hbase" % "hbase" % "1.2.4", 
    "org.apache.hbase" % "hbase-client" % "1.2.4", 
    "org.apache.hbase" % "hbase-common" % "1.2.4", 

    // OrientDB 
    "com.orientechnologies" % "orientdb-graphdb" % "2.2.20" 

).map(_.excludeAll(ExclusionRule("org.slf4j", "slf4j-log4j12"), ExclusionRule("log4j", "log4j"))) 

libraryDependencies ++= Seq(
    "org.apache.hbase" % "hbase-server" % "1.2.4" 
).map(_.excludeAll(
    ExclusionRule("com.sun.jersey", "jersey-server"), 
    ExclusionRule("tomcat"), 
    ExclusionRule("log4j", "log4j") 
)) 


/* Assembly  */ 

mainClass in assembly := Some("com.x.x.hello.app.HX") 
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false) 

assemblyMergeStrategy in assembly := { 
    case PathList("META-INF", xs@_*) => MergeStrategy.discard 
    case x => MergeStrategy.first 
} 

スタックトレース：

java.io.InvalidClassException：org.apache.spark .unsafe.types.UTF8String;ローカルクラスが互換性がありません：ストリームclassdesc serialVersionUID = -2992553500466442037、ローカルクラスserialVersionUID = -5670082246090726217 at java.io.ObjectStreamClass.initNonProxy（ObjectStreamClass.java:616）〜[na：1.8.0_45] at java.io.ObjectInputStream.readNonProxyDesc （ObjectInputStream.java：1813）〜[na：1.8.0_45] at java.io.ObjectInputStream.readClassDesc（ObjectInputStream.java:1713）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream .java：2000）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java ：2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] at scala.collection.immutable.List $ SerializationProxy.readObject（List.scala：479）〜[scala-library-2.11。 8.jar：1.0.0-M1]sun.reflect.GeneratedMethodAccessor4.invoke（不明なソース）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8 .0_45] at java.lang.reflect.Mavod.invoke（Method.java:498）〜[na：1.8.0_45] at java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass.java:1058）〜[なし：1.8 .0_45] at java .io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io .ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45]のjava.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream .readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0 （ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.re [ObjectInputStream.java：2169]〜[na：1.8.0_45] のjava.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ ObjectInputStream.java：1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] scala.collection.immutable.List $ SerializationProxyにあります。 [不明なソース]〜[na：na] at sun.reflectでreadObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] sun.reflect.GeneratedMethodAccessor4.invoke .DelegatingMethodAccessorImpl。（java.lang.reflect.Method.invoke（Method.java:498）~~ [na：1.8.0_45] at java.io.ObjectStreamClass）を呼び出す（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] ObjectInputStream.readSerialData（ObjectInputStream.java:2136）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（）{0：0}〜[na：1.8.0_45]の場合は、 ObjectInputStream.java：2047〜[na：1.8.0_45] のjava.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] のjava.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java： ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2169）〜[na：1.8.0_45] の場合は、 2027 ）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜 [na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na ：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8 .0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[なし：1.8.0_45 ]（ObjectInputStream.java：1545）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] atat java.io.ObjectInputStream.readObject0 java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java。 io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io. ObjectInputStream.readSerialData（ObjectInputStream.java：2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] （ObjectInputStream.java：1545）〜[na：1.8.0_45] のjava.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java .io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io .ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45]のjava.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream .readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45]at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInput Stream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream。 readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] のjava.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ ObjectInputStream.java：1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] scala.collection.immutable.List $ SerializationProxyにあります。 [不明なソース]〜[na：na] at sun.reflectでreadObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] sun.reflect.GeneratedMethodAccessor4.invoke .DelegatingMethodAccessorImpl。（java.lang.reflect.Method.invoke（Method.java:498）~~ [na：1.8.0_45] at java.io.ObjectStreamClass）を呼び出す（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] ObjectInputStream.readSerialData（ObjectInputStream.java:2136）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（）{0：0}〜[na：1.8.0_45]の場合は、 ObjectInputStream.java：2047〜[na：1.8.0_45] のjava.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] のjava.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java： ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2169）〜[na：1.8.0_45] の場合は、 2027 ）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜 [na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na ：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8 .0_45] at org.apache.spark.serializer.JavaDeserializationStream.readObject（JavaSerializer.scala：75）〜[spark-core_2.11-2.1.0.jar：2.1.0] at org.apache.spark.serializer .JavaSerializerInstance.deserialize（J spac-core_2.1.0.jar：2.1.0] at org.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：80）〜[spark-core_2。 11.2.1.0.jar：2.1.0] at org.apache.spark.scheduler.Task.run（Task.scala：99）〜[spark-core_2.11-2.1.0.jar：2.1.0] at org.apache.spark.executor.Executor $ TaskRunner.run（Executor.scala：322）〜[spark-core_2.11-2.1.0.jar：2.1.0] at java.util.concurrent.ThreadPoolExecutor。 runWorker（ThreadPoolExecutor.java:1142）〜[na：1.8.0_45] at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）〜[na：1.8.0_45] at java.lang。 Thread.run（Thread.java:748）〜[na：1.8.0_45]

Answer 1

spark-subm itversionはあなたのシステムのスカラとスカラバージョンを表示しますが、あなたのsbtはシステムのスカラバージョンを使用しています。だから、

"org.apache.spark" % "spark-core_2.11" % "2.1.1", 
    "org.apache.spark" % "spark-sql_2.11" % "2.1.1" , 
    "org.apache.spark" % "spark-mllib_2.11" % "2.1.1" , 

そして、それが解決しない場合に

"org.apache.spark" %% "spark-core" % "2.1.0", 
    "org.apache.spark" %% "spark-sql" % "2.1.0" , 
    "org.apache.spark" %% "spark-mllib" % "2.1.0" , 

を変更してください、あなたの質問では、アプリケーションとリモートマシンのを提出しているか、お使いのシステムのScalaバージョンに更新してくださいスカラとスパークのバージョン。