スケーリングジョブ実行中にNoSuchMethodErrorが発生する

Question

CDH 4.5で実行中のスケーリングジョブをCDH 5.5.1にアップグレードしようとしています。 ジョブはjson4sを使用してjsonデータを解析します。クラスタでジョブを実行すると、以下のエラーが表示されます。私のローカルマシンでうまく動作します。

私はスカラを使用しています。2.10 &スケーリング0.15.0バージョン。ここで

ERROR [main] cascading.flow.stream.SourceStage: caught throwable 
cascading.pipe.OperatorException: [JsonLine(/data/raw/gon...][read() @ com.xxx.xxx.GJob.<init>(GonzalesJob.scala:15)] operator Each failed executing operation 
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107) 
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39) 
at cascading.flow.stream.SourceStage.map(SourceStage.java:102) 
at cascading.flow.stream.SourceStage.run(SourceStage.java:58) 
at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:130) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase: method <init>(Lcom/fasterxml/jackson/databind/JavaType;)V not found 
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.<init>(UnsortedMapDeserializerModule.scala:39) 
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializerResolver$.findMapLikeDeserializer(UnsortedMapDeserializerModule.scala:97) 
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCustomMapLikeDeserializer(BasicDeserializerFactory.java:1030) 
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createMapLikeDeserializer(BasicDeserializerFactory.java:994) 
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:384) 
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:354) 
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:267) 
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:247) 
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:146) 
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:322) 
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:2990) 
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2884) 
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2041) 
at com.twitter.scalding.JsonLine$$anonfun$transformForRead$1.applyOrElse(JsonLine.scala:70) 
at com.twitter.scalding.JsonLine$$anonfun$transformForRead$1.applyOrElse(JsonLine.scala:68) 
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33) 
at com.twitter.scalding.CollectFunction.operate(Operations.scala:94) 
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99) 
... 11 more 

は私の依存関係

mvn dependency:tree 

--- maven-dependency-plugin:2.8:tree (default-cli) @ --- 
[INFO] bskyb.bigdata:gonzales-etl:jar:1.3 
[INFO] +- org.scala-lang:scala-library:jar:2.10.5:compile 
[INFO] +- org.scalatest:scalatest_2.10:jar:2.2.2:test 
[INFO] | \- org.scala-lang:scala-reflect:jar:2.10.4:compile 
[INFO] +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.0-cdh5.5.1:compile 
[INFO] | +- com.google.guava:guava:jar:11.0.2:compile 
[INFO] | | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile 
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile 
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile 
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile 
[INFO] | +- log4j:log4j:jar:1.2.17:compile 
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.5.1:compile 
[INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.5.1:compile 
[INFO] | | \- com.google.protobuf:protobuf-java:jar:2.5.0:compile 
[INFO] | \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0-cdh5.5.1:compile 
[INFO] |  +- javax.xml.bind:jaxb-api:jar:2.2.2:compile 
[INFO] |  | +- javax.xml.stream:stax-api:jar:1.0-2:compile 
[INFO] |  | \- javax.activation:activation:jar:1.1:compile 
[INFO] |  +- org.apache.commons:commons-compress:jar:1.4.1:compile 
[INFO] |  | \- org.tukaani:xz:jar:1.0:compile 
[INFO] |  +- commons-codec:commons-codec:jar:1.4:compile 
[INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile 
[INFO] |  +- com.sun.jersey:jersey-core:jar:1.9:compile 
[INFO] |  +- com.sun.jersey:jersey-client:jar:1.9:compile 
[INFO] |  +- com.google.inject.extensions:guice-servlet:jar:3.0:compile 
[INFO] |  +- commons-io:commons-io:jar:2.4:compile 
[INFO] |  +- com.google.inject:guice:jar:3.0:compile 
[INFO] |  | +- javax.inject:javax.inject:jar:1:compile 
[INFO] |  | \- aopalliance:aopalliance:jar:1.0:compile 
[INFO] |  +- com.sun.jersey:jersey-server:jar:1.9:compile 
[INFO] |  | \- asm:asm:jar:3.1:compile 
[INFO] |  +- com.sun.jersey:jersey-json:jar:1.9:compile 
[INFO] |  | +- org.codehaus.jettison:jettison:jar:1.1:compile 
[INFO] |  | \- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile 
[INFO] |  \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile 
[INFO] +- com.twitter:scalding-commons_2.10:jar:0.11.0:compile 
[INFO] | +- com.twitter:scalding-args_2.10:jar:0.11.0:compile 
[INFO] | +- com.backtype:dfs-datastores-cascading:jar:1.3.4:compile 
[INFO] | +- com.backtype:dfs-datastores:jar:1.3.4:compile 
[INFO] | | \- jvyaml:jvyaml:jar:1.0.0:compile 
[INFO] | +- com.twitter:bijection-core_2.10:jar:0.6.3:compile 
[INFO] | +- com.twitter:algebird-core_2.10:jar:0.7.0:compile 
[INFO] | | \- com.googlecode.javaewah:JavaEWAH:jar:0.6.6:compile 
[INFO] | +- com.twitter:chill_2.10:jar:0.4.0:compile 
[INFO] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:compile 
[INFO] | |  +-  com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile 
[INFO] | |  | \- org.ow2.asm:asm:jar:4.0:compile 
[INFO] | |  +- com.esotericsoftware.minlog:minlog:jar:1.2:compile 
[INFO] | |  \- org.objenesis:objenesis:jar:1.2:compile 
[INFO] | +- com.twitter.elephantbird:elephant-bird-cascading2:jar:4.4:compile 
[INFO] | +- com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16:compile 
[INFO] | +- org.apache.thrift:libthrift:jar:0.5.0:compile 
[INFO] | \- org.slf4j:slf4j-api:jar:1.6.6:compile 
[INFO] +- com.twitter:scalding-core_2.10:jar:0.15.0:compile 
[INFO] | +- com.twitter:scalding-serialization_2.10:jar:0.15.0:compile 
[INFO] | +- com.twitter:maple:jar:0.15.0:compile 
[INFO] | +- cascading:cascading-core:jar:2.6.1:compile 
[INFO] | | +- riffle:riffle:jar:0.1-dev:compile 
[INFO] | | +- thirdparty:jgrapht-jdk1.6:jar:0.8.1:compile 
[INFO] | | \- org.codehaus.janino:janino:jar:2.7.5:compile 
[INFO] | |  \- org.codehaus.janino:commons-compiler:jar:2.7.5:compile 
[INFO] | +- cascading:cascading-local:jar:2.6.1:compile 
[INFO] | +- cascading:cascading-hadoop:jar:2.6.1:compile 
[INFO] | +- com.twitter:chill-hadoop:jar:0.6.0:compile 
[INFO] | +- com.twitter:chill-algebird_2.10:jar:0.6.0:compile 
[INFO] | \- com.twitter:chill-java:jar:0.6.0:compile 
[INFO] +- com.twitter:scalding-date_2.10:jar:0.15.0:compile 
[INFO] +- com.twitter:scalding-json_2.10:jar:0.15.0:compile 
**[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.2:compile** 
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.6:compile 
[INFO] | \- org.json4s:json4s-native_2.10:jar:3.2.11:compile 
[INFO] +- parallelai:parallelai.spyglass:jar:2.10_0.10_CDH5.2.1_4.4:compile 
[INFO] | +- org.apache.hadoop:hadoop-client:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | +- org.apache.hadoop:hadoop-common:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | | +- org.apache.commons:commons-math3:jar:3.1.1:compile 
[INFO] | | | +- xmlenc:xmlenc:jar:0.52:compile 
[INFO] | | | +- commons-net:commons-net:jar:3.1:compile 
[INFO] | | | +- commons-configuration:commons-configuration:jar:1.6:compile 
[INFO] | | | | +- commons-digester:commons-digester:jar:1.8:compile 
[INFO] | | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile 
[INFO] | | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile 
[INFO] | | | +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile 
[INFO] | | | +- org.apache.avro:avro:jar:1.7.6-cdh5.2.1:compile 
[INFO] | | | | \- org.xerial.snappy:snappy-java:jar:1.0.5:compile 
[INFO] | | | +- com.google.code.gson:gson:jar:2.2.4:compile 
[INFO] | | | +- org.apache.curator:curator-client:jar:2.6.0:compile 
[INFO] | | | \- org.apache.curator:curator-recipes:jar:2.6.0:compile 
[INFO] | | +- org.apache.hadoop:hadoop-hdfs:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | | \- io.netty:netty:jar:3.6.2.Final:compile 
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile 
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.5.0-cdh5.2.1:compile 
[INFO] | +- org.apache.hbase:hbase-server:jar:0.98.6-cdh5.2.1:compile 
[INFO] | | +- org.apache.hbase:hbase-common:jar:0.98.6-cdh5.2.1:compile 
[INFO] | | +- org.apache.hbase:hbase-protocol:jar:0.98.6-cdh5.2.1:compile 
[INFO] | | +- org.apache.hbase:hbase-client:jar:0.98.6-cdh5.2.1:compile 
[INFO] | | +- org.apache.hbase:hbase-prefix-tree:jar:0.98.6-cdh5.2.1:runtime 
[INFO] | | | \- org.apache.hbase:hbase-common:jar:tests:0.98.6-cdh5.2.1:runtime 
[INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile 
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile 
[INFO] | | +- org.apache.hbase:hbase-hadoop-compat:jar:0.98.6-cdh5.2.1:compile 
[INFO] | | +- org.apache.hbase:hbase-hadoop2-compat:jar:0.98.6-cdh5.2.1:compile 
[INFO] | | +- com.yammer.metrics:metrics-core:jar:2.2.0:compile 
[INFO] | | +- com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.1:compile 
[INFO] | | +- org.apache.commons:commons-math:jar:2.1:compile 
[INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.2.1:compile 
[INFO] | | +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile 
[INFO] | | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26.cloudera.4:compile 
[INFO] | | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile 
[INFO] | | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile 
[INFO] | | +- tomcat:jasper-compiler:jar:5.5.23:compile 
[INFO] | | +- tomcat:jasper-runtime:jar:5.5.23:compile 
[INFO] | | | \- commons-el:commons-el:jar:1.0:compile 
[INFO] | | +- org.jamon:jamon-runtime:jar:2.3.1:compile 
[INFO] | | +- org.cloudera.htrace:htrace-core:jar:2.04:compile 
[INFO] | | +- org.apache.hadoop:hadoop-auth:jar:2.5.0-cdh5.2.1:compile 
[INFO] | | | +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile 
[INFO] | | | | \- org.apache.httpcomponents:httpcore:jar:4.2.4:compile 
[INFO] | | | +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile 
[INFO] | | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile 
[INFO] | | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile 
[INFO] | | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile 
[INFO] | | | \- org.apache.curator:curator-framework:jar:2.6.0:compile 
[INFO] | | +- org.apache.hadoop:hadoop-core:jar:2.5.0-mr1-cdh5.2.1:compile 
[INFO] | | | +- javax.servlet.jsp:jsp-api:jar:2.1:compile 
[INFO] | | | +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile 
[INFO] | | | +- hsqldb:hsqldb:jar:1.8.0.10:compile 
[INFO] | | | \- org.eclipse.jdt:core:jar:3.1.1:compile 
[INFO] | | +- org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.5.0-cdh5.2.1:compile 
[INFO] | | | \- commons-daemon:commons-daemon:jar:1.0.13:compile 
[INFO] | | +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile 
[INFO] | | \- junit:junit:jar:4.11:compile 
[INFO] | |  \- org.hamcrest:hamcrest-core:jar:1.3:compile 
[INFO] | +- com.jcraft:jsch:jar:0.1.51:compile 
[INFO] | +- com.twitter.elephantbird:elephant-bird-core:jar:4.1:compile 
[INFO] | | \- com.googlecode.json-simple:json-simple:jar:1.1:compile 
[INFO] | \- com.twitter.elephantbird:elephant-bird-hadoop-compat:jar:4.1:compile 
[INFO] +- com.typesafe:config:jar:1.2.1:compile 
[INFO] +- org.json4s:json4s-jackson_2.10:jar:3.3.0.RC1:compile 
[INFO] | \- org.json4s:json4s-core_2.10:jar:3.3.0.RC1:compile 
[INFO] |  +- org.json4s:json4s-ast_2.10:jar:3.3.0.RC1:compile 
[INFO] |  \- org.json4s:json4s-scalap_2.10:jar:3.3.0.RC1:compile 
**[INFO] \- com.fasterxml.jackson.core:jackson-databind:jar:2.4.2:compile 
[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.4.0:compile 
[INFO] \- com.fasterxml.jackson.core:jackson-core:jar:2.4.2:compile** 

Answer 1

通常、Hadoopにはクラスパス上に多数のjarのバージョンがあります。 mapreduce.job.user.classpath.firstをtrueに設定して、クラスパスを設定し直すことができます。または、hadoopデプロイメントが使用しているのと同じバージョンのjacksonを使用するようにプロジェクトをダウングレードすることもできます。

非分離クラスパスはおそらく最大のハープの痛みです。

Answer 2

NoSuchMethodErrorでジャクソンのためのgrepは常に互換性のないjarファイルの看板です。あなたのケースでは、私は両方のファイルの同じバージョンに切り替えてください

jackson.module.scala間と jackson.databindそれがあると言うでしょう。