のHadoop 2.6.0-cdh5.7.1とドルイドのエラーローディングバッチデータ（マップ減らす）

Question

Iはドルイドを構築し、その中にバッチデータをロードするとき、次のように、エラーが発生します。

 2016-10-19T07:06:08,656 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 12 
     Job Counters 
      Failed map tasks=4 
      Launched map tasks=4 
      Other local map tasks=3 
      Data-local map tasks=1 
      Total time spent by all maps in occupied slots (ms)=13910 
      Total time spent by all reduces in occupied slots (ms)=0 
      Total time spent by all map tasks (ms)=13910 
      Total vcore-seconds taken by all map tasks=13910 
      Total megabyte-seconds taken by all map tasks=14243840 
     Map-Reduce Framework 
      CPU time spent (ms)=0 
      Physical memory (bytes) snapshot=0 
      Virtual memory (bytes) snapshot=0 
    2016-10-19T07:06:08,657 ERROR [task-runner-0-priority-0] io.druid.indexer.DetermineHashedPartitionsJob - Job failed: job_1476065812127_0012 
    2016-10-19T07:06:08,657 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[var/druid/hadoop-tmp/wikiticker/2016-10-19T070532.438Z/9d778c51830e449f831a3966a679df11] 
    2016-10-19T07:06:08,682 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikiticker_2016-10-19T07:05:32.450Z, type=index_hadoop, dataSource=wikiticker}] 
    java.lang.RuntimeException: java.lang.reflect.InvocationTargetException 
     at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?] 
     at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79] 
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79] 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79] 
     at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79] 
    Caused by: java.lang.reflect.InvocationTargetException 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79] 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79] 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79] 
     at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79] 
     at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     ... 7 more 
    Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed! 
     at io.druid.indexer.JobHelper.runJobs(JobHelper.java:343) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1] 
     at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1] 
     at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79] 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79] 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79] 
     at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79] 
     at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1] 
     ... 7 more 
    2016-10-19T07:06:08,687 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2016-10-19T07:05:32.450Z] status changed to [FAILED]. 
    2016-10-19T07:06:08,688 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { 
     "id" : "index_hadoop_wikiticker_2016-10-19T07:05:32.450Z", 
     "status" : "FAILED", 
     "duration" : 33220 
    } 

そしてofficial solutionによると、私はいくつかの操作をしましたが、また少し使用：

• 編集Druidののpom.xmlに依存関係が私のHadoopのバージョンにジャクソンのバージョンと一致し、ドルイド

  Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.annotations()Ljava/lang/Iterable; 
  

  を再コンパイルします

• 利用のmaven-シェードプラグインは脂肪ジャードルイド・サービス-0.9.1.1-selfcontained.jar

  java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath ~/druid-0.9.1.1/tmp/druid-services-0.9.1.1-selfcontained.jar:~/druid-0.9.1.1/conf/druid/_common/:~/jackson-* -Ddruid.extensions.loadList=[] io.druid.cli.Main index hadoop --coordinate "org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1" ../quickstart/wikiticker-index.json 
  

excuteこのcommondを作るために、私が得た：

INFO: An exception was caught and reported. Message: java.lang.NullPointerException 
    java.lang.NullPointerException 
     at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95) 
     at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) 
     at com.google.inject.spi.Elements.getElements(Elements.java:103) 
     at com.google.inject.spi.Elements.getElements(Elements.java:94) 
     at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173) 
     at com.google.inject.AbstractModule.configure(AbstractModule.java:62) 
     at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) 
     at com.google.inject.spi.Elements.getElements(Elements.java:103) 
     at com.google.inject.spi.Elements.getElements(Elements.java:94) 
     at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:153) 
     at com.google.inject.AbstractModule.configure(AbstractModule.java:62) 
     at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) 
     at com.google.inject.spi.Elements.getElements(Elements.java:103) 
     at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136) 
     at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) 
     at com.google.inject.Guice.createInjector(Guice.java:96) 
     at com.google.inject.Guice.createInjector(Guice.java:73) 
     at com.google.inject.Guice.createInjector(Guice.java:62) 
     at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) 
     at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) 
     at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) 
     at io.druid.cli.Main.main(Main.java:105) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115) 
     at io.druid.cli.Main.main(Main.java:105) 

    2016-10-20T05:05:52,401 ERROR [main] io.druid.cli.CliHadoopIndexer - failure!!!! 
    java.lang.reflect.InvocationTargetException 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79] 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79] 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79] 
     at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79] 
     at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115) [druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
    Caused by: com.google.inject.CreationException: Guice creation errors: 

    1) An exception was caught and reported. Message: null 
     at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173) 

    2) Binding to null instances is not allowed. Use toProvider(Providers.of(null)) if this is your intended behaviour. 
     at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:93) 

    3) Could not find a suitable constructor in io.druid.metadata.MetadataStorageTablesConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private. 
     at io.druid.metadata.MetadataStorageTablesConfig.class(MetadataStorageTablesConfig.java:35) 
     at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95) 

    3 errors 
     at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.Main.main(Main.java:105) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     ... 6 more 
    Caused by: java.lang.NullPointerException 
     at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements.getElements(Elements.java:94) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements.getElements(Elements.java:94) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:153) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     at io.druid.cli.Main.main(Main.java:105) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1] 
     ... 6 more 

誰もが同じ問題を抱えていますか？誰でも助けることができますか？

Answer 1

インデックス作成ジョブを実行できました。陰干しされた瓶は必要ありません。ここに手順があります。

1）CDHに一致するHadoopクライアントをダウンロードします。 druid配布ディレクトリで、このコマンドを実行します。 Hadoopクライアントをhadoop-dependenciesディレクトリにダウンロードします。

./bin/pull-deps -r https://repository.cloudera.com/content/repositories/releases/ -h org.apache.hadoop:hadoop-client:2.6.0-mr1-cdh5.7.1 

2）クライアントバージョンをmiddlemanager設定で設定します。

hadoopDependencyCoordinates=["org.apache.hadoop:hadoop-client:2.6.0-mr1-cdh5.7.1"] 

3）インデックスサービスを使用して、Hadoopジョブを実行しました。 JSONを作成し、jobPropertiesのように入力します。

"jobProperties": { 
    "mapreduce.job.classloader": "true", 
    "mapreduce.job.classloader.system.classes": "-javax.validation.,java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop.,org.xerial.snappy." 
} 

あなたはまだいくつかのエラーを表示し、それがCDHとドルイドに存在することがわかっている場合は、必要に応じて、リストにクラスを追加します。

4）HTTP POSTによるJSONのオーバーロード