は、バリュークラスのシリアライザが見つかりませんでした：「org.apache.hadoop.hbase.client.Result」

Question

私はHBaseのからデータを読み込み、sequenceFileとして保存しようとしているが、しかし、取得

java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 

エラーです。私は2件の類似の記事を見

：これら2つのポストに続い

hadoop writables NotSerializableException with Apache Spark API

と

Spark HBase Join Error: object not serializable class: org.apache.hadoop.hbase.client.Result

を、私は三つのクラスとKYROクラスを登録しませんが、まだ運。ここで

は私のプログラムです：

 String tableName = "validatorTableSample"; 
     System.out.println("Start indexing hbase: " + tableName); 
     SparkConf sparkConf = new SparkConf().setAppName("HBaseRead"); 
     Class[] classes = {org.apache.hadoop.io.LongWritable.class, org.apache.hadoop.io.Text.class, org.apache.hadoop.hbase.client.Result.class}; 
     sparkConf.registerKryoClasses(classes); 
     JavaSparkContext sc = new JavaSparkContext(sparkConf); 
     Configuration conf = HBaseConfiguration.create(); 
     conf.set(TableInputFormat.INPUT_TABLE, tableName); 
//  conf.setStrings("io.serializations", 
//   conf.get("io.serializations"), 
//   MutationSerialization.class.getName(), 
//   ResultSerialization.class.getName()); 
     conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); 

     JavaPairRDD<ImmutableBytesWritable, Result> hBasePairRDD = sc.newAPIHadoopRDD(
      conf, 
      TableInputFormat.class, 
      ImmutableBytesWritable.class, 
      Result.class); 

     hBasePairRDD.saveAsNewAPIHadoopFile("/tmp/tempOutputPath", ImmutableBytesWritable.class, Result.class, SequenceFileOutputFormat.class); 
     System.out.println("Finished readFromHbaseAndSaveAsSequenceFile() ........."); 

ここでエラースタックトレースです：ここでは

java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1254) 
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1112) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1095) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 
    at org.apache.spark.scheduler.Task.run(Task.scala:86) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
17/05/25 10:58:38 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.io.IOException: Could not find a serializer for the Value class: 'org.apache.hadoop.hbase.client.Result'. Please ensure that the configuration 'io.serializations' is properly configured, if you're usingcustom serialization. 
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1254) 
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1156) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273) 
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1112) 
    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1095) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 
    at org.apache.spark.scheduler.Task.run(Task.scala:86) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 

17/05/25 10:58:38 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job 

Answer 1

は、我々のデータを格納するためのHBaseを使用して、このので、それは

を動作させるために必要だったものですレデューサーはその結果をHBaseテーブルに出力し、Hadoopはデータをシリアル化する方法がわからないことを伝えています。それが私たちがそれを助ける必要がある理由です。 setUp内でio.serializations変数を設定します

conf.setStrings("io.serializations", new String[]{hbaseConf.get("io.serializations"), MutationSerialization.class.getName(), ResultSerialization.class.getName()});