HiveでTotalOrderPartitionerを使用できません：パーティションファイルを読み取れません

Question

Hiveからバルクロード用のHBase Hfilesを生成しようとしています。

java.lang.IllegalArgumentException: Can't read partitions file 

さらなる詳細：

カスタムパーティショナファイルはハイブのテーブルから作成されます。

私たちの主な問題は、

org.apache.hadoop.mapred.lib.TotalOrderPartitioner; 

を使用しているとき、それはカスタムパーティショナファイルを見つけることができないということです

CREATE EXTERNAL TABLE netezza.ais_lowres_mmsi_range_keys(hbase_key_range_start string) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe' 
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveNullValueSequenceFileOutputFormat' 
LOCATION '/tmp/ais_lowres_mmsi_range_keys'; 

INSERT OVERWRITE TABLE netezza.ais_lowres_mmsi_range_keys SELECT r_start FROM tmp_rows ORDER BY r_start; 



# The tmp_rows table holds the partition splits of our current HBase table 

# Table content is copied to a file as per: https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad 
hadoop fs -cp /tmp/ais_lowres_mmsi_range_keys/* /tmp/ais_lowres_mmsi_range_keys_list 

# Hive and HBase jars are added 
ADD JAR /usr/hdp/2.3.2.0-2950/hive/lib/hive-hbase-handler-1.2.1.2.3.2.0-2950.jar; 
ADD JAR /usr/hdp/2.3.2.0-2950/hbase/lib/hbase-server-1.1.2.2.3.2.0-2950.jar; 
ADD JAR /usr/hdp/2.3.2.0-2950/hbase/lib/hbase-common-1.1.2.2.3.2.0-2950.jar; 
ADD JAR /usr/hdp/2.3.2.0-2950/hbase/lib/hbase-client-1.1.2.2.3.2.0-2950.jar; 
ADD JAR /usr/hdp/2.3.2.0-2950/hbase/lib/hbase-protocol-1.1.2.2.3.2.0-2950.jar; 
SET hive.aux.jars.path = /tmp/hive-hbase-handler-1.2.1.2.3.2.0-2950.jar,/tmp/hbase-server-1.1.2.2.3.2.0-2950.jar,/tmp/hbase-client-1.1.2.2.3.2.0-2950.jar,/tmp/hbase-common-1.1.2.2.3.2.0-2950.jar,/tmp/hbase-protocol-1.1.2.2.3.2.0-2950.jar; 

SET hive.execution.engine=mr; 
SET mapreduce.job.reduces=$((num_range+1)); # The number of reducers is set to the number of partition splits +1 
SET hive.mapred.partitioner=org.apache.hadoop.mapred.lib.TotalOrderPartitioner; 
SET total.order.partitioner.natural.order=false; 
SET total.order.partitioner.path=/tmp/ais_lowres_mmsi_range_keys_list;   
SET hfile.compression=gz; 

INSERT OVERWRITE TABLE tmp_table 
SELECT [cols] FROM ais_lowres_mmsi_distinct 
CLUSTER BY hbase_key; 

次のエラーが生成されます。

###################################################################### 
Starting Job = job_1458218583243_0631, Tracking URL = http://osl5303.cm.cluster:8088/proxy/application_1458218583243_0631/ 
Kill Command = /usr/hdp/2.3.2.0-2950/hadoop/bin/hadoop job -kill job_1458218583243_0631 
Hadoop job information for Stage-1: number of mappers: 19; number of reducers: 49 
2016-03-30 08:19:39,534 Stage-1 map = 0%, reduce = 0% 
2016-03-30 08:19:55,084 Stage-1 map = 100%, reduce = 100% 
Ended Job = job_1458218583243_0631 with errors 
Error during job, obtaining debugging information... 
Examining task ID: task_1458218583243_0631_m_000009 (and more) from job job_1458218583243_0631 
Examining task ID: task_1458218583243_0631_m_000017 (and more) from job job_1458218583243_0631 
Examining task ID: task_1458218583243_0631_m_000008 (and more) from job job_1458218583243_0631 
Examining task ID: task_1458218583243_0631_m_000001 (and more) from job job_1458218583243_0631 
Examining task ID: task_1458218583243_0631_m_000008 (and more) from job job_1458218583243_0631 
Examining task ID: task_1458218583243_0631_m_000003 (and more) from job job_1458218583243_0631 

Task with the most failures(4): 
----- 
Task ID: 
    task_1458218583243_0631_m_000012 

URL: 
    http://osl5303.cm.cluster:8088/taskdetails.jsp?jobid=job_1458218583243_0631&tipid=task_1458218583243_0631_m_000012 
----- 
Diagnostic Messages for this Task: 
Error: java.lang.IllegalArgumentException: Can't read partitions file 
    at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) 
    at org.apache.hadoop.mapred.MapTask$OldOutputCollector.<init>(MapTask.java:592) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:422) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) 
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.io.FileNotFoundException: File file:/grid/3/hadoop/yarn/local/usercache/ketot/appcache/application_1458218583243_0631/container_e22_1458218583243_0631_01_000086/_partition.lst does not exist 
    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:609) 
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:822) 
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:599) 
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) 
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1752) 
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1776) 
    at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:301) 
    at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:88) 
    ... 10 more 


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask 
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 19 Reduce: 49 HDFS Read: 0 HDFS Write: 0 FAIL 
Total MapReduce CPU Time Spent: 0 msec 
######################################################################################### 

指定されたカスタムパスは無視されたようです。さらに、TotalOrderPartitionerが探すパスは、ローカルファイルシステム上であり、存在しないパスです。助言がありますか？

Answer 1

設定total.order.partitioner.pathが間違っています。その結果、TotalOrderParitionerがデフォルトを使用しようとしています。これはどこにも言及されていないようですが、私はそれを理解するためにソースコードを調べなければなりませんでした！

それが更新されています

となりました：

mapreduce.totalorderpartitioner.path

は廃止されたプロパティの完全なリストについては、https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/DeprecatedProperties.htmlを参照してください。