Java MapReduce and HCatalog with Dynamic Partitions

[#1]

Couldn’t get Java MapReduce work with HCatalog in last HDP release, before HDP 2.3

Can anyone indicate if

1. writing to Hive tables with HCatalog using Java MapReduce work in HDP2.3 (an example would be great) and
2. writing to dynamic partitions with Java MapReduce is feasible (an example would be great)?

Background: HCatalog Limitations with Java MapReduce

Dynamic Partitions (at Write):

• Null Pointer Exception in DynamicPartFileRecordWriterContainer on null part-keys (perhaps this issue: https://issues.apache.org/jira/browse/HIVE-11470 )

• Partition column cannot be empty or NULL

Error:

15/11/30 15:11:56 INFO mapreduce.Job: Running job: job_1448337121999_0211
15/11/30 15:12:01 INFO mapreduce.Job: Job job_1448337121999_0211 running in uber mode : false
15/11/30 15:12:01 INFO mapreduce.Job:  map 0% reduce 0%
15/11/30 15:12:05 INFO mapreduce.Job:  map 100% reduce 0%
15/11/30 15:12:09 INFO mapreduce.Job: Task Id : attempt_1448337121999_0211_r_000000_0, Status : FAILED

Error: java.lang.NullPointerException 
at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:139)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:110)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:107)
at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

The DynamicPartitionTestJava package is attached

[#2]

@Ancil McBarnett Could you share error message? Also, if you can share the code then it will help.

[#3 · Accepted solution]

@Ancil McBarnett

See if this helps (from hcatalog project):

https://github.com/apache/hcatalog/blob/branch-0.1/src/test/org/apache/hcatalog/mapreduce/TestHCatPa...

[#4]

Thanks I see a newer version at

https://github.com/apache/hive/blob/master/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapr...

I need to review. If someone can help me test I would appreciate it.

[#5]

I uploaded the code and updated the Question with the error message. @Guilherme Braccialli @Neeraj Sabharwal

[#6]

@Ancil McBarnett please close

[#7]

It has now been patched.

https://issues.apache.org/jira/browse/HIVE-11470