Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Java MapReduce and HCatalog with Dynamic Partitions

avatar

Couldn’t get Java MapReduce work with HCatalog in last HDP release, before HDP 2.3

Can anyone indicate if

  1. writing to Hive tables with HCatalog using Java MapReduce work in HDP2.3 (an example would be great) and
  2. writing to dynamic partitions with Java MapReduce is feasible (an example would be great)?

Background: HCatalog Limitations with Java MapReduce

Dynamic Partitions (at Write):

  • Partition column cannot be empty or NULL

Error:

15/11/30 15:11:56 INFO mapreduce.Job: Running job: job_1448337121999_0211
15/11/30 15:12:01 INFO mapreduce.Job: Job job_1448337121999_0211 running in uber mode : false
15/11/30 15:12:01 INFO mapreduce.Job:  map 0% reduce 0%
15/11/30 15:12:05 INFO mapreduce.Job:  map 100% reduce 0%
15/11/30 15:12:09 INFO mapreduce.Job: Task Id : attempt_1448337121999_0211_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException 
at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:139)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:110)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:107)
at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

The DynamicPartitionTestJava package is attached

1 ACCEPTED SOLUTION
6 REPLIES 6

avatar
Master Mentor

@Ancil McBarnett Could you share error message? Also, if you can share the code then it will help.

avatar

avatar

Thanks I see a newer version at

https://github.com/apache/hive/blob/master/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapr...

I need to review. If someone can help me test I would appreciate it.

avatar

I uploaded the code and updated the Question with the error message. @Guilherme Braccialli @Neeraj Sabharwal

avatar
Master Mentor

@Ancil McBarnett please close

avatar