Support Questions
Find answers, ask questions, and share your expertise

Java MapReduce and HCatalog with Dynamic Partitions

Solved Go to solution

Java MapReduce and HCatalog with Dynamic Partitions

Couldn’t get Java MapReduce work with HCatalog in last HDP release, before HDP 2.3

Can anyone indicate if

  1. writing to Hive tables with HCatalog using Java MapReduce work in HDP2.3 (an example would be great) and
  2. writing to dynamic partitions with Java MapReduce is feasible (an example would be great)?

Background: HCatalog Limitations with Java MapReduce

Dynamic Partitions (at Write):

  • Partition column cannot be empty or NULL

Error:

15/11/30 15:11:56 INFO mapreduce.Job: Running job: job_1448337121999_0211
15/11/30 15:12:01 INFO mapreduce.Job: Job job_1448337121999_0211 running in uber mode : false
15/11/30 15:12:01 INFO mapreduce.Job:  map 0% reduce 0%
15/11/30 15:12:05 INFO mapreduce.Job:  map 100% reduce 0%
15/11/30 15:12:09 INFO mapreduce.Job: Task Id : attempt_1448337121999_0211_r_000000_0, Status : FAILED
Error: java.lang.NullPointerException 
at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:139)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:110)
at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:107)
at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

The DynamicPartitionTestJava package is attached

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

6 REPLIES 6
Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

@Ancil McBarnett Could you share error message? Also, if you can share the code then it will help.

Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

Thanks I see a newer version at

https://github.com/apache/hive/blob/master/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapr...

I need to review. If someone can help me test I would appreciate it.

Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

I uploaded the code and updated the Question with the error message. @Guilherme Braccialli @Neeraj Sabharwal

Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

Mentor

@Ancil McBarnett please close

Highlighted

Re: Java MapReduce and HCatalog with Dynamic Partitions

Don't have an account?