https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97701#M11135 <P>Couldn’t get Java MapReduce work with HCatalog in last HDP release, before HDP 2.3 </P><P>Can anyone indicate if</P><OL> <LI> writing to Hive tables with HCatalog using Java MapReduce work in HDP2.3 (an example would be great) and</LI><LI>writing to dynamic partitions with Java MapReduce is feasible (an example would be great)?</LI></OL><P><STRONG>Background: HCatalog Limitations with Java MapReduce</STRONG></P><P><STRONG><EM>Dynamic Partitions (at Write):</EM></STRONG></P><UL><LI>Null Pointer Exception in DynamicPartFileRecordWriterContainer on null part-keys (perhaps this issue: <A href="https://issues.apache.org/jira/browse/HIVE-11470" target="_blank">https://issues.apache.org/jira/browse/HIVE-11470</A> )</LI></UL><UL><LI>Partition column cannot be empty or NULL</LI></UL><P>Error:</P><PRE>15/11/30 15:11:56 INFO mapreduce.Job: Running job: job_1448337121999_0211 15/11/30 15:12:01 INFO mapreduce.Job: Job job_1448337121999_0211 running in uber mode : false 15/11/30 15:12:01 INFO mapreduce.Job: map 0% reduce 0% 15/11/30 15:12:05 INFO mapreduce.Job: map 100% reduce 0% 15/11/30 15:12:09 INFO mapreduce.Job: Task Id : attempt_1448337121999_0211_r_000000_0, Status : FAILED</PRE><PRE>Error: java.lang.NullPointerException at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:139) at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:110) at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105) at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:107) at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:1) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)</PRE><P>The <A href="https://community.cloudera.com/legacyfs/online/attachments/600-dynamicpartitiontestjava.txt">DynamicPartitionTestJava</A> package is attached</P> Mon, 30 Nov 2015 14:02:39 GMT amcbarnett 2015-11-30T14:02:39Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97701#M11135 <P>Couldn’t get Java MapReduce work with HCatalog in last HDP release, before HDP 2.3 </P><P>Can anyone indicate if</P><OL> <LI> writing to Hive tables with HCatalog using Java MapReduce work in HDP2.3 (an example would be great) and</LI><LI>writing to dynamic partitions with Java MapReduce is feasible (an example would be great)?</LI></OL><P><STRONG>Background: HCatalog Limitations with Java MapReduce</STRONG></P><P><STRONG><EM>Dynamic Partitions (at Write):</EM></STRONG></P><UL><LI>Null Pointer Exception in DynamicPartFileRecordWriterContainer on null part-keys (perhaps this issue: <A href="https://issues.apache.org/jira/browse/HIVE-11470" target="_blank">https://issues.apache.org/jira/browse/HIVE-11470</A> )</LI></UL><UL><LI>Partition column cannot be empty or NULL</LI></UL><P>Error:</P><PRE>15/11/30 15:11:56 INFO mapreduce.Job: Running job: job_1448337121999_0211 15/11/30 15:12:01 INFO mapreduce.Job: Job job_1448337121999_0211 running in uber mode : false 15/11/30 15:12:01 INFO mapreduce.Job: map 0% reduce 0% 15/11/30 15:12:05 INFO mapreduce.Job: map 100% reduce 0% 15/11/30 15:12:09 INFO mapreduce.Job: Task Id : attempt_1448337121999_0211_r_000000_0, Status : FAILED</PRE><PRE>Error: java.lang.NullPointerException at org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:139) at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:110) at org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:54) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105) at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:107) at com.kgi.test.mapreduce.hcatalog.DynamicPartitionTest$Reduce.reduce(DynamicPartitionTest.java:1) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)</PRE><P>The <A href="https://community.cloudera.com/legacyfs/online/attachments/600-dynamicpartitiontestjava.txt">DynamicPartitionTestJava</A> package is attached</P> Mon, 30 Nov 2015 14:02:39 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97701#M11135 amcbarnett 2015-11-30T14:02:39Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97702#M11136 <P><A rel="user" href="https://community.cloudera.com/users/369/amcbarnett.html" nodeid="369">@Ancil McBarnett</A> Could you share error message? Also, if you can share the code then it will help. </P> Mon, 30 Nov 2015 20:48:06 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97702#M11136 nsabharwal 2015-11-30T20:48:06Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97703#M11137 <A rel="user" href="https://community.cloudera.com/users/369/amcbarnett.html" nodeid="369">@Ancil McBarnett</A><P>See if this helps (from hcatalog project):</P><P><A target="_blank" href="https://github.com/apache/hcatalog/blob/branch-0.1/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java">https://github.com/apache/hcatalog/blob/branch-0.1/src/test/org/apache/hcatalog/mapreduce/TestHCatPartitioned.java</A></P> Tue, 01 Dec 2015 19:44:31 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97703#M11137 gbraccialli3 2015-12-01T19:44:31Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97704#M11138 <P>I uploaded the code and updated the Question with the error message. <A rel="user" href="https://community.cloudera.com/users/238/gbraccialli.html" nodeid="238">@Guilherme Braccialli</A> <A rel="user" href="https://community.cloudera.com/users/140/nsabharwal.html" nodeid="140">@Neeraj Sabharwal</A></P> Wed, 02 Dec 2015 07:10:11 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97704#M11138 amcbarnett 2015-12-02T07:10:11Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97705#M11139 <P>Thanks I see a newer version at </P><P><A href="https://github.com/apache/hive/blob/master/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatDynamicPartitioned.java" target="_blank">https://github.com/apache/hive/blob/master/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatDynamicPartitioned.java</A></P><P>I need to review. If someone can help me test I would appreciate it.</P> Wed, 02 Dec 2015 07:18:03 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97705#M11139 amcbarnett 2015-12-02T07:18:03Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97706#M11140 <P><A rel="user" href="https://community.cloudera.com/users/369/amcbarnett.html" nodeid="369">@Ancil McBarnett</A> please close</P> Wed, 03 Feb 2016 23:50:22 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97706#M11140 aervits 2016-02-03T23:50:22Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97707#M11141 <P>It has now been patched. </P><P><A href="https://issues.apache.org/jira/browse/HIVE-11470" target="_blank">https://issues.apache.org/jira/browse/HIVE-11470</A></P> Tue, 08 Mar 2016 03:33:09 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Java-MapReduce-and-HCatalog-with-Dynamic-Partitions/m-p/97707#M11141 amcbarnett 2016-03-08T03:33:09Z