Oozie workflow with sqoop export failed while using more than 1 mapper in command.

[#1]

Hi Team,

We are trying to export data from hive table to RDBMS but while using sqoop export command with more than 1 mapper. Oozie workflow is failing with the following exception :

org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/utils/MetaStoreUtils
at org.apache.hive.hcatalog.mapreduce.InternalUtil.getSerdeProperties(InternalUtil.java:165)
at org.apache.hive.hcatalog.mapreduce.InternalUtil.initializeDeserializer(InternalUtil.java:155)
at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.createDeserializer(HCatRecordReader.java:127)
at org.apache.hive.hcatalog.mapreduce.HCatRecordReader.initialize(HCatRecordReader.java:92)
at org.apache.sqoop.mapreduce.hcat.SqoopHCatRecordReader.initialize(SqoopHCatRecordReader.java:68)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:554)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:792)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.utils.MetaStoreUtils
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more

Can you please assist.

Regards,

Deepa

[#2]

@Djain 

Hi Deepa,

It seems you are missing the Hive Clients from the NodeManagers, which could be causing the issue.

Can you please make sure you have hive clients installed on all the NodeManager nodes and then confirm you are hitting the issue.

Also, if you are facing the problem, will you be able to upload the complete yarn app logs.

Thanks,

Rohit Rai Malhotra