Oozie error when using with Hive2

[#1]

Hi Guys,

We are trying to use Oozie for our ETL process by connecting to Hive2 but while working with it facing issues during the jobs. Need your help to resolve the issue.

HDP-2.5.3 is deployed in our cluster and below the workflow.xml file:

<workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1"> <start to='hive2tab'/> <action name='hive2tab'> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>aa.bb.com:8050</job-tracker> <name-node>hdfs://cc.dd.com:8020</name-node> <jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url> <password>ZZZZ</password> <script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script> </hive2> <ok to='end'/> <error to='end'/> </action> <kill name='kill'> <message>JOb_failed</message> </kill> <end name='end'/> </workflow-app>

This is a simple job to execute a .hql script. However, when running this job we can see the below error in the Oozie log:

2017-05-18 17:25:32,510 WARN Hive2ActionExecutor:523 - SERVER[dn2.tcsgegdc.com] USER[tcs_ge_user] GROUP[-] TOKEN[] APP[Meg_Test1] JOB[0000004-170518130535586-oozie-oozi-W] ACTION[0000004-170518130535586-oozie-oozi-W@Hive1_Table] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2212) ... 9 more

Please help us in resolving this issue as this is very important for us.

Thanks and Regards,

Rajdip

[#2]

Workflow xml is:

test.xml

[#3]

<workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1">
    <start to='hive2tab'/>
    <action name='hive2tab'>
        <hive2 xmlns="uri:oozie:hive2-action:0.1">
            <job-tracker>aa.bb.com:8050</job-tracker>
            <name-node>hdfs://cc.dd.com:8020</name-node>
            <jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url>
	<password>ZZZZ</password>
	<script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script>
        </hive2>
        <ok to='end'/>
        <error to='end'/>
    </action>
    <kill name='kill'>
        <message>JOb_failed</message>
    </kill>
    <end name='end'/>
</workflow-app>

[#4]

Really need your help to resolve the issue. Because we have planned to use Hive2 for our use case.

[#5]

@rajdip chaudhuri - Since oozie job id was generated and then the Hive2Main failed, I would recommend you to check the yarn logs for the oozie id mentioned in the error log you have posted.

To check yarn log use :

yarn logs -applicationId 0000004-170518130535586-oozie-oozi-W

Mostly this yarn log will give you enough hints, however, feel free to revert with the same in case you need more help.

As always, if you find this answer useful, please don't forget to accept the answer.

[#6]

A couple of things you might try:

* Set oozie.use.system.libpath=true in the job.properties file and ensure you have the proper jars in sharelib.

* Add the Hive settings file to the workflow.xml, such as <job-xml>hive-site.xml</job-xml>.

[#7]

@rajdip chaudhuri

Can you please make sure that you have oozie-sharelib-hive2-<version>.jar in your oozie(hive2) sharelib?

If it's there then probably you are missing oozie.use.system.libpath=true in your job.properties/job.xml file. Please try to add and re-run.

E.g.

hadoop fs -ls -R /user/oozie/share/lib/|grep oozie-sharelib-hive2-4.2.0.2.5.3.29-1.jar

[#8]

Hi @Kuldeep Kulkarni jar file as you mentioned is there in the path hdfs://nn.tcsgegdc.com:8020/user/oozie/share/lib/lib_20170419171320/hive2/oozie-sharelib-hive2-4.2.0.2.5.3.0-37.jar and have added the parameters as you mentioned in the workflow xml. Please note that we are not using job properties file yet. After the changes we are still facing the same issue of :

java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found

Changed xml :

<workflow-app name='Raj' xmlns="uri:oozie:workflow:0.1">
    <start to='hive2tab'/>
    <action name='hive2tab'>
        <hive2 xmlns="uri:oozie:hive2-action:0.1">
            <job-tracker>dn1.tcsgegdc.com:8050</job-tracker>
            <name-node>hdfs://nn.tcsgegdc.com:8020</name-node>
            <configuration>
                <property>
                    <name>oozie.use.system.libpath</name>
                    <value>true</value>
                </property>
				<property>
                    <name>oozie.libpath</name>
                    <value>hdfs://nn.tcsgegdc.com:8020/user/oozie/share/lib/lib_20170419171320/hive2</value>
                </property>
            </configuration>
			<jdbc-url>jdbc:hive2://3.209.124.207:10000/demo</jdbc-url>
			<password>tcs_ge_user</password>
			<script>hdfs://nn.tcsgegdc.com:8020/user/tcs_ge_user/oozie_test/tab3.hql</script>
        </hive2>
        <ok to='end'/>
        <error to='end'/>
    </action>
    <kill name='kill'>
        <message>JOb_failed</message>
    </kill>
    <end name='end'/>
</workflow-app>

[#9]

Hi @Kuldeep Kulkarni, using Hive instead of Hive2 from oozie is also not working. When we are using Hive getting the below error:

java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found

Really looking for some help here as we are totally stuck.

[#10]

@Kuldeep Kulkarni we have deleted the oozie service and added the service again from Ambari. But now below command doesn't give the list of jars that are there in the oozie sharedlib

oozie admin -oozie http://dn2.tcsgegdc.com:11000/oozie -shareliblist hive* > hive_shareliblist.txt 2>&1

oozie admin -oozie http://dn2.tcsgegdc.com:11000/oozie -shareliblist hive2* > hive2_shareliblist_OLD.txt 2>&1

We have tried lot of changes as has been suggested at the other HCC links, but nothing is working. Really need some help here. We think its an issue of HDP.

[#11]

Hi Guys, help is really appreciated. Totally stuck with it for both Hive and Hive2.

[#12]

Hi,

Any ideas please ? Is this an error in HDP ?

[#13]

Hi Guys,

Can you please guide us? Not able to progress with it at all. Really looking for some help.

[#14]

@rajdip chaudhuri - No its not an issue with HDP.

Please try to re-create sharelib using below commands.

#Command 1
/usr/hdp/<version>/oozie/bin/oozie-setup.sh sharelib create -locallib /usr/hdp/<version>/oozie/oozie-sharelib.tar.gz -fs hdfs://<namenode>:8020


#Command 2
oozie admin -oozie http://<oozie-server>:11000/oozie -sharelibupdate

[#15]

Hi @Kuldeep Kulkarni we tried that also but didn't work. Even we uninstalled the service and installed Oozie again. We also changed the oozie library permission to 755. All these steps resulted in same error.

What we should do now? Using HDP 2.5.3 cluster. Is this an issue in this version?

Please guide us.

[#16]

We can try running these commands next week Monday. Should we run this command as the oozie user in the cluster? Also please note we are running the oozie job from individual developer passwords.

[#17]

@rajdip chaudhuri

How are you submitting the job? I would suggest to add oozie.use.system.libpath=true in job.properties or job.xml and remove both the libpath related properties from your workflow.xml

[#18]

Since this is the first time we are doing so we using the workflow XML file directly while running the job. We added the oozie.use.system.libpath=true parameter the XML along with the other ones as in above comments. SO, do you want to add just this parameter and remove the other ? Also can you share a XML file which works on HDP-2.5.3 while connecting to beeline with auth over non-kerberized cluster?

Please suggest.