Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie error when using with Hive2

Oozie error when using with Hive2

Contributor

Hi Guys,

We are trying to use Oozie for our ETL process by connecting to Hive2 but while working with it facing issues during the jobs. Need your help to resolve the issue.

HDP-2.5.3 is deployed in our cluster and below the workflow.xml file:

<workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1"> <start to='hive2tab'/> <action name='hive2tab'> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>aa.bb.com:8050</job-tracker> <name-node>hdfs://cc.dd.com:8020</name-node> <jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url> <password>ZZZZ</password> <script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script> </hive2> <ok to='end'/> <error to='end'/> </action> <kill name='kill'> <message>JOb_failed</message> </kill> <end name='end'/> </workflow-app>

This is a simple job to execute a .hql script. However, when running this job we can see the below error in the Oozie log:

2017-05-18 17:25:32,510 WARN Hive2ActionExecutor:523 - SERVER[dn2.tcsgegdc.com] USER[tcs_ge_user] GROUP[-] TOKEN[] APP[Meg_Test1] JOB[0000004-170518130535586-oozie-oozi-W] ACTION[0000004-170518130535586-oozie-oozi-W@Hive1_Table] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2212) ... 9 more

Please help us in resolving this issue as this is very important for us.

Thanks and Regards,

Rajdip

17 REPLIES 17

Re: Oozie error when using with Hive2

Contributor

Workflow xml is:

test.xml

Re: Oozie error when using with Hive2

Contributor
<workflow-app name='Meg' xmlns="uri:oozie:workflow:0.1">
    <start to='hive2tab'/>
    <action name='hive2tab'>
        <hive2 xmlns="uri:oozie:hive2-action:0.1">
            <job-tracker>aa.bb.com:8050</job-tracker>
            <name-node>hdfs://cc.dd.com:8020</name-node>
            <jdbc-url>jdbc:hive2://xx.zz.com:10000/demo</jdbc-url>
	<password>ZZZZ</password>
	<script>hdfs://cc.dd.com:8020/user/WWWWW/oozie_test/tab3.hql</script>
        </hive2>
        <ok to='end'/>
        <error to='end'/>
    </action>
    <kill name='kill'>
        <message>JOb_failed</message>
    </kill>
    <end name='end'/>
</workflow-app>

Re: Oozie error when using with Hive2

Contributor

Really need your help to resolve the issue. Because we have planned to use Hive2 for our use case.

Re: Oozie error when using with Hive2

@rajdip chaudhuri - Since oozie job id was generated and then the Hive2Main failed, I would recommend you to check the yarn logs for the oozie id mentioned in the error log you have posted.

To check yarn log use :

yarn logs -applicationId 0000004-170518130535586-oozie-oozi-W

Mostly this yarn log will give you enough hints, however, feel free to revert with the same in case you need more help.

As always, if you find this answer useful, please don't forget to accept the answer.

Re: Oozie error when using with Hive2

Rising Star

A couple of things you might try:

* Set oozie.use.system.libpath=true in the job.properties file and ensure you have the proper jars in sharelib.

* Add the Hive settings file to the workflow.xml, such as <job-xml>hive-site.xml</job-xml>.

Re: Oozie error when using with Hive2

Super Guru
@rajdip chaudhuri

Can you please make sure that you have oozie-sharelib-hive2-<version>.jar in your oozie(hive2) sharelib?

If it's there then probably you are missing oozie.use.system.libpath=true in your job.properties/job.xml file. Please try to add and re-run.

E.g.

hadoop fs -ls -R /user/oozie/share/lib/|grep oozie-sharelib-hive2-4.2.0.2.5.3.29-1.jar
Highlighted

Re: Oozie error when using with Hive2

Contributor

Hi @Kuldeep Kulkarni jar file as you mentioned is there in the path hdfs://nn.tcsgegdc.com:8020/user/oozie/share/lib/lib_20170419171320/hive2/oozie-sharelib-hive2-4.2.0.2.5.3.0-37.jar and have added the parameters as you mentioned in the workflow xml. Please note that we are not using job properties file yet. After the changes we are still facing the same issue of :

java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.Hive2Main not found

Changed xml :

<workflow-app name='Raj' xmlns="uri:oozie:workflow:0.1">
    <start to='hive2tab'/>
    <action name='hive2tab'>
        <hive2 xmlns="uri:oozie:hive2-action:0.1">
            <job-tracker>dn1.tcsgegdc.com:8050</job-tracker>
            <name-node>hdfs://nn.tcsgegdc.com:8020</name-node>
            <configuration>
                <property>
                    <name>oozie.use.system.libpath</name>
                    <value>true</value>
                </property>
				<property>
                    <name>oozie.libpath</name>
                    <value>hdfs://nn.tcsgegdc.com:8020/user/oozie/share/lib/lib_20170419171320/hive2</value>
                </property>
            </configuration>
			<jdbc-url>jdbc:hive2://3.209.124.207:10000/demo</jdbc-url>
			<password>tcs_ge_user</password>
			<script>hdfs://nn.tcsgegdc.com:8020/user/tcs_ge_user/oozie_test/tab3.hql</script>
        </hive2>
        <ok to='end'/>
        <error to='end'/>
    </action>
    <kill name='kill'>
        <message>JOb_failed</message>
    </kill>
    <end name='end'/>
</workflow-app>

Re: Oozie error when using with Hive2

Contributor

Hi @Kuldeep Kulkarni, using Hive instead of Hive2 from oozie is also not working. When we are using Hive getting the below error:

java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found

Really looking for some help here as we are totally stuck.

Re: Oozie error when using with Hive2

Contributor

@Kuldeep Kulkarni we have deleted the oozie service and added the service again from Ambari. But now below command doesn't give the list of jars that are there in the oozie sharedlib

oozie admin -oozie http://dn2.tcsgegdc.com:11000/oozie -shareliblist hive* > hive_shareliblist.txt 2>&1

oozie admin -oozie http://dn2.tcsgegdc.com:11000/oozie -shareliblist hive2* > hive2_shareliblist_OLD.txt 2>&1

We have tried lot of changes as has been suggested at the other HCC links, but nothing is working. Really need some help here. We think its an issue of HDP.

Don't have an account?
Coming from Hortonworks? Activate your account here