Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HUE/Oozie hive action errors when running workflow

avatar
Explorer

Running CDH 5.5.1, Kerberos enabled cluster.

 

I  have a hive .sql script that runs fine interactively.  When I try to launch via Oozie using Hive action it errors out.  Log file below.  There isn't much info in the logs.  Any ideas on how to troubleshoot?

 

Error message:

 

 Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]

 

Full log below:

2016-03-10 02:55:22,474 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@:start:] Start action [0000035-160308051341496-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-10 02:55:22,476 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@:start:] [***0000035-160308051341496-oozie-oozi-W@:start:***]Action status=DONE
2016-03-10 02:55:22,476 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@:start:] [***0000035-160308051341496-oozie-oozi-W@:start:***]Action updated in DB!
2016-03-10 02:55:22,521 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] Start action [0000035-160308051341496-oozie-oozi-W@hive2-7b7a] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-10 02:55:25,848 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] checking action, hadoop job ID [job_1457413978008_0096] status [RUNNING]
2016-03-10 02:55:25,849 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] [***0000035-160308051341496-oozie-oozi-W@hive2-7b7a***]Action status=RUNNING
2016-03-10 02:55:25,849 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] [***0000035-160308051341496-oozie-oozi-W@hive2-7b7a***]Action updated in DB!
2016-03-10 02:55:47,945 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[myhive2server.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] callback for action [0000035-160308051341496-oozie-oozi-W@hive2-7b7a]
2016-03-10 02:55:48,076 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] action completed, external ID [job_1457413978008_0096]
2016-03-10 02:55:48,082 WARN org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
2016-03-10 02:55:48,113 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] ERROR is considered as FAILED for SLA
2016-03-10 02:55:48,137 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@Kill] Start action [0000035-160308051341496-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-10 02:55:48,139 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@Kill] [***0000035-160308051341496-oozie-oozi-W@Kill***]Action status=DONE
2016-03-10 02:55:48,140 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@Kill] [***0000035-160308051341496-oozie-oozi-W@Kill***]Action updated in DB!

 

workflow.xml below:

 

<workflow-app name="dev-epm-rmt-test-hive" xmlns="uri:oozie:workflow:0.5">
  <global>
            <configuration>
                <property>
                    <name>mapreduce.job.queuename</name>
                    <value>root.epm</value>
                </property>
            </configuration>
  </global>
  <credentials>
    <credential name="hive2" type="hive2">
      <property>
        <name>hive2.jdbc.url</name>
        <value>jdbc:hive2://myhive2server.com:10000/default</value>
      </property>
      <property>
        <name>hive2.server.principal</name>
        <value>hive/myhive2server.com@MYREALM.ORG</value>
      </property>
    </credential>
  </credentials>
    <start to="hive2-7b7a"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="hive2-7b7a" cred="hive2">
        <hive2 xmlns="uri:oozie:hive2-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <jdbc-url>jdbc:hive2://myhive2server.com:10000/default</jdbc-url>
            <password>hive2</password>
            <script>/code/contacts.sql</script>
            <file>/code/contacts.sql#contacts.sql</file>
        </hive2>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>
 

 

 

1 ACCEPTED SOLUTION

avatar
Explorer

I was able to solve this by finding the URL to the Job History Server UI.  Stdout and Stderr can be sen here.

View solution in original post

6 REPLIES 6

avatar
Explorer

I was able to solve this by finding the URL to the Job History Server UI.  Stdout and Stderr can be sen here.

avatar
New Contributor

How did you use the url to resolve the issue? Did you identify a root cause and resolution action? I'm experiencing similar error

avatar
Explorer

I think there was a bug in 5.5.1 where you could not review standard out and standard err outputs of shell scripts and hive scripts called from an oozie workflow.  That was fixed in later versions. The job history was used as a workaround to review the error messages and find root cause.

 

If you are calling the hive action/script from a shell script instead of a hive action you will also need to do the following.
1.Use kinit (if running a secure cluster) and kinit to an account that has access to the hive schema.
2.You must set the mapreduce.job.credentials.binary to $HADOOP_TOKEN_FILE_LOCATION in the hive script;

Sample code below:

HIVEQUERY='hive -S -e "SET mapreduce.job.credentials.binary=$HADOOP_TOKEN_FILE_LOCATION;set mapred.job.queue.name=myquename;SELECT count(*) FROM schema.table;"'

avatar
New Contributor
Thanks for your response. In my case I'm using a simple HIVE action with no shell script. I'll dig into this some more, hopefully I can find the root cause in my case. I'll report if I find anything that would be useful to a large audience

avatar
Explorer

One last thing worth mentioning.  If you are using a hive action, you will need to ensure that you have the HiveServer2 URL identified on the properties tab of Hue/oozie for the action.  

E.g., jdbc:hive2://<HIVE_SERVER_HOSTNAME>:10000/default

avatar
New Contributor
You were right on point. I didn't have this property specified and when I initially added it I forgot to save it. Got it working. Gracias!!!