HUE/Oozie hive action errors when running workflow

[#1]

Running CDH 5.5.1, Kerberos enabled cluster.

I  have a hive .sql script that runs fine interactively.  When I try to launch via Oozie using Hive action it errors out.  Log file below.  There isn't much info in the logs.  Any ideas on how to troubleshoot?

Error message:

 Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]

Full log below:

2016-03-10 02:55:22,474 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@:start:] Start action [0000035-160308051341496-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-10 02:55:22,476 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@:start:] [***0000035-160308051341496-oozie-oozi-W@:start:***]Action status=DONE
2016-03-10 02:55:22,476 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@:start:] [***0000035-160308051341496-oozie-oozi-W@:start:***]Action updated in DB!
2016-03-10 02:55:22,521 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] Start action [0000035-160308051341496-oozie-oozi-W@hive2-7b7a] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-10 02:55:25,848 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] checking action, hadoop job ID [job_1457413978008_0096] status [RUNNING]
2016-03-10 02:55:25,849 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] [***0000035-160308051341496-oozie-oozi-W@hive2-7b7a***]Action status=RUNNING
2016-03-10 02:55:25,849 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] [***0000035-160308051341496-oozie-oozi-W@hive2-7b7a***]Action updated in DB!
2016-03-10 02:55:47,945 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[myhive2server.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] callback for action [0000035-160308051341496-oozie-oozi-W@hive2-7b7a]
2016-03-10 02:55:48,076 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] action completed, external ID [job_1457413978008_0096]
2016-03-10 02:55:48,082 WARN org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
2016-03-10 02:55:48,113 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@hive2-7b7a] ERROR is considered as FAILED for SLA
2016-03-10 02:55:48,137 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@Kill] Start action [0000035-160308051341496-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-03-10 02:55:48,139 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@Kill] [***0000035-160308051341496-oozie-oozi-W@Kill***]Action status=DONE
2016-03-10 02:55:48,140 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[myhive2server.com] USER[myuser] GROUP[-] TOKEN[] APP[dev-epm-rmt-test-hive] JOB[0000035-160308051341496-oozie-oozi-W] ACTION[0000035-160308051341496-oozie-oozi-W@Kill] [***0000035-160308051341496-oozie-oozi-W@Kill***]Action updated in DB!

workflow.xml below:

<workflow-app name="dev-epm-rmt-test-hive" xmlns="uri:oozie:workflow:0.5">
  <global>
            <configuration>
                <property>
                    <name>mapreduce.job.queuename</name>
                    <value>root.epm</value>
                </property>
            </configuration>
  </global>
  <credentials>
    <credential name="hive2" type="hive2">
      <property>
        <name>hive2.jdbc.url</name>
        <value>jdbc:hive2://myhive2server.com:10000/default</value>
      </property>
      <property>
        <name>hive2.server.principal</name>
        <value>hive/myhive2server.com@MYREALM.ORG</value>
      </property>
    </credential>
  </credentials>
    <start to="hive2-7b7a"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="hive2-7b7a" cred="hive2">
        <hive2 xmlns="uri:oozie:hive2-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <jdbc-url>jdbc:hive2://myhive2server.com:10000/default</jdbc-url>
            <password>hive2</password>
            <script>/code/contacts.sql</script>
            <file>/code/contacts.sql#contacts.sql</file>
        </hive2>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>
 

[#2 · Accepted solution]

I was able to solve this by finding the URL to the Job History Server UI.  Stdout and Stderr can be sen here.

[#3]

How did you use the url to resolve the issue? Did you identify a root cause and resolution action? I'm experiencing similar error

[#4]

I think there was a bug in 5.5.1 where you could not review standard out and standard err outputs of shell scripts and hive scripts called from an oozie workflow.  That was fixed in later versions. The job history was used as a workaround to review the error messages and find root cause.

If you are calling the hive action/script from a shell script instead of a hive action you will also need to do the following.
1.Use kinit (if running a secure cluster) and kinit to an account that has access to the hive schema.
2.You must set the mapreduce.job.credentials.binary to $HADOOP_TOKEN_FILE_LOCATION in the hive script;

Sample code below:

HIVEQUERY='hive -S -e "SET mapreduce.job.credentials.binary=$HADOOP_TOKEN_FILE_LOCATION;set mapred.job.queue.name=myquename;SELECT count(*) FROM schema.table;"'

[#5]

Thanks for your response. In my case I'm using a simple HIVE action with no shell script. I'll dig into this some more, hopefully I can find the root cause in my case. I'll report if I find anything that would be useful to a large audience

[#6]

One last thing worth mentioning.  If you are using a hive action, you will need to ensure that you have the HiveServer2 URL identified on the properties tab of Hue/oozie for the action.  

E.g., jdbc:hive2://<HIVE_SERVER_HOSTNAME>:10000/default

[#7]

You were right on point. I didn't have this property specified and when I initially added it I forgot to save it. Got it working. Gracias!!!