Created 02-25-2016 03:26 AM
Using HDP sandbox 2.3.2 on VMWare. Trying to run very simple one line hive query using oozie.
1. Designed oozie workflow using Hue.
2. Added Hive-site as Job xml and as well as file resource.
3. Copied all lib/hive, lib/hive2 etc to /user/oozie/share/ hdfs path
4. Done few addl things as mentioned in https://community.hortonworks.com/questions/8552/hive-action-in-failing-in-oozie.html
But my Hive action result in oozie web ui is
ERROR. When i goto see job logs as mentioned in this post - https://community.hortonworks.com/articles/9148/troubleshooting-an-oozie-flow.html, i see status as Succeeded.
"2016-02-25 02:49:16,934 INFO jobhistory.JobSummary (HistoryFileManager.java:moveToDone(376)) - jobId=job_1456308349646_0117,submitTime=1456368470425,launchTime=1456368497066,firstMapTaskLaunchTime=1456368502963,firstReduceTaskLaunchTime=0,finishTime=1456368554934,resourcesPerMap=250,resourcesPerReduce=0,numMaps=1,numReduces=0,user=hue,queue=default,status=SUCCEEDED,mapSlotSeconds=51,reduceSlotSeconds=0,jobName=oozie:launcher:T\=hive:W\=FirstHueWf:A\=Iot:ID\=0000037-160224100502172-oozie-oozi-W"
Why are we getting Hive action as Failed/Killed when underlying job is completed ?
*Though underlying job says it is complete, i didnt get result of my Hive query.
Created 02-25-2016 03:30 AM
yarn log -applicationid job_1456308349646_0117 --> This can help you resolve the issue
Created 02-26-2016 10:00 AM
Tried peeking at Job logs and application logs. Oozie job has 2 hive actions. First HIVE action simply creates table and loads data from another table. Here 2 jobs are being created - One as Mapreduce and one as Tez. I see both jobs getting compelted w/o any issues and even when i look in HIVE, my new table and data is there. But Oozie is not moving to next hive action and throws 12 error code. This is what i found in oozie's job logs.
2016-02-26 07:03:21,459 WARN HiveActionExecutor:523 - SERVER[sandbox.hortonworks.com] USER[hue] GROUP[-] TOKEN[] APP[FirstHueWf] JOB[0000051-160224100502172-oozie-oozi-W] ACTION[0000051-160224100502172-oozie-oozi-W@test] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [12] 2016-02-26 07:03:21,611 INFO ActionEndXCommand:520 - SERVER[sandbox.hortonworks.com] USER[hue] GROUP[-] TOKEN[] APP[FirstHueWf] JOB[0000051-160224100502172-oozie-oozi-W] ACTION[0000051-160224100502172-oozie-oozi-W@test] ERROR is considered as FAILED for SLA
I suspect oozie for some reason is not able to move to next DAG action in HDP 2.3.2.
Created 02-25-2016 03:40 AM
Additionally i saw this Warning in Job history for 8 jars...is this the issue ?
org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://sandbox.hortonworks.com:8020/user/oozie/share/lib/lib_20151027144433/hive/aws-java-sdk-1.7.4.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://sandbox.hortonworks.com:8020/user/oozie/share/lib/lib_20151027144433/oozie/aws-java-sdk-1.7.4.jar This will be an error in Hadoop 2
Created 02-25-2016 04:31 AM
I'm almost confident it's a sharelib issue. Try to copy files that do not exist in new shareelib from old. Make backups before you do anything
Created 02-26-2016 10:23 AM
Here's my sample for Pig with hcatalog. It would be the same as Hive. https://github.com/dbist/oozie directory called apps/hcatalog.
Replace sharelib.for.pig with Hive
Created 07-06-2016 05:15 PM
@Artem Ervits I have the same issue I put jdbc jar file in share lib but I still get the same error. Do you have any suggestion?
Created 07-06-2016 05:30 PM
@hoda moradi you need to update oozie with new share lib https://oozie.apache.org/docs/4.2.0/DG_CommandLineTool.html#Update_system_sharelib
Created 07-06-2016 06:44 PM
@Artem Ervits I updated the share lib but I still get the same error.
Created 07-06-2016 04:58 PM
@Ramanathan Ramaiyah did you solve this issue? I get the same error recently.