Created 05-21-2016 04:33 AM
I am trying to run a Oozie job that runs a pig script. My job.properties file is local and my workflow.xml is in HDFS. Below is my properties file.
nameNode=hdfs://192.168.56.104:8020 jobTracker=192.168.56.104:8050 queueName=default flattenRoot=Flatten_Tweet2 oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/oozie/${flattenRoot} oozie.action.sharelib.for.pig=pig,hcatalog,hive
and here is my workflow file.
<workflow-app name='pig-wf' xmlns="uri:oozie:workflow:0.3"> <start to='pig-node'/> <action name='pig-node'> <pig> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <script>Flatten_Tweet2.pig</script> <file>lib/hive-site.xml</file> </pig> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Pig failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>
I keep getting the below error no matter what I try.
Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]
Any help would be apprecated. Thanks
Created 05-22-2016 05:51 AM
After a lot of trial and error, reading documentation and beers. I was able to figure out that the issue was the name of my pig jar. I renamed it to pig.jar in the lib folder, backed out all changes to my pig-env,sh and my oozie job ran and succeeded. Thank you to everyone for your help and especially to Jitendra for taking the time to give suggestions.
Created 05-21-2016 07:09 PM
Can you please check oozie launcher logs? Please refer this article.
You can post logs here so that we can have a look at it, most probably "stderr" part of launcher logs should give you a hint.
Created 05-21-2016 09:29 PM
I have took a look at the stderr logs and got this as an output. Does this make sense?
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/23/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/44/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Details at logfile: /hadoop/yarn/local/usercache/mbharrel/appcache/application_1463627394303_0038/container_e16_1463627394303_0038_01_000002/pig-job_1463627394303_0038.log
Created 05-21-2016 09:48 PM
Actually that is not the issue. When I looked down in the log I found the below. I will investigate but appreciate any guidance as well.
ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath).
Created 05-21-2016 10:05 PM
Seems like pig doesn't know where to get conf files, Can you please set below env properties inside pig-env.sh and run again?
export HADOOP_CONF_DIR=$HADOOP_CONF_DIR:/etc/hadoop/conf
export PIG_CLASSPATH=$PIG_CLASSPATH:$HADOOP_CONF_DIR
Also please let us know your hdp version and did you changed any properties in cluster recently? I think this shouldn't be default behavior.
Created 05-21-2016 10:42 PM
Thanks for sharing that info, can you please run below command and see which pig jar having these conf files.
grep -iR hadoop-site.xml /usr/hdp/<version>/pig/
Created 05-21-2016 10:55 PM
Please check for core-site.xml also.
grep -iR core-site.xml /usr/hdp/<version>/pig/
If you don't see any jar file in the output then I think you can try copying these two xml from /etc/hadoop/conf to pig/conf dir and see if that resolve the issue.
Created 05-21-2016 11:03 PM
I did not get any pig jars when I ran grep -iR core-site.xml /usr/hdp/2.4.0.0-169/pig. The only output was the below. Should I copy the xml files to the pig/conf as you instructed?
/usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"core-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-4247: S3 properties are not picked up from core-site.xml in local mode (cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3145: Parameters in core-site.xml and mapred-site.xml are not correctly substituted (cheolsoo)
Created 05-21-2016 11:08 PM
Yes, please try.
Created 05-21-2016 11:38 PM
Yes, thats fine.