Support Questions

Find answers, ask questions, and share your expertise

Launcher ERROR - Pig.Main

Contributor

I am trying to run a Oozie job that runs a pig script. My job.properties file is local and my workflow.xml is in HDFS. Below is my properties file.

nameNode=hdfs://192.168.56.104:8020 jobTracker=192.168.56.104:8050 queueName=default flattenRoot=Flatten_Tweet2 oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/oozie/${flattenRoot} oozie.action.sharelib.for.pig=pig,hcatalog,hive

and here is my workflow file.

<workflow-app name='pig-wf' xmlns="uri:oozie:workflow:0.3"> <start to='pig-node'/> <action name='pig-node'> <pig> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <script>Flatten_Tweet2.pig</script> <file>lib/hive-site.xml</file> </pig> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Pig failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>

I keep getting the below error no matter what I try.

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]

Any help would be apprecated. Thanks

1 ACCEPTED SOLUTION

Contributor

After a lot of trial and error, reading documentation and beers. I was able to figure out that the issue was the name of my pig jar. I renamed it to pig.jar in the lib folder, backed out all changes to my pig-env,sh and my oozie job ran and succeeded. Thank you to everyone for your help and especially to Jitendra for taking the time to give suggestions.

View solution in original post

16 REPLIES 16

Super Guru
@Montrial Harrell

Can you please check oozie launcher logs? Please refer this article.

You can post logs here so that we can have a look at it, most probably "stderr" part of launcher logs should give you a hint.

Contributor

I have took a look at the stderr logs and got this as an output. Does this make sense?

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/23/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/44/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Details at logfile: /hadoop/yarn/local/usercache/mbharrel/appcache/application_1463627394303_0038/container_e16_1463627394303_0038_01_000002/pig-job_1463627394303_0038.log

Contributor

Actually that is not the issue. When I looked down in the log I found the below. I will investigate but appreciate any guidance as well.

ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath).

@Montrial Harrell

Seems like pig doesn't know where to get conf files, Can you please set below env properties inside pig-env.sh and run again?

export HADOOP_CONF_DIR=$HADOOP_CONF_DIR:/etc/hadoop/conf

export PIG_CLASSPATH=$PIG_CLASSPATH:$HADOOP_CONF_DIR

Also please let us know your hdp version and did you changed any properties in cluster recently? I think this shouldn't be default behavior.

@Montrial Harrell

Thanks for sharing that info, can you please run below command and see which pig jar having these conf files.

grep -iR hadoop-site.xml /usr/hdp/<version>/pig/

@Montrial Harrell

Please check for core-site.xml also.

grep -iR core-site.xml /usr/hdp/<version>/pig/

If you don't see any jar file in the output then I think you can try copying these two xml from /etc/hadoop/conf to pig/conf dir and see if that resolve the issue.

Contributor

I did not get any pig jars when I ran grep -iR core-site.xml /usr/hdp/2.4.0.0-169/pig. The only output was the below. Should I copy the xml files to the pig/conf as you instructed?

/usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"core-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-4247: S3 properties are not picked up from core-site.xml in local mode (cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3145: Parameters in core-site.xml and mapred-site.xml are not correctly substituted (cheolsoo)

@Montrial Harrell

Yes, please try.

@Montrial Harrell

Yes, thats fine.

Contributor

I have copied both files from /etc/hadoop/conf to /usr/hdp/2.4.0.0-169/pig/conf and chmod to 755. Is this correct?

Contributor

The job still fails with the same error in the log oozie log.

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain]

Contributor

I am running the HDP Sandbox. My hadoop version command returns the below and I have not changed anything to my recollection about the environment.

hadoop version Hadoop 2.7.1.2.4.0.0-169 Subversion git@github.com:hortonworks/hadoop.git -r 26104d8ac833884c8776473823007f176854f2eb Compiled by jenkins on 2016-02-10T06:18Z Compiled with protoc 2.5.0 From source with checksum cf48a4c63aaec76a714c1897e2ba8be6 This command was run using /usr/hdp/2.4.0.0-169/hadoop/hadoop-common-2.7.1.2.4.0.0-169.jar

Contributor

Here is the output Jitendra. Do you see anything wrong?

/usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"hadoop-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3200: MiniCluster should delete hadoop-site.xml on shutDown (prkommireddi via cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-2491: Pig docs still mention hadoop-site.xml (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-1791: System property mapred.output.compress, but pig-cluster-hadoop-site.xml doesn't (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-1186: Pig do not take values in "pig-cluster-hadoop-site.xml" (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt: 'pig-cluster-hadoop-site.xml' in the non HOD case just like it does in the /usr/hdp/2.4.0.0-169/pig/RELEASE_NOTES.txt:variable to point to the directory with your hadoop-site.xml file and then run

Contributor

Here is what my new pig-env.sh looks like.

JAVA_HOME=/usr/lib/jvm/java HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client} export HADOOP_CONF_DIR=$HADOOP_CONF_DIR:/etc/hadoop/conf export PIG_CLASSPATH=$PIG_CLASSPATH:$HADOOP_CONF_DIR if [ -d "/usr/lib/tez" ]; then PIG_OPTS="$PIG_OPTS -Dmapreduce.framework.name=yarn" fi

Contributor

After a lot of trial and error, reading documentation and beers. I was able to figure out that the issue was the name of my pig jar. I renamed it to pig.jar in the lib folder, backed out all changes to my pig-env,sh and my oozie job ran and succeeded. Thank you to everyone for your help and especially to Jitendra for taking the time to give suggestions.

New Contributor

Can you please mention the details and the location of the file you renamed?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.