question Re: Launcher ERROR - Pig.Main in Support Questions

Launcher ERROR - Pig.Main

Montrial — Sat, 21 May 2016 11:33:02 GMT

I am trying to run a Oozie job that runs a pig script. My job.properties file is local and my workflow.xml is in HDFS. Below is my properties file.

nameNode=hdfs://192.168.56.104:8020 jobTracker=192.168.56.104:8050 queueName=default flattenRoot=Flatten_Tweet2 oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/oozie/${flattenRoot} oozie.action.sharelib.for.pig=pig,hcatalog,hive

and here is my workflow file.

<workflow-app name='pig-wf' xmlns="uri:oozie:workflow:0.3"> <start to='pig-node'/> <action name='pig-node'> <pig> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <script>Flatten_Tweet2.pig</script> <file>lib/hive-site.xml</file> </pig> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Pig failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>

I keep getting the below error no matter what I try.

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]

Any help would be apprecated. Thanks

Re: Launcher ERROR - Pig.Main

KuldeepK — Sun, 22 May 2016 02:09:23 GMT

@Montrial Harrell

Can you please check oozie launcher logs? Please refer this article.

You can post logs here so that we can have a look at it, most probably "stderr" part of launcher logs should give you a hint.

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 04:29:44 GMT

I have took a look at the stderr logs and got this as an output. Does this make sense?

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/23/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/44/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Details at logfile: /hadoop/yarn/local/usercache/mbharrel/appcache/application_1463627394303_0038/container_e16_1463627394303_0038_01_000002/pig-job_1463627394303_0038.log

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 04:48:45 GMT

Actually that is not the issue. When I looked down in the log I found the below. I will investigate but appreciate any guidance as well.

ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath).

Re: Launcher ERROR - Pig.Main

jyadav — Sun, 22 May 2016 05:05:21 GMT

@Montrial Harrell

Seems like pig doesn't know where to get conf files, Can you please set below env properties inside pig-env.sh and run again?

export HADOOP_CONF_DIR=$HADOOP_CONF_DIR:/etc/hadoop/conf

export PIG_CLASSPATH=$PIG_CLASSPATH:$HADOOP_CONF_DIR

Also please let us know your hdp version and did you changed any properties in cluster recently? I think this shouldn't be default behavior.

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 05:28:01 GMT

I am running the HDP Sandbox. My hadoop version command returns the below and I have not changed anything to my recollection about the environment.

hadoop version Hadoop 2.7.1.2.4.0.0-169 Subversion git@github.com:hortonworks/hadoop.git -r 26104d8ac833884c8776473823007f176854f2eb Compiled by jenkins on 2016-02-10T06:18Z Compiled with protoc 2.5.0 From source with checksum cf48a4c63aaec76a714c1897e2ba8be6 This command was run using /usr/hdp/2.4.0.0-169/hadoop/hadoop-common-2.7.1.2.4.0.0-169.jar

Re: Launcher ERROR - Pig.Main

jyadav — Sun, 22 May 2016 05:42:19 GMT

@Montrial Harrell

Thanks for sharing that info, can you please run below command and see which pig jar having these conf files.

grep -iR hadoop-site.xml /usr/hdp/<version>/pig/

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 05:46:35 GMT

Here is the output Jitendra. Do you see anything wrong?

/usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"hadoop-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3200: MiniCluster should delete hadoop-site.xml on shutDown (prkommireddi via cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-2491: Pig docs still mention hadoop-site.xml (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-1791: System property mapred.output.compress, but pig-cluster-hadoop-site.xml doesn't (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-1186: Pig do not take values in "pig-cluster-hadoop-site.xml" (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt: 'pig-cluster-hadoop-site.xml' in the non HOD case just like it does in the /usr/hdp/2.4.0.0-169/pig/RELEASE_NOTES.txt:variable to point to the directory with your hadoop-site.xml file and then run

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 05:51:12 GMT

Here is what my new pig-env.sh looks like.

JAVA_HOME=/usr/lib/jvm/java HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client} export HADOOP_CONF_DIR=$HADOOP_CONF_DIR:/etc/hadoop/conf export PIG_CLASSPATH=$PIG_CLASSPATH:$HADOOP_CONF_DIR if [ -d "/usr/lib/tez" ]; then PIG_OPTS="$PIG_OPTS -Dmapreduce.framework.name=yarn" fi

Re: Launcher ERROR - Pig.Main

jyadav — Sun, 22 May 2016 05:55:44 GMT

@Montrial Harrell

Please check for core-site.xml also.

grep -iR core-site.xml /usr/hdp/<version>/pig/

If you don't see any jar file in the output then I think you can try copying these two xml from /etc/hadoop/conf to pig/conf dir and see if that resolve the issue.

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 06:03:21 GMT

I did not get any pig jars when I ran grep -iR core-site.xml /usr/hdp/2.4.0.0-169/pig. The only output was the below. Should I copy the xml files to the pig/conf as you instructed?

/usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"core-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-4247: S3 properties are not picked up from core-site.xml in local mode (cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3145: Parameters in core-site.xml and mapred-site.xml are not correctly substituted (cheolsoo)

Re: Launcher ERROR - Pig.Main

jyadav — Sun, 22 May 2016 06:08:42 GMT

@Montrial Harrell

Yes, please try.

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 06:30:24 GMT

I have copied both files from /etc/hadoop/conf to /usr/hdp/2.4.0.0-169/pig/conf and chmod to 755. Is this correct?

Re: Launcher ERROR - Pig.Main

jyadav — Sun, 22 May 2016 06:38:54 GMT

@Montrial Harrell

Yes, thats fine.

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 06:56:25 GMT

The job still fails with the same error in the log oozie log.

Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain]

Re: Launcher ERROR - Pig.Main

Montrial — Sun, 22 May 2016 12:51:10 GMT

After a lot of trial and error, reading documentation and beers. I was able to figure out that the issue was the name of my pig jar. I renamed it to pig.jar in the lib folder, backed out all changes to my pig-env,sh and my oozie job ran and succeeded. Thank you to everyone for your help and especially to Jitendra for taking the time to give suggestions.

Re: Launcher ERROR - Pig.Main

narayantalk2u_ — Wed, 08 Aug 2018 16:05:42 GMT

Can you please mention the details and the location of the file you renamed?