question Re: location of hadoop and pig logs in Archives of Support Questions (Read Only)

location of hadoop and pig logs

amit_29sharmait — Sun, 03 Apr 2016 08:58:33 GMT

My pig script failed when I am trying to do order by. I want to check the logs but I can't see the location of the logs. I got to know from the post that it should be set in the file (/etc/hadoop/hadoop-env.sh) but the file has everything commented as shown below. Does that mean it is not being written? how can I set those values to look into the logs as my scripts is failing again and again and I want to check the logs to troubleshoot it. also my file ( hadoop-env.sh)seems to be at different path(/etc/hadoop/conf.install) is there anything wrong ??

# On secure datanodes, user to run the datanode as after dropping privileges.

# This **MUST** be uncommented to enable secure HDFS if using privileged ports # to provide authentication of data transfer protocol. This **MUST NOT** be

# defined if SASL is configured for authentication of data transfer protocol # using non-privileged ports.

#export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER} # Where log files are stored. $HADOOP_HOME/logs by default. #export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER

Re: location of hadoop and pig logs

aervits — Sun, 03 Apr 2016 09:24:05 GMT

Your pig job execution is set in pig.properties file. By default, your failed job log is saved in the same directory you're in. Log file is prefixed with pig_. You can override that in Ambari configs or passing arguments at the time you execute job. Review the pig.properties section of pig wiki and here's a sample pig properties file from trunk. https://svn.apache.org/repos/asf/pig/trunk/conf/pig.properties

https://pig.apache.org/docs/r0.15.0/start.html#properties

Re: location of hadoop and pig logs

amit_29sharmait — Sun, 03 Apr 2016 10:25:25 GMT

yes, you are right. It is creating logs into the the same file but it is when a pig script failing. It is not creating log where map reduce is failing. I need to analyse map reduce logs in order to resolve the issue.

Re: location of hadoop and pig logs

Former Member — Sun, 03 Apr 2016 11:55:46 GMT

Hello Amit, As you mentioned that you "need to analyse map reduce logs in order to resolve the issue."

So did you check the following dir to find the mapreduce logs?

/var/log/hadoop-mapreduce/mapred/ (for History logs)

/var/log/hadoop-yarn/yarn (for hadoop-mapreduce.jobsummary.log)

Re: location of hadoop and pig logs

amit_29sharmait — Sun, 03 Apr 2016 13:15:28 GMT

Thanks this is what i was looking for.... Although this logs does not give much informaiton. Do you know if there is way to make log more readable.? or probably i'll have to ask another question.

Re: location of hadoop and pig logs

Former Member — Sun, 03 Apr 2016 13:33:49 GMT

If you want to debug the Jobs that are failing then you might enable the DEBUG something like following:

hadoop jar hadoop-mapreduce-examples.jar TestCount -Dyarn.app.mapreduce.am.log.level=TRACE -Dmapreduce.map.log.level=TRACE -Dmapreduce.reduce.log.level=TRACE

Re: location of hadoop and pig logs

aervits — Sun, 03 Apr 2016 18:19:08 GMT

Understood, it was not clear from your question, to me it sounded more like configuration question.

Re: location of hadoop and pig logs

amit_29sharmait — Tue, 05 Apr 2016 01:28:58 GMT

Can you or someone help me with the main question, I am not able to figure out the reason for job failure. after googling it i think it may be related to the variable not set ..I am adding more details about how the variables are showing null... Could you please help..

Re: location of hadoop and pig logs

amit_29sharmait — Tue, 05 Apr 2016 01:29:04 GMT

[hdfs@ bin]$ ls -lrt
total 32
-rwxr-xr-x. 1 root root 1857 Dec 15 22:25 rcc
-rwxr-xr-x. 1 root root 6594 Dec 15 22:25 hadoop.distro 
-rwxr-xr-x. 1 root root  775 Dec 15 22:26 yarn 
-rwxr-xr-x. 1 root root  782 Dec 15 22:26 mapred 
-rwxr-xr-x. 1 root root  775 Dec 15 22:26 hdfs 
-rwxr-xr-x. 1 root root  807 Dec 15 22:26 hadoop-fuse-dfs
-rwxr-xr-x. 1 root root  772 Apr  4 14:21

hadoop
[hdfs@ bin]$ cat hadoop
#!/bin/bash
# Autodetect JAVA_HOME if not defined
if [ -e /usr/libexec/bigtop-detect-javahome ]; then
  . /usr/libexec/bigtop-detect-javahome
elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
  . /usr/lib/bigtop-utils/bigtop-detect-javahome
fi
export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/2.3.4.0-3485/hadoop}
export HADOOP_MAPRED_HOME=${HADOOP_MAPRED_HOME:-/usr/hdp/2.3.4.0-3485/hadoop-mapreduce}
export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-/usr/hdp/2.3.4.0-3485/hadoop-yarn}
export HADOOP_LIBEXEC_DIR=${HADOOP_HOME}/libexec
export HDP_VERSION=${HDP_VERSION:-2.3.4.0-3485}
export HADOOP_OPTS="${HADOOP_OPTS} -Dhdp.version=${HDP_VERSION}"
export YARN_OPTS="${YARN_OPTS} -Dhdp.version=${HDP_VERSION}"
exec /usr/hdp/2.3.4.0-3485//hadoop/bin/hadoop.distro "$@"

[hdfs@ip- bin]$ echo $HADOOP_HOME
[hdfs@ip-bin]$ pwd
/usr/hdp/current/hadoop-client/bin
[hdfs@ip-bin]$

Re: location of hadoop and pig logs

aervits — Tue, 05 Apr 2016 01:42:15 GMT

provide your script.

Re: location of hadoop and pig logs

sayankh — Mon, 11 Apr 2016 23:17:36 GMT

@Amit Sharma

You can review the failed job log from ResourceManager UI.

Ambari > YARN > Quick Links > ResourceManager UI