how to suppress the logging messages generated by hadoop on the console

vishalkharde02 — Fri, 13 May 2016 03:31:26 GMT

i am using hadoop 2.5.2.I am new to hadoop and map reduce . I m trying a map reduce code to perform sentiment analysis.But when i execute the code, lot of logging messages are generated by hadoop on the console.Thats the reason it takes a lot of time in giving analysis result for each sentence

Following are the logging messages....

2016-05-1223:03:05,396 INFO  jvm.JvmMetrics(JvmMetrics.java:init(71))-Cannot initialize JVM Metrics with processName=JobTracker, sessionId=- already initialized
2016-05-1223:03:05,397 INFO  jvm.JvmMetrics(JvmMetrics.java:init(71))-Cannot initialize JVM Metrics with processName=JobTracker, sessionId=- already initialized
2016-05-1223:03:05,400 WARN  mapreduce.JobSubmitter(JobSubmitter.java:copyAndConfigureFiles(150))-Hadoop command-line option parsing not performed.Implement the Toolinterface and execute your application with ToolRunner to remedy this.2016-05-1223:03:05,403 WARN  mapreduce.JobSubmitter(JobSubmitter.java:copyAndConfigureFiles(259))-No job jar file set.User classes may not be found.SeeJob or Job#setJar(String).2016-05-1223:03:05,412 INFO  mapred.FileInputFormat(FileInputFormat.java:listStatus(247))-Total input paths to process :32016-05-1223:03:05,421 INFO  mapreduce.JobSubmitter(JobSubmitter.java:submitJobInternal(396))- number of splits:32016-05-1223:03:05,430 INFO  mapreduce.JobSubmitter(JobSubmitter.java:printTokens(479))-Submitting tokens for job: job_local1427897879_0120
2016-05-1223:03:05,444 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;Ignoring.2016-05-1223:03:05,445 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: hadoop.tmp.dir;Ignoring.2016-05-1223:03:05,445 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;Ignoring.2016-05-1223:03:05,482 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;Ignoring.2016-05-1223:03:05,482 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: hadoop.tmp.dir;Ignoring.2016-05-1223:03:05,483 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;Ignoring.2016-05-1223:03:05,483 INFO  mapreduce.Job(Job.java:submit(1289))-The url to track the job: http://localhost:8080/2016-05-1223:03:05,483 INFO  mapreduce.Job(Job.java:monitorAndPrintJob(1334))-Running job: job_local1427897879_0120
2016-05-1223:03:05,483 INFO  mapred.LocalJobRunner(LocalJobRunner.java:createOutputCommitter(471))-OutputCommitter set in config null2016-05-1223:03:05,484 INFO  mapred.LocalJobRunner(LocalJobRunner.java:createOutputCommitter(489))-OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter2016-05-1223:03:05,485 INFO  mapred.LocalJobRunner(LocalJobRunner.java:runTasks(448))-Waitingfor map tasks
2016-05-1223:03:05,485 INFO  mapred.LocalJobRunner(LocalJobRunner.java:run(224))-Starting task: attempt_local1427897879_0120_m_000000_0
2016-05-1223:03:05,486 INFO  mapred.Task(Task.java:initialize(587))-UsingResourceCalculatorProcessTree:[]2016-05-1223:03:05,486 INFO  mapred.MapTask(MapTask.java:updateJobWithSplit(462))-Processing split: file:/home/hduser/workspace/Test1/training/pool.txt:0+179612016-05-1223:03:05,487 INFO  mapred.MapTask(MapTask.java:runOldMapper(416))- numReduceTasks:12016-05-1223:03:05,487 INFO  mapred.MapTask(MapTask.java:createSortingCollector(388))-Map output collector class= org.apache.hadoop.mapred.MapTask$MapOutputBuffer2016-05-1223:03:05,513 INFO  mapred.MapTask(MapTask.java:setEquator(1182))-(EQUATOR)0 kvi 26214396(104857584)2016-05-1223:03:05,513 INFO  mapred.MapTask(MapTask.java:init(975))- mapreduce.task.io.sort.mb:100

I tried using following configuration suggested in some forum but its not working.I have tried setting following in hadoop-env.sh

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"

I have also tried editing the log4j.properties file by setting following values..

hadoop.root.logger=WARN,DRFA 
hadoop.log.dir=. 
hadoop.log.file=hadoop.log

But still i am not able to get rid of this logging messages generated by hadoop mapreduce during runtime execution.Its delaying my output on the console as well.Is their any workaround or any java code which i can embed in my code and suppress the messages.any suggestions,help..Anyone knows??

Thanks a lot!!

Re: how to suppress the logging messages generated by hadoop on the console

divakarreddy_a — Fri, 13 May 2016 03:38:56 GMT

@Vishal kharde

please check this.

mapreduce.output.fileoutputformat.compress to True in mapred-site.xml

Re: how to suppress the logging messages generated by hadoop on the console

vishalkharde02 — Fri, 13 May 2016 13:57:59 GMT

@Divakar Annapureddy . I have tried setting the above propertyu mentioned (mapreduce.output.fileoutputformat.compress) in mapred-site.xml but still no success in suppressing the log messages on the console.

Re: how to suppress the logging messages generated by hadoop on the console

divakarreddy_a — Fri, 13 May 2016 23:06:34 GMT

I hope this link will help link

Re: how to suppress the logging messages generated by hadoop on the console

vishalkharde02 — Sat, 14 May 2016 01:31:35 GMT

@Divakar Annapureddy I have tried the same as mentioned in the link (i hv mentioned the same changes in my problem discussed above) which u provided but its not working.

Re: how to suppress the logging messages generated by hadoop on the console

vishalkharde02 — Sat, 14 May 2016 13:44:50 GMT

I have found solution for this.

1.mapreduce.map.log.level can take values as OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting could be overridden if "mapreduce.job.log4j-properties-file" is set.

2. mapreduce.reduce.log.level can also take values as OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE and ALL. The setting could be overridden if "mapreduce.job.log4j-properties-file" is set.So its better to make sure that "mapreduce.job.log4j-properties-file" is not set.

We have to set following properties in the mapred-site.xml.

<property>
<name>mapreduce.map.log.level</name>
<value>OFF</value>
</property>

<property>
<name>mapreduce.reduce.log.level</name>
<value>OFF</value>
</property>

question Re: how to suppress the logging messages generated by hadoop on the console in Archives of Support Questions (Read Only)

how to suppress the logging messages generated by hadoop on the console

Re: how to suppress the logging messages generated by hadoop on the console

Re: how to suppress the logging messages generated by hadoop on the console

Re: how to suppress the logging messages generated by hadoop on the console

Re: how to suppress the logging messages generated by hadoop on the console

Re: how to suppress the logging messages generated by hadoop on the console