Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to suppress the logging messages generated by hadoop on the console

avatar

i am using hadoop 2.5.2.I am new to hadoop and map reduce . I m trying a map reduce code to perform sentiment analysis.But when i execute the code, lot of logging messages are generated by hadoop on the console.Thats the reason it takes a lot of time in giving analysis result for each sentence

Following are the logging messages....

2016-05-1223:03:05,396 INFO  jvm.JvmMetrics(JvmMetrics.java:init(71))-Cannot initialize JVM Metrics with processName=JobTracker, sessionId=- already initialized
2016-05-1223:03:05,397 INFO  jvm.JvmMetrics(JvmMetrics.java:init(71))-Cannot initialize JVM Metrics with processName=JobTracker, sessionId=- already initialized
2016-05-1223:03:05,400 WARN  mapreduce.JobSubmitter(JobSubmitter.java:copyAndConfigureFiles(150))-Hadoop command-line option parsing not performed.Implement the Toolinterface and execute your application with ToolRunner to remedy this.2016-05-1223:03:05,403 WARN  mapreduce.JobSubmitter(JobSubmitter.java:copyAndConfigureFiles(259))-No job jar file set.User classes may not be found.SeeJob or Job#setJar(String).2016-05-1223:03:05,412 INFO  mapred.FileInputFormat(FileInputFormat.java:listStatus(247))-Total input paths to process :32016-05-1223:03:05,421 INFO  mapreduce.JobSubmitter(JobSubmitter.java:submitJobInternal(396))- number of splits:32016-05-1223:03:05,430 INFO  mapreduce.JobSubmitter(JobSubmitter.java:printTokens(479))-Submitting tokens for job: job_local1427897879_0120
2016-05-1223:03:05,444 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;Ignoring.2016-05-1223:03:05,445 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: hadoop.tmp.dir;Ignoring.2016-05-1223:03:05,445 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/staging/hduser1427897879/.staging/job_local1427897879_0120/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;Ignoring.2016-05-1223:03:05,482 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;Ignoring.2016-05-1223:03:05,482 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: hadoop.tmp.dir;Ignoring.2016-05-1223:03:05,483 WARN  conf.Configuration(Configuration.java:loadProperty(2368))- file:/home/hduser/workspace/Test1/build/test/mapred/local/localRunner/hduser/job_local1427897879_0120/job_local1427897879_0120.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;Ignoring.2016-05-1223:03:05,483 INFO  mapreduce.Job(Job.java:submit(1289))-The url to track the job: http://localhost:8080/2016-05-1223:03:05,483 INFO  mapreduce.Job(Job.java:monitorAndPrintJob(1334))-Running job: job_local1427897879_0120
2016-05-1223:03:05,483 INFO  mapred.LocalJobRunner(LocalJobRunner.java:createOutputCommitter(471))-OutputCommitter set in config null2016-05-1223:03:05,484 INFO  mapred.LocalJobRunner(LocalJobRunner.java:createOutputCommitter(489))-OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter2016-05-1223:03:05,485 INFO  mapred.LocalJobRunner(LocalJobRunner.java:runTasks(448))-Waitingfor map tasks
2016-05-1223:03:05,485 INFO  mapred.LocalJobRunner(LocalJobRunner.java:run(224))-Starting task: attempt_local1427897879_0120_m_000000_0
2016-05-1223:03:05,486 INFO  mapred.Task(Task.java:initialize(587))-UsingResourceCalculatorProcessTree:[]2016-05-1223:03:05,486 INFO  mapred.MapTask(MapTask.java:updateJobWithSplit(462))-Processing split: file:/home/hduser/workspace/Test1/training/pool.txt:0+179612016-05-1223:03:05,487 INFO  mapred.MapTask(MapTask.java:runOldMapper(416))- numReduceTasks:12016-05-1223:03:05,487 INFO  mapred.MapTask(MapTask.java:createSortingCollector(388))-Map output collector class= org.apache.hadoop.mapred.MapTask$MapOutputBuffer2016-05-1223:03:05,513 INFO  mapred.MapTask(MapTask.java:setEquator(1182))-(EQUATOR)0 kvi 26214396(104857584)2016-05-1223:03:05,513 INFO  mapred.MapTask(MapTask.java:init(975))- mapreduce.task.io.sort.mb:100

I tried using following configuration suggested in some forum but its not working.I have tried setting following in hadoop-env.sh

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_ROOT_LOGGER="WARN,DRFA"

I have also tried editing the log4j.properties file by setting following values..

hadoop.root.logger=WARN,DRFA 
hadoop.log.dir=. 
hadoop.log.file=hadoop.log

But still i am not able to get rid of this logging messages generated by hadoop mapreduce during runtime execution.Its delaying my output on the console as well.Is their any workaround or any java code which i can embed in my code and suppress the messages.any suggestions,help..Anyone knows??

Thanks a lot!!

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar

@Vishal kharde

please check this.

mapreduce.output.fileoutputformat.compress to True in mapred-site.xml

avatar

I hope this link will help link

avatar

@Divakar Annapureddy I have tried the same as mentioned in the link (i hv mentioned the same changes in my problem discussed above) which u provided but its not working.

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login