10-19-2015 01:17 PM
We were having a process in CDH4 which was combination of Shell script and MR. Which was working fine in CDH4. We recently moved to CDH5.3 where we replace jobtracker with Yarn our current process fail.
Reason for failure is we were reading mapreduce status in script from logs. In CDH4 MR logs are written in [output]/_logs/history file where as in CDH5 all logs are moved to common location. To read the logs in CDH5 we need to know the job id which can only be get from logs. So it’s a catch 22 problem.
My question is can we force MR2 (Yarn) to write same logs on same location as we were doing in CDH4. This way we need not to make major changes to the script. Because our script is heavily dependent of content of logs to get jobid, jobstatus and number of records process to make post process decisions.
Thanks in Advance for the help.
11-08-2015 10:14 AM