history task records disappear after hadoop reboot

[#1]

I run some mr tasks and can see the task status on yarn web. But when I reboot the hadoop cluster, the task records disappear. all the necessary config have been set and the jobhistory app was running too.

Any suggestion for debug? 

[#2]

Hi @Bro

Are you able to fetch logs using below?

1] yarn logs -applicationId <app_id> -appOwner <user>

2] are you able to see application_id in JHS or Cloudera>Yarn>Applications ?

Check this property once yarn.resourcemanager.max-completed-applications

Thanks!

[#3]

hi Asim,

Thanks for the reply.

I can see the log using the command while the hadoop cluster is running.

After I reboot the cluster, the history logs disappead on the page.

The config is as below:

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<!-- 历史服务器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop201:10020</value>
</property>

<!-- 历史服务器web端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop201:19888</value>
</property>

<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/opt/module/hadoop-3.1.3/logs/his_log/done</value>
<description>MR JobHistory Server管理的日志的存放位置,默认:/mr-history/done</description>
</property>

<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/opt/module/hadoop-3.1.3/logs/his_log</value>
<description>MapReduce作业产生的日志存放位置，默认值:/mr-history/tmp</description>
</property>

<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/opt/module/hadoop-3.1.3/logs/mr-stage-his</value>
<description></description>
</property>

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds</name>
<value>3600</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/opt/module/hadoop-3.1.3/logs/resource_manager_logs</value>
</property>

<!-- 设置日志聚集服务器地址 -->
<property>
<name>yarn.log.server.url</name>
<value>http://hadoop201:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>5184000</value>
</property>

much appreciated for the help.