Monitoring Yarn application logs

[#1]

We have a cloudera infrastructure that runs Java and Python jobs.

Is there an established and recommended solution to monitor our application logs generated by these Yarn based Jobs? Traditionally if you have a stand alone Java application, then we use log scrapers/monitors tools, that send emails when they find matches for standard expressions like ERROR etc.  So wondering what is the similar tool for Cloudera Application?

[#2]

@SeanU 

You can select the role group for the YARN role for which you want to configure log events, or search for "Rules to Extract Events from Log Files". These generate the events for the service and you can set alerts for these log events. Alert publisher can be configured to generate emails for alerts.

Refer

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_monitoring_settings.html#cmug_...

Hope this helps,
Paras
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

[#3]

Thanks @paras Does this help with logs generated by the applications? For e.g. say my java application uses slf4j/log4j2 and writes an ERROR or Exception into the logs, can this monitoring be used to pick them up and send an email? Thanks.

[#4]

@SeanU 

This level of detailed log scanning and alert functionality is not available.

The existing service role logs for which rules can be set will not contain each application exceptions logged since detailed information is present in the application logs. 

You can check the available job history server logs and resource manager logs available to check if the logged in information during application run time helps serve your purpose.

[#5]

Hi @SeanU 

Thank you for reaching out to community!

To better assist you with this post, could you tell us if you are using Ambari or Cloudera Manager to manage your cluster? 

Yarn application logs can be monitored using Yarn Web UI. For more details, please check [1] and [2] based on the distribution you are using.

There are many other tools available for monitoring Hadoop clusters as mentioned in [3]   

[1] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_yarn_applications.html

[2] https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/data-operating-system/content/monitoring_clust...

[3] https://community.cloudera.com/t5/Support-Questions/what-is-the-most-best-monitoring-tool-for-hadoop... .

[#6]

@Madhur we are using Cloudera Manager. But as I was asking, my question is not about going to the portal and checking logs, but a log monitor that can generate emails for effective production support, like it is commonly done in stand alone applications.