Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Monitoring Yarn application logs

Highlighted

Monitoring Yarn application logs

New Contributor

We have a cloudera infrastructure that runs Java and Python jobs.

Is there an established and recommended solution to monitor our application logs generated by these Yarn based Jobs? Traditionally if you have a stand alone Java application, then we use log scrapers/monitors tools, that send emails when they find matches for standard expressions like ERROR etc.  So wondering what is the similar tool for Cloudera Application?

5 REPLIES 5
Highlighted

Re: Monitoring Yarn application logs

Expert Contributor

@SeanU 

 

You can select the role group for the YARN role for which you want to configure log events, or search for "Rules to Extract Events from Log Files". These generate the events for the service and you can set alerts for these log events. Alert publisher can be configured to generate emails for alerts.

 

Refer

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_monitoring_settings.html#cmug_...

 

Hope this helps,
Paras
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Highlighted

Re: Monitoring Yarn application logs

New Contributor

Thanks @paras Does this help with logs generated by the applications? For e.g. say my java application uses slf4j/log4j2 and writes an ERROR or Exception into the logs, can this monitoring be used to pick them up and send an email? Thanks.

Highlighted

Re: Monitoring Yarn application logs

Expert Contributor

@SeanU 

 

This level of detailed log scanning and alert functionality is not available.

The existing service role logs for which rules can be set will not contain each application exceptions logged since detailed information is present in the application logs. 

 

You can check the available job history server logs and resource manager logs available to check if the logged in information during application run time helps serve your purpose.

Re: Monitoring Yarn application logs

Moderator

Hi @SeanU 

 

Thank you for reaching out to community!

To better assist you with this post, could you tell us if you are using Ambari or Cloudera Manager to manage your cluster? 

Yarn application logs can be monitored using Yarn Web UI. For more details, please check [1] and [2] based on the distribution you are using.

There are many other tools available for monitoring Hadoop clusters as mentioned in [3]   

 

[1] https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_yarn_applications.html

[2] https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/data-operating-system/content/monitoring_clust...

[3] https://community.cloudera.com/t5/Support-Questions/what-is-the-most-best-monitoring-tool-for-hadoop... .


Madhuri Adipudi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Highlighted

Re: Monitoring Yarn application logs

New Contributor

@Madhur we are using Cloudera Manager. But as I was asking, my question is not about going to the portal and checking logs, but a log monitor that can generate emails for effective production support, like it is commonly done in stand alone applications.

Don't have an account?
Coming from Hortonworks? Activate your account here