Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is the best way to collect logs from EC2 cluster while using CM?

What is the best way to collect logs from EC2 cluster while using CM?

New Contributor

Hi,

 

I am trying to collect all the Hadoop, HBase and ZooKeeper logs from each node in my cluster to some central location, lets say my local machine. What is the best way to achieve this? Does Cloudera or Cloudera Manager offer any such facilty?

 

Thanks.

2 REPLIES 2

Re: What is the best way to collect logs from EC2 cluster while using CM?

Master Guru
Cloudera Manager already does log searches from its Logs tab on top. You can also download a log file from CM by visiting an instance's page and the log file link under it.

For example:

CM -> HDFS -> Instances -> NameNode -> Process -> "Role Log Details" -> Download Log File.

AFAICT, there's no way to automate this to be collected to your own location via CM, so you'll need to build your own tooling for such a need.
Highlighted

Re: What is the best way to collect logs from EC2 cluster while using CM?

You can get log information via the API, as documented in this endpoint:
http://cloudera.github.io/cm_api/apidocs/v6/path__clusters_-clusterName-_services_-serviceName-_role...

This will give you the most recent role logs (I believe since it last started). I don't think there's a great way to stream these to some other location via the API, but you could probably set up flume or something to do that.