@Bhapra,   Please post the information in the log specified in the screen shot so we can see if logging shows what the cause was. ... View more

Oops... clicked "POST" before telling you how to get the agent to dump the thread stacks to the agent log.  You can run the following:   kill -SIGQUIT `cat /var/run/cloudera-scm-agent/cloudera-scm-agent.pid`   This will not cause the agent to restart or anything so it won't impact processing. If you can run the kill -SIGQUIT a few times that would give us an idea of how the threads are progressing. ... View more

@zy001,   I agree that it is possible (and even likely) that the issue is more on the agent side.  What we really need to see is what the agent is doing when at the time it is having trouble connecting.  A great way to have a peek at the internals is to send a SIGQUIT signal to the agent which will trigger it to dump thread stacks to the agent.  If you could run this a few times while CM is showing that the agent is out of contact with the host monitor, it might give us some clues if the agent is under stress at that time or not. ... View more

Hi @LarryP,   We need to know exactly what action you are taking and what happens when you take it.  While your issue may be the same or similar to what was described earlier in this thread, we can't tell till we see your info.   Please also review your Hue log (/var/log/hue/runcpserver.log by default) at the time the problem occurs and share any log messages/exceptions you see at that time.  This should provide some background to start looking at the issue.     ... View more

@zy001,   Thanks for discussing the issue you are facing. This issue does not pose any threat to the functionality of your cluster, so the impact should be minimal.   Let's start with what we know:   - agent attempts to make an HTTP connection and gets the following:   [29/Dec/2018 16:45:08 +0000] 24147 MonitorDaemon-Reporter throttling_logger ERROR (3 skipped) Error sending messages to firehose: mgmt-SERVICEMONITOR-02008505edb7b85b2295119db7eba412 . . . . File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 32] Broken pipe   This tells us a couple of important things:   (1)   The agent was able to make a TCP connection to the Host Monitor on port 9555 (the Host Monitor Listen Port).  After that, the communication was severed as the connection appears to have been dropped somewhere between the agent and the Host Monitor Server   (2)   This appears to happen frequently.     The agent will periodically capture host information and then upload that to the Host Monitor for indexing and storage.  The first place to look would be the Host Monitor log in /var/log/:   mgmt-cmf-mgmt-HOSTMONITOR*   You mentioned you could not find the error log but there is only the one log for all log information.   If you do not see anything wrong there, then we might need to employ more advanced diagnostics to determine what is happening. ... View more

@dennistanpunya,   I am really not sure how to disable "auto-tls" so you can configure your own cert paths, but the following may work:   (1)   Go to Administration --> Settings   Select "Security" on the left.   Search for Automatic configuration of TLS for services   (2)   If you do see a configuration, choose No automatic configuration of TLS for services and SAVE   (3)   Restart Cloudera Manager with "service cloudera-scm-server restart"   I took a look at the code and I think this is the main on/off switch for auto_tls. NOTE:  you will need to manually configure the config.ini for all nodes' agents to point to your key files, certificates, truststore, and key password files.   It is possible this won't work entirely as expected, though, as I don't know of anyone who has disabled auto_tls.   ... View more

Hi @Tulasi,   The exception in the agent and error in CM indicate that the agent is not presenting its certificate for authentication properly.  This is often due to something being wrong regarding your client_key_file configuration in /etc/cloudera-scm-agent/config.ini.   Can you check to see what is configured with and then verify the contents and that it is a valid PEM formatted key?   You can try this on the agent host to verify your agent has the right key and key password configured:     # openssl rsa -in `grep client_key_file /etc/cloudera-scm-agent/config.ini | sed 's/client_key_file=//'` -check -passin pass:`grep client_keypw_file /etc/cloudera-scm-agent/config.ini |sed 's/client_keypw_file=//'| xargs cat`   The output should show "RSA key ok" and the base64 encoded key text.   ... View more

@dennistanpunya,   I opened an internal Cloudera Jira to make the licensing more obvious. Note that the limitation was introduced in Cloudera 6.0 (not 6.1 as I mentioned earlier)   The only place I found the mention of the certificate automation was in the data sheet:   https://www.cloudera.com/content/dam/www/marketing/resources/datasheets/cloudera-enterprise-datasheet.pdf.landing.html   We are sorry that you had to go through all this troubleshooting. It is much appreciated that you brought this to our attention, though. Thanks again,   Ben ... View more

@dennistanpunya,   Yes, an enterprise license is required to use auto-tls.  This was introduced in Cloudera Manager 6.1   It appears we may not have made that clear in the documentation.  I'll address this internally. For now, you can try the feature with the Trial license, but the feature will not function if an express license is in use.   ... View more