Cloudera Manager Agent failed to receive heart beat during the installation.

[#1]

Hi Cloudera Team,

Cloudera manager installation is successful on AWS node which is running on RHEL 7.4.

But when adding this node to cluster we are getting agent herat beat failure and throwing errors like..

Installation failed. Failed to receive heartbeat from agent.

Ensure that the host's hostname is configured properly.

Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).

Ensure that ports 9000 and 9001 are not in use on the host being added.

Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).

If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here

IN log file:

BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.out | sed 's/^/>>/'

>>[17/Nov/2017 05:17:16 +0000] 28084 MainThread agent INFO SCM Agent Version: 5.13.0

>>[17/Nov/2017 05:17:16 +0000] 28084 MainThread agent WARNING Expected mode 0751 for /var/run/cloudera-scm-agent but was 0755

>>[17/Nov/2017 05:17:16 +0000] 28084 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent

>>[17/Nov/2017 05:17:16 +0000] 28084 MainThread agent INFO SCM Agent Version: 5.13.0

>>[17/Nov/2017 05:17:16 +0000] 28084 MainThread agent WARNING Expected mode 0751 for /var/run/cloudera-scm-agent but was 0755

>>[17/Nov/2017 05:17:16 +0000] 28084 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent

END (0)

BEGIN tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/'

tail: tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory

cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory

END (0)

end of agent logs.

scm agent started

Installation script completed successfully.

we have varified all the hostname port communications are fine, but still it is failing.

Could please look on this issue  and provide the solution.

Thanks

Narayan

[#2]

Hi Cloudera support team,

The filesystem structure on AWS RHEL 7.4 server is like this below..

tmps filesystem is mounted on /run /sys/fs/cgroup, /run/user is this causing the issue.

Could you please check on it.

[narayana.raovanka@newvoe5-ccs_a_p_1 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.6G     0  3.6G   0% /dev
tmpfs           3.6G     0  3.6G   0% /dev/shm
tmpfs           3.6G  8.4M  3.6G   1% /run
tmpfs           3.6G     0  3.6G   0% /sys/fs/cgroup
/dev/xvda1       64G   20G   41G  33% /
/dev/xvdb       250G  4.1G  246G   2% /u01
tmpfs           722M     0  722M   0% /run/user/996
tmpfs           722M     0  722M   0% /run/user/1010
tmpfs           722M     0  722M   0% /run/user/0

[#3 · Accepted solution]

@narayanaraov,

The problem appears to occur since the scm prepare script cannot tail the agent log.  This leads to the Wizard in Cloudera Manager to believe that the installation did not work properly.

See the error in the log snippet you provided:

tail: tail: cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory

cannot open '/var/log/cloudera-scm-agent//cloudera-scm-agent.log' for reading: No such file or directory

Check to see if the log file exists.  If it does not, that indicates that the agent may not have been able to start.  Try starting the agent manually on that host with "service cloudera-scm-agent restart"

Then, check /var/log/cloudera-scm-agent/cloudera-scm-agent.log again and see if it exists and if there are any errors in it.

Since we see in your log snippet that a tail of "/var/log/cloudera-scm-agent//cloudera-scm-agent.out " was successful, it is odd that the agent log didn't exist unless the agent process existed unexpectedly

[#4]

Thanks ,

After verifying logs :

I have change the hostname without special characters and updated the hostname entry in /etc/sysconfig/network

and Adjusted Custom security group in AWS console after that installation was succussful.

Now we want to customize the Security group in AWS, could please suggest which are protocols and services and ports need to open for cloudera.

Thanks alot for your help...