cdh6.3.2 15528 MainThread agent ERROR Heartbeating to 192.168.88.54:7182 failed.

[#1]

/etc/cloudera-scm-agent/config.ini

[General]
server_host=192.168.88.54
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=INFO
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=0
max_cert_depth=9
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]

The Internet said that the problem of auto-tls, but I did not open, try a lot of online methods can not solve, can anyone help me solve this problem

[#2]

Hello @SIGOYI 

Thank you for reaching out to the community

I don't this is a Auto-TLS issue at this point. As use_tls=0 which means TLS is not enforced from the agent level

Could you please check /var/log/cloudera-scm-agent/cloudera-scm-agent.log to check this further

Also which TLS are you using on CM is it Auto-TLS or Manual TLS? If this is Auto-TLS, are the certificates signed by CM (Auto-TLS usecase 1) or your Custom CA (Auto-TLS use case)?

[#3]

hello @upadhyayk04 

Thanks for your reply, I did not enable tls. The following is the error log of /var/log/cloudera-scm-agent/cloudera-scm-agent.log. This is my first attempt to deploy this, and I hope to get your help

[#4]

Hello @SIGOYI 

Can you please confirm is 192.168.88.54

I think is getting an SSL from somewhere. Can you go to CM UI >> Administration >> Settings >> Click on Security and check if TLS is enforced on the agent from CM UI

[#5]

Yes, the server is at 192.168.88.54 and agent192.168.88.55 uses telnet to test port 7182 at 192.168.88.54

[#6]

Hello,

Thank you for your update

So SSL Is not a problem here. Also as you mentioned that telnet is working which means connectivity is fine

Please check the below information now

a.) Is port 9000 and other ports and enabled nothing else is running on them

https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/installation/topics/cdpdc-ports-used-by-cm.ht...

b.) Can you compare the UUID of the host present in the file /var/lib/cloudera-scm-agent/uuid on the agent with the one that is visible on CM UI (ON CM UI >> Host >> all hosts >> Select this host and you will the UUID there) it should match with the one in uuid file on host if not then please collect the file using the below command

# echo -n "UUID_copied from UI" > /var/lib/cloudera-scm-agent/uuid

c.) Also check /var/log/messages from the node do you see any errors there

d.) Check the /var/log/cloudera-scm-server/cloudera-scm-server.log and see if you observed any errors for this node over there

e.) Also, if you have any other working node, I would suggest copying the /etc/cloudera-scm-agent/config.ini from working node to this node and then try restarting the agent

Let me know if you have any further queries

[#7]

Hello@upadhyayk04 

Thanks for your reply

a) These are newly opened VMS, which have no other applications except cdh, and the ports are not occupied by others

b) Because the cluster is not created successfully, there is no information in the host list

c) Except for the error information in the following pictures, other logs do not contain error information

d) /var/log/cloudera-scm-server/cloudera-scm-server.log No error is displayed. Only one agent fails to be deployed

e) Not a single node can be added at present

[#8]

Hello @SIGOYI 

I suspect some issues with config.ini

Please move the current config.ini (/etc/cloudera-scm-agent/config.ini) to some other  location and copy /etc/cloudera-scm-agent/config.ini from CM server to this node and just replace server_host=192.168.88.54 (if this is localhost)

Also run the below command

# nslookup 192.168.88.54 (Get the hostname of CM server here)

# nslookup <CM_SERVER_HOSTNAME>

# telnet 192.168.88.54 7182 (just repeat once again)

Then do a hard restart of the agent

# systemctl stop cloudera-scm-agent

# systemctl stop cloudera-scm-supervisord

# systemctl start cloudera-scm-agent

[#9]

Hello @SIGOYI 

Just want to follow up

Are you still facing issue if yes then lets capture the TCP dump on port 7182 of CM server and see if the request is reaching out to CM or not form the affected node

[#10]

hello@upadhyayk04 

Thank you for your attention. After I changed the host name, it became normal. The host name I used before was 54-yandongjin-c7. I don't know why I couldn't identify the agent, but later I changed it to cdh1.It's ready to use.

But now I have a new problem,Error display  "Unable to issue query: could not connect to the Host Monitor" ,and I can't see cloudera management service on the screen