Cloudera Enterprise Trial 6.1.0 - Add Host Error

[#1]

I am getting the following error when I am trying to add host to cloudera:

[25/Mar/2020 04:56:45 +0000] 1611 MainThread agent ERROR Failed to handle Heartbeat Response:{....} (A big response) 

Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1526, in handle_heartbeat_response
self._handle_heartbeat_response(response)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1661, in _handle_heartbeat_response
self._update_parcel_activation_state(response)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1572, in _update_parcel_activation_state
manage_old_parcels = old_response.get("create_parcel_symlinks")
AttributeError: 'NoneType' object has no attribute 'get'

This causes the following error:

Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 125, in refresh
pid = ParcelId.dir(child)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel_id.py", line 75, in dir
raise Exception("Invalid parcel directory: %s" % (dir))
Exception: Invalid parcel directory: CDH

I have checked the /etc/hosts file, which seems fine and consistent. Any other way to debug this.

Thanks,

[#2]

@ankesh_clo Can you check /var/lib/cloudera-scm-agent/ dir on the new host, Then delete the file response.avro if exists. 

After that go to CM > Hosts > All hosts > Click on the newly added host and match the HOST ID with /var/lib/cloudera-scm-agent/uuid file if this is not in sync then please modify the uuid file as per HOST ID and restart the agent.

[#3]

@GangWar  Thanks for the reply.

But this didn't solve the issue. There is no parcel downloaded and no progress.  The same error appears on the agent as posted above.

For restarting the agent, I use sudo service cloudera-scm-agent restart.

Please let me know where I am going wrong.

Thanks!

[#4]

@ankesh_clo  This seems an issue with Symlink as well looking at the stack trace. Try
the below method once.

1. Stop the Agent.
2. Remove the broken symlinks from /etc/alternatives
and corresponding conf files from /var/lib/alternatives.
3. Remove the
parcels/files from:
/opt/cloudera/parcels
/opt/cloudera/parcels/.flood
/opt/cloudera/parcel-cache4. 
4. Start the agent. The agent will
redistribute the parcel and fix the Alternatives.

[#5]

@GangWar I do not see broken links in /etc/alternatives and there is no folder named /var/lib/alternatives.

Also, there are no parcel files downloaded on the new host. The parcel and parcel-cache folders are empty.

Should I add a more detailed log? 

Thanks,

[#6]

Yes, Can you upload agent CM server log file and agent log file (New host), let's see if we can find something.

[#7]

@GangWar  Thanks for the help. I am attaching the link to log file from server and agent host.

Agent: https://ideone.com/TNKtRZ

Server: https://ideone.com/RHt1Yl

Ps: Sorry there was an issue uploading the log file. I have uploaded the relevant portion.

Please visit the link and download the log file (download at top of editor, below url box) 

[#8]

@ankesh_clo Looking at the logs the issue is also seems because of wrong parcel URL.  

2020-04-29 01:28:33,070 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (1 skipped) Failed to download manifest. Status code: 404 URI: https://archive.cloudera.com/accumulo-c6/parcels/latest/manifest.json/

The correct link is https://archive.cloudera.com/accumulo/parcels/latest/

 Either correct this link in Parcel configuration or just remove this Accumulo parcel link and try again. 

[#9]

@GangWar I removed the link from configuration as I am not using accumulo. The download error is gone but the errors still persists.

[#10]

Hi @ankesh_clo,

Could you please share outputs from below three commands on the host? This will help us to find out if there are any permissions issues etc.

ls -altr /var/lib/cloudera-scm-agent

and

ls -altr /opt/cloudera

and

ls -altr /opt/cloudera/parcels

Thanks,

Li