Cloudera Manager Hangs When Installing Parcels

[#1]

Hello,

Been working with Cloudera for about 2 years now and up until recently, when the Cloudera Express 5.9 version was released, I had no issue adding a cluster. Starting with 5.9, when adding a cluster, I can find and select the hosts, parcels, install the CM agent and get a good status on a heartbeat. When the Installing Selected Parcels step comes in, I show 100% downloaded, but 0/0 in Distributed, Unpacked, and Activated. The issue exists regardless of which parcel or how many types I choose to install. My logs show the new cluster has no member hosts. I then have to open a new window, add the hosts to the cluster, distribute and upgrade CDH to get the parcels to distribute, unpack, and activate.

I thought it may have been a bug within the Cloudera Express 5.9.1, so I stood a new server up and installed the trial version of Enterprise Data Hub 5.10.1 only to have the same result. I can't find Cloudera reporting this as a known issue or any other instances of this exact problem. Can someone please help?

My configuration is an AWS environment with a single RHEL 6.8 (CE 5.9.1) and RHEL 7.3 (EDH 5.10.1) server; I am not running single-user mode.

Below is from the Enterprise Data Hub trial version 5.10.1, however the same errors and issues exist in the Express 5.9.1 version.

An excerpt of the log:

2017-03-30 16:13:48,639 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelInstallerImpl: Error while installing parcel
com.cloudera.parcel.ParcelException: Cluster <b>cluster</b> has no member hosts. Add hosts to the cluster and then try again.
        at com.cloudera.parcel.components.ParcelManagerImpl.distribute(ParcelManagerImpl.java:362)
        at com.cloudera.parcel.components.ParcelInstallerImpl.handleJob(ParcelInstallerImpl.java:173)
        at com.cloudera.parcel.components.ParcelInstallerImpl.run(ParcelInstallerImpl.java:145)
        at com.cloudera.parcel.components.ParcelInstallerImpl.run(ParcelInstallerImpl.java:34)
        at com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable.call(ReadWriteDatabaseTaskCallable.java:36)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Screenshot of where it hangs:

Back at the Dashboard

The hosts have the agent installed and appear in good condition:

Thanks in advance.

[#2]

Hello,

Thank you for reaching out to us, in order to assist you we would need to see additional log information from Cloudera Manager. The error you have referenced indicates that no host have been added to the cluster you are attempting to install parcels to. Based on what is here at this moment this does not appear to be a bug or a change in our software behavior.

In addition to that the error you provided has a differeing cluster name than the ones you provided in your screenshots. We also note that your screenshots show 'Cluster 1' as having CDH 5.0.0 deployed via packages and not via parcels. The All Host page does not show us which host are presently assigned to a specific cluster without filtering.

If you have added host to Cluster 1 or another cluster please use the 'Cluster' filter on the side of the All Host page so that we can see which host are assigned to that cluster. Please confirm the cluster name, CDH release, Packaging method, and any previous deployment steps you have taken.

[#3 · Accepted solution]

Thank you for working with me via messages to resolve my issue. 

As you stated in your message to me, FQDN resolution was the cause of my problem. Since we're not using a DNS solution for our environment, once I entered the cluster hosts FQDNs into the CM's /etc/hosts, the parcels no longer hung at distribution and the installation was able to continue.

[#4]

Hi Me too facing the same issue today. how to enter FQDNs into the CM's /etc/hosts. Kindly guide me

[#5]

I am doing cloudera manager instalation in AWS ec2 cloud. 3 hosts are there in my cluster. I think this is the last step to get my complely installed cloudera manager.

[#6]

Is there any experts to chat and solve my isssue

[#7]

Hello baghdaddy,

How have you fixed this problem? Can you explain it to me? I'm facing the same problem.

[#8]

I could fix this issue while installing CDP Data Center Trial.

First, you need to get hostname of all hosts in your cluster.

After that, try to edit /etc/hosts file as below:

<host 1's local IP> <hostname1>

<host 2's local IP> <hostname2>

Save it and restart hosts.

[#9]

Adding to this response, my issue was I missed to insert the CM's hostname+ip in the /etc/hosts config file of the client nodes.

While the installation was still stuck I added the missing line to the /etc/hosts file on the client machine and the installation moved on fine.

[#10]

Another edge case that you might hit is if a host has both IPv6 and IPv4 address (and only really have IPv4 connectivity due to firewalls etc).  Running "lsof -p (pid-of-flood-process)" on the agent box and seeing "SYN_SENT" was the clue here.