Support Questions
Find answers, ask questions, and share your expertise

Azure HDP cluster disappeared from Cloudbreak GUI and after reappearing won't sync

Azure HDP cluster disappeared from Cloudbreak GUI and after reappearing won't sync

I installed a HDP 2.5 cluster on Azure using Cloudbreak 1.6.3. Everything was running fine, until Microsoft stopped both the cluster and the cloudbreak VM because of a supposed 'service violation'. After that issue was resolved I restarted the cloudbreak vm and was told to upgrade to version 1.14.0 and regenerate the docker-compose.yml by the cdb doctor command. So I did. After starting cloudbreak and logging in, all seemed fine, only the reference to my cluster was missing. I also noticed that the GUI takes a long time loading (the page is greyed out and a 'Loading...' sign is displayed. It might be timing out on something...

  • Starting the nodes manually resulted in Cloudbreak showing the cluster again. But subsequently syncing it from within cloudbreak results in an error 'Cluster can't be synchronized. [ stack status: STOPPED, cluster status: AVAILABLE ]'.
  • Refreshing the Cloudbreak GUI makes the stop button appear, but clicking it results in 'cluster not available' error.
  • Stopping all the nodes manually and syncing results in an error 'Cluster and it's stack sync failed please try again: Stack 11 has flows under operation, request not allowed'. The error stays the same, even after retry after some time.

How can I get the link to my (currently stopped) cluster back in the cloudbreak GUI and sync it?

1 REPLY 1
Highlighted

Re: Azure HDP cluster disappeared from Cloudbreak GUI and after reappearing won't sync

I could get a step further by

  • manually updating the cluster status in the status table to 'STOPPED'
  • manually starting all node vm's, logging in and halting them
  • restarting cloudbreak

Then after some time cloudbreak syncs with the cluster and can startup the cluster. However it says that the infrastructure is started and that a cluster startup is ongoing. In Azure however the VM's still have a 'stopped' status.

When I look in the Cloudbreak startup log, I see an error:

Failed to convert from type [java.util.LinkedHashSet<?>] to type [com.sequenceiq.cloudbreak.api.model.AutoscaleStackResponse] for value 'com.sequenceiq.cloudbreak.domain.Stack@25194c0b'

Even after restarting Cloudbreak, these errors reoccur and the GUI still hangs in the 'update in progress: Starting Ambari cluster' fase...