Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloudbreak downscale fails when no Datanodes in cluster (bug?)

avatar

Hello,

Probably a bug: I have a Storm cluster set up using Cloudbreak. The cluster includes only the minimum set of services needed to run Storm (Nimbus, Supervisors, Zookeeper + Ambari Metrics) - blueprint attached. One of the instance groups includes only the Storm supervisors, and I want this group to be able to scale. Scaling up works properly. Scaling down is impossible, an error is thrown. Looking at the logs, I can see that (probably) Cloudbreak is trying to find Datanodes to decommission - but since the cluster doesn't include any, an error is thrown:

cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] pollWithTimeout:30 INFO  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Polling attempt 1.
cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] getRawResource:79 INFO  c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] AmbariClient getRawResource, resourceRequestMap: {path=https://172.31.38.186:9443/api/v1/clusters/storm-min-4/services/HDFS/components/NAMENODE, query={fields=metrics/dfs/namenode/DecomNodes}}
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] getRawResource:86 ERROR c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Error occurred during GET request to {}, exception:
cloudbreak_1   | groovyx.net.http.HttpResponseException: Not Found
cloudbreak_1   | at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:263)
//...
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Exception occurred in the polling: Text must not be null or empty
cloudbreak_1   | java.lang.IllegalArgumentException: Text must not be null or empty
cloudbreak_1   | at groovy.json.JsonSlurper.parseText(JsonSlurper.java:203)
cloudbreak_1   | at groovy.json.JsonSlurper$parseText.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper.getDecommissioningDataNodes(ServiceAndHostService.groovy:227)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper$getDecommissioningDataNodes$3.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.AmbariClient.getDecommissioningDataNodes(AmbariClient.groovy)
//...

Is there any way to bypass this issue without adding an HDFS node to the cluster?

I am using Cloudbreak 1.16.5storm-min-blueprint.txt

1 ACCEPTED SOLUTION

avatar
Super Collaborator

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Br

R

View solution in original post

9 REPLIES 9

avatar
Super Guru

@Leszek Leszczynski

It could be a bug. The reason is well explained in this article, however, in your case, data node services are not present as expected.

https://community.hortonworks.com/articles/12981/impact-of-hdfs-using-cloudbreak-scale-dwon.html

I'll escalate the question to Cloudbreak team.

avatar
Super Collaborator

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Br

R

avatar
Super Guru

@rdoktorics

Richard, Cloudbreak 1.16.5 is the version that is presented on hortonworks.com website, Software Download/ HDP section. However, documentation shows Cloudbreak 2.4 (see https://docs.hortonworks.com/). Is that right? Where should Leszek go to download Cloudbreak 2.4 for his HDP 2.6.4 installation requirement?

avatar

Updating the Cloudbreak installation fixed the issue.

avatar
Super Guru

@Leszek Leszczynski

This is a known bug with the version you use. Let's wait for Richard to advise.

avatar

Thank you for your answers.

@rdoktorics I am actually using Cloudbreak to manage two HDP clusters, and I was just checking the possibility of managing a Storm cluster with this tool. Is it possible to upgrade the current installation? I assume that the answer is "yes, follow the procedure from the webpage", but since it means moving between major versions there might be some hurdles.

avatar

@Leszek Leszczynski Yes, you can upgrade and your clusters will not be affected. The steps to upgrade are here:

https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.0/content/cb-upgrade/index.html

avatar

@Leszek Leszczynski

Have you managed to update and resolve your original issue?

avatar

Yes, I have successfully upgraded my Cloudbreak installation and the problem is gone.