Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Solved Go to solution

Cloudbreak downscale fails when no Datanodes in cluster (bug?)

New Contributor

Hello,

Probably a bug: I have a Storm cluster set up using Cloudbreak. The cluster includes only the minimum set of services needed to run Storm (Nimbus, Supervisors, Zookeeper + Ambari Metrics) - blueprint attached. One of the instance groups includes only the Storm supervisors, and I want this group to be able to scale. Scaling up works properly. Scaling down is impossible, an error is thrown. Looking at the logs, I can see that (probably) Cloudbreak is trying to find Datanodes to decommission - but since the cluster doesn't include any, an error is thrown:

cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] pollWithTimeout:30 INFO  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Polling attempt 1.
cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] getRawResource:79 INFO  c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] AmbariClient getRawResource, resourceRequestMap: {path=https://172.31.38.186:9443/api/v1/clusters/storm-min-4/services/HDFS/components/NAMENODE, query={fields=metrics/dfs/namenode/DecomNodes}}
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] getRawResource:86 ERROR c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Error occurred during GET request to {}, exception:
cloudbreak_1   | groovyx.net.http.HttpResponseException: Not Found
cloudbreak_1   | at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:263)
//...
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Exception occurred in the polling: Text must not be null or empty
cloudbreak_1   | java.lang.IllegalArgumentException: Text must not be null or empty
cloudbreak_1   | at groovy.json.JsonSlurper.parseText(JsonSlurper.java:203)
cloudbreak_1   | at groovy.json.JsonSlurper$parseText.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper.getDecommissioningDataNodes(ServiceAndHostService.groovy:227)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper$getDecommissioningDataNodes$3.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.AmbariClient.getDecommissioningDataNodes(AmbariClient.groovy)
//...

Is there any way to bypass this issue without adding an HDFS node to the cluster?

I am using Cloudbreak 1.16.5storm-min-blueprint.txt

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Expert Contributor

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Br

R

9 REPLIES 9

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski

It could be a bug. The reason is well explained in this article, however, in your case, data node services are not present as expected.

https://community.hortonworks.com/articles/12981/impact-of-hdfs-using-cloudbreak-scale-dwon.html

I'll escalate the question to Cloudbreak team.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Expert Contributor

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Br

R

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@rdoktorics

Richard, Cloudbreak 1.16.5 is the version that is presented on hortonworks.com website, Software Download/ HDP section. However, documentation shows Cloudbreak 2.4 (see https://docs.hortonworks.com/). Is that right? Where should Leszek go to download Cloudbreak 2.4 for his HDP 2.6.4 installation requirement?

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

New Contributor

Updating the Cloudbreak installation fixed the issue.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski

This is a known bug with the version you use. Let's wait for Richard to advise.

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

New Contributor

Thank you for your answers.

@rdoktorics I am actually using Cloudbreak to manage two HDP clusters, and I was just checking the possibility of managing a Storm cluster with this tool. Is it possible to upgrade the current installation? I assume that the answer is "yes, follow the procedure from the webpage", but since it means moving between major versions there might be some hurdles.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski Yes, you can upgrade and your clusters will not be affected. The steps to upgrade are here:

https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.0/content/cb-upgrade/index.html

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski

Have you managed to update and resolve your original issue?

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

New Contributor

Yes, I have successfully upgraded my Cloudbreak installation and the problem is gone.