Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Solved Go to solution
Highlighted

Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Hello,

Probably a bug: I have a Storm cluster set up using Cloudbreak. The cluster includes only the minimum set of services needed to run Storm (Nimbus, Supervisors, Zookeeper + Ambari Metrics) - blueprint attached. One of the instance groups includes only the Storm supervisors, and I want this group to be able to scale. Scaling up works properly. Scaling down is impossible, an error is thrown. Looking at the logs, I can see that (probably) Cloudbreak is trying to find Datanodes to decommission - but since the cluster doesn't include any, an error is thrown:

cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] pollWithTimeout:30 INFO  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Polling attempt 1.
cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] getRawResource:79 INFO  c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] AmbariClient getRawResource, resourceRequestMap: {path=https://172.31.38.186:9443/api/v1/clusters/storm-min-4/services/HDFS/components/NAMENODE, query={fields=metrics/dfs/namenode/DecomNodes}}
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] getRawResource:86 ERROR c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Error occurred during GET request to {}, exception:
cloudbreak_1   | groovyx.net.http.HttpResponseException: Not Found
cloudbreak_1   | at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:263)
//...
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Exception occurred in the polling: Text must not be null or empty
cloudbreak_1   | java.lang.IllegalArgumentException: Text must not be null or empty
cloudbreak_1   | at groovy.json.JsonSlurper.parseText(JsonSlurper.java:203)
cloudbreak_1   | at groovy.json.JsonSlurper$parseText.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper.getDecommissioningDataNodes(ServiceAndHostService.groovy:227)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper$getDecommissioningDataNodes$3.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.AmbariClient.getDecommissioningDataNodes(AmbariClient.groovy)
//...

Is there any way to bypass this issue without adding an HDFS node to the cluster?

I am using Cloudbreak 1.16.5storm-min-blueprint.txt

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Expert Contributor

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Br

R

View solution in original post

9 REPLIES 9
Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski

It could be a bug. The reason is well explained in this article, however, in your case, data node services are not present as expected.

https://community.hortonworks.com/articles/12981/impact-of-hdfs-using-cloudbreak-scale-dwon.html

I'll escalate the question to Cloudbreak team.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Expert Contributor

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Br

R

View solution in original post

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@rdoktorics

Richard, Cloudbreak 1.16.5 is the version that is presented on hortonworks.com website, Software Download/ HDP section. However, documentation shows Cloudbreak 2.4 (see https://docs.hortonworks.com/). Is that right? Where should Leszek go to download Cloudbreak 2.4 for his HDP 2.6.4 installation requirement?

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Updating the Cloudbreak installation fixed the issue.

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski

This is a known bug with the version you use. Let's wait for Richard to advise.

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Thank you for your answers.

@rdoktorics I am actually using Cloudbreak to manage two HDP clusters, and I was just checking the possibility of managing a Storm cluster with this tool. Is it possible to upgrade the current installation? I assume that the answer is "yes, follow the procedure from the webpage", but since it means moving between major versions there might be some hurdles.

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski Yes, you can upgrade and your clusters will not be affected. The steps to upgrade are here:

https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.0/content/cb-upgrade/index.html

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

@Leszek Leszczynski

Have you managed to update and resolve your original issue?

Highlighted

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Yes, I have successfully upgraded my Cloudbreak installation and the problem is gone.

Don't have an account?
Coming from Hortonworks? Activate your account here