question Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?) in Archives of Support Questions (Read Only)

Cloudbreak downscale fails when no Datanodes in cluster (bug?)

leszek_leszczyn — Fri, 30 Mar 2018 19:21:55 GMT

Hello,

Probably a bug: I have a Storm cluster set up using Cloudbreak. The cluster includes only the minimum set of services needed to run Storm (Nimbus, Supervisors, Zookeeper + Ambari Metrics) - blueprint attached. One of the instance groups includes only the Storm supervisors, and I want this group to be able to scale. Scaling up works properly. Scaling down is impossible, an error is thrown. Looking at the logs, I can see that (probably) Cloudbreak is trying to find Datanodes to decommission - but since the cluster doesn't include any, an error is thrown:

cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] pollWithTimeout:30 INFO  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Polling attempt 1.
cloudbreak_1   | 2018-03-30 10:07:30,986 [reactorDispatcher-6] getRawResource:79 INFO  c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] AmbariClient getRawResource, resourceRequestMap: {path=https://172.31.38.186:9443/api/v1/clusters/storm-min-4/services/HDFS/components/NAMENODE, query={fields=metrics/dfs/namenode/DecomNodes}}
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] getRawResource:86 ERROR c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Error occurred during GET request to {}, exception:
cloudbreak_1   | groovyx.net.http.HttpResponseException: Not Found
cloudbreak_1   | at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:263)
//...
cloudbreak_1   | 2018-03-30 10:07:31,006 [reactorDispatcher-6] pollWithTimeout:34 WARN  c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Exception occurred in the polling: Text must not be null or empty
cloudbreak_1   | java.lang.IllegalArgumentException: Text must not be null or empty
cloudbreak_1   | at groovy.json.JsonSlurper.parseText(JsonSlurper.java:203)
cloudbreak_1   | at groovy.json.JsonSlurper$parseText.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper.getDecommissioningDataNodes(ServiceAndHostService.groovy:227)
cloudbreak_1   | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper$getDecommissioningDataNodes$3.call(Unknown Source)
cloudbreak_1   | at com.sequenceiq.ambari.client.AmbariClient.getDecommissioningDataNodes(AmbariClient.groovy)
//...

Is there any way to bypass this issue without adding an HDFS node to the cluster?

I am using Cloudbreak 1.16.5storm-min-blueprint.txt

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

cstanca — Sat, 31 Mar 2018 02:00:54 GMT

@Leszek Leszczynski

It could be a bug. The reason is well explained in this article, however, in your case, data node services are not present as expected.

https://community.hortonworks.com/articles/12981/impact-of-hdfs-using-cloudbreak-scale-dwon.html

I'll escalate the question to Cloudbreak team.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

rdoktorics — Sat, 31 Mar 2018 02:55:36 GMT

@Leszek Leszczynski

My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

cstanca — Sat, 31 Mar 2018 07:01:06 GMT

@rdoktorics

Richard, Cloudbreak 1.16.5 is the version that is presented on hortonworks.com website, Software Download/ HDP section. However, documentation shows Cloudbreak 2.4 (see https://docs.hortonworks.com/). Is that right? Where should Leszek go to download Cloudbreak 2.4 for his HDP 2.6.4 installation requirement?

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

cstanca — Sat, 31 Mar 2018 07:03:02 GMT

@Leszek Leszczynski

This is a known bug with the version you use. Let's wait for Richard to advise.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

leszek_leszczyn — Sat, 31 Mar 2018 15:04:10 GMT

Thank you for your answers.

@rdoktorics I am actually using Cloudbreak to manage two HDP clusters, and I was just checking the possibility of managing a Storm cluster with this tool. Is it possible to upgrade the current installation? I assume that the answer is "yes, follow the procedure from the webpage", but since it means moving between major versions there might be some hurdles.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

Dominika — Mon, 02 Apr 2018 23:35:19 GMT

@Leszek Leszczynski Yes, you can upgrade and your clusters will not be affected. The steps to upgrade are here:

https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.0/content/cb-upgrade/index.html

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

darvasip — Thu, 05 Apr 2018 17:56:07 GMT

@Leszek Leszczynski

Have you managed to update and resolve your original issue?

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

leszek_leszczyn — Thu, 05 Apr 2018 20:05:29 GMT

Yes, I have successfully upgraded my Cloudbreak installation and the problem is gone.

Re: Cloudbreak downscale fails when no Datanodes in cluster (bug?)

leszek_leszczyn — Thu, 05 Apr 2018 20:06:08 GMT

Updating the Cloudbreak installation fixed the issue.