Created 03-30-2018 12:21 PM
Hello,
Probably a bug: I have a Storm cluster set up using Cloudbreak. The cluster includes only the minimum set of services needed to run Storm (Nimbus, Supervisors, Zookeeper + Ambari Metrics) - blueprint attached. One of the instance groups includes only the Storm supervisors, and I want this group to be able to scale. Scaling up works properly. Scaling down is impossible, an error is thrown. Looking at the logs, I can see that (probably) Cloudbreak is trying to find Datanodes to decommission - but since the cluster doesn't include any, an error is thrown:
cloudbreak_1 | 2018-03-30 10:07:30,986 [reactorDispatcher-6] pollWithTimeout:30 INFO c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Polling attempt 1. cloudbreak_1 | 2018-03-30 10:07:30,986 [reactorDispatcher-6] getRawResource:79 INFO c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] AmbariClient getRawResource, resourceRequestMap: {path=https://172.31.38.186:9443/api/v1/clusters/storm-min-4/services/HDFS/components/NAMENODE, query={fields=metrics/dfs/namenode/DecomNodes}} cloudbreak_1 | 2018-03-30 10:07:31,006 [reactorDispatcher-6] getRawResource:86 ERROR c.s.a.c.AmbariClientUtils - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Error occurred during GET request to {}, exception: cloudbreak_1 | groovyx.net.http.HttpResponseException: Not Found cloudbreak_1 | at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:263) //... cloudbreak_1 | 2018-03-30 10:07:31,006 [reactorDispatcher-6] pollWithTimeout:34 WARN c.s.c.s.PollingService - [owner:990ffe52-4eb1-4fc6-a0ac-eb8c50db8751] [type:STACK] [id:12] [name:storm-min-4] Exception occurred in the polling: Text must not be null or empty cloudbreak_1 | java.lang.IllegalArgumentException: Text must not be null or empty cloudbreak_1 | at groovy.json.JsonSlurper.parseText(JsonSlurper.java:203) cloudbreak_1 | at groovy.json.JsonSlurper$parseText.call(Unknown Source) cloudbreak_1 | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper.getDecommissioningDataNodes(ServiceAndHostService.groovy:227) cloudbreak_1 | at com.sequenceiq.ambari.client.services.ServiceAndHostService$Trait$Helper$getDecommissioningDataNodes$3.call(Unknown Source) cloudbreak_1 | at com.sequenceiq.ambari.client.AmbariClient.getDecommissioningDataNodes(AmbariClient.groovy) //...
Is there any way to bypass this issue without adding an HDFS node to the cluster?
I am using Cloudbreak 1.16.5storm-min-blueprint.txt
Created 03-30-2018 07:55 PM
My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4
Br
R
Created 03-30-2018 07:00 PM
It could be a bug. The reason is well explained in this article, however, in your case, data node services are not present as expected.
https://community.hortonworks.com/articles/12981/impact-of-hdfs-using-cloudbreak-scale-dwon.html
I'll escalate the question to Cloudbreak team.
Created 03-30-2018 07:55 PM
My first question is can you upgrade to 2.4? That is latest GA release and if you just now started testing Cloudbreak I suggest to use 2.4
Br
R
Created 03-31-2018 12:01 AM
Richard, Cloudbreak 1.16.5 is the version that is presented on hortonworks.com website, Software Download/ HDP section. However, documentation shows Cloudbreak 2.4 (see https://docs.hortonworks.com/). Is that right? Where should Leszek go to download Cloudbreak 2.4 for his HDP 2.6.4 installation requirement?
Created 04-05-2018 01:06 PM
Updating the Cloudbreak installation fixed the issue.
Created 03-31-2018 12:03 AM
This is a known bug with the version you use. Let's wait for Richard to advise.
Created 03-31-2018 08:04 AM
Thank you for your answers.
@rdoktorics I am actually using Cloudbreak to manage two HDP clusters, and I was just checking the possibility of managing a Storm cluster with this tool. Is it possible to upgrade the current installation? I assume that the answer is "yes, follow the procedure from the webpage", but since it means moving between major versions there might be some hurdles.
Created 04-02-2018 04:35 PM
@Leszek Leszczynski Yes, you can upgrade and your clusters will not be affected. The steps to upgrade are here:
https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.0/content/cb-upgrade/index.html
Created 04-05-2018 10:56 AM
Have you managed to update and resolve your original issue?
Created 04-05-2018 01:05 PM
Yes, I have successfully upgraded my Cloudbreak installation and the problem is gone.