Cloudbreak 2.7.1 or 2.8TP with Openstack Rocky

[#1]

I am working on a private cloud that has Openstack Rocky on it. I have tried multiple times to spin up clusters with both Cloudbreak 2.7.1 and 2.8TP with no luck. Installing either version of cloudbreak seems to work fine, but each time I try to create a cluster, I get an almost immediate failure. I have looked through the cloudbreak logs and have not found any helpful errors and am just about at my wits end with this. I have noticed that Cloudbreak does not seem to be able to retrieve the image catalog that I have locally...I had my Openstack admin upload the images as shown in the documentation, which I can see from my Openstack console/UI as expected...but in cloudbreak, when I look at the default cloudbreak image catalog, I see what is expected, but with my internal catalog, depending on which URL I use, I get either "multiple choices" errors or "401" errors. The urls I use are http://myopenstack:9292 or http://myopenstack:9292/v2/images. Apart from that, the only errors I am currently getting when I create a cluster is this:

Error during stack creation flow:
cloudbreak_1   | org.openstack4j.api.exceptions.ServerResponseException: Internal Server Error
cloudbreak_1   | 	at org.openstack4j.core.transport.HttpExceptionHandler.mapException(HttpExceptionHandler.java:40)
cloudbreak_1   | 	at org.openstack4j.core.transport.HttpExceptionHandler.mapException(HttpExceptionHandler.java:23)
cloudbreak_1   | 	at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:51)
cloudbreak_1   | 	at org.openstack4j.core.transport.HttpEntityHandler.handle(HttpEntityHandler.java:24)
cloudbreak_1   | 	at org.openstack4j.connectors.jersey2.HttpResponseImpl.getEntity(HttpResponseImpl.java:63)
cloudbreak_1   | 	at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:225)
cloudbreak_1   | 	at org.openstack4j.openstack.internal.BaseOpenStackService$Invocation.execute(BaseOpenStackService.java:207)
cloudbreak_1   | 	at org.openstack4j.openstack.heat.internal.StackServiceImpl.getStackByName(StackServiceImpl.java:91)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.openstack.common.OpenStackStackValidator.validate(OpenStackStackValidator.java:27)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.ProvisionValidationHandler.accept(ProvisionValidationHandler.java:48)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.ProvisionValidationHandler.accept(ProvisionValidationHandler.java:21)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.ProvisionValidationHandler$$FastClassBySpringCGLIB$$5f1962a5.invoke(<generated>)
cloudbreak_1   | 	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
cloudbreak_1   | 	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
cloudbreak_1   | 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
cloudbreak_1   | 	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
cloudbreak_1   | 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1   | 	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
cloudbreak_1   | 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
cloudbreak_1   | 	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
cloudbreak_1   | 	at com.sequenceiq.cloudbreak.cloud.handler.ProvisionValidationHandler$$EnhancerBySpringCGLIB$$67d65420.accept(<generated>)
cloudbreak_1   | 	at reactor.bus.EventBus$3.accept(EventBus.java:317)
cloudbreak_1   | 	at reactor.bus.EventBus$3.accept(EventBus.java:310)
cloudbreak_1   | 	at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
cloudbreak_1   | 	at reactor.bus.routing.TraceableDelegatingRouter.route(TraceableDelegatingRouter.java:51)
cloudbreak_1   | 	at reactor.bus.EventBus.accept(EventBus.java:591)
cloudbreak_1   | 	at reactor.bus.EventBus.accept(EventBus.java:63)
cloudbreak_1   | 	at reactor.core.dispatch.AbstractLifecycleDispatcher.route(AbstractLifecycleDispatcher.java:160)
cloudbreak_1   | 	at reactor.core.dispatch.MultiThreadDispatcher$MultiThreadTask.run(MultiThreadDispatcher.java:74)
cloudbreak_1   | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
cloudbreak_1   | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
cloudbreak_1   | 	at java.base/java.lang.Thread.run(Thread.java:844)

[#2]

Hi @Aaron Bossert,

please note that Openstack Rocky is unfortunately not yet supported.

Apart from that, starting from cloudbreak 2.7.1, you no longer need to import your images to openstack: if it is not present, first time it will be copied from the image repository during cluster creation.

Documentation: https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.7.1/content/os-image-import/index....

Regarding the problem you mentioned with the image catalog: can you create a cluster with any of the images from the default image catalog shipped with cloudbreak?

Please let me know it that has helped!

[#3]

Thanks for the response...I have tried with the default images and the ones I imported...neither has worked. I get the error shown above when I use the default images...when I use the ones I uploaded, I get 401 errors. Also, given that Mitaka is 5 versions behind current and was EOL as of April of 2017 (more than a year ago), is there a date expected for support of newer versions of OpenStack? Unfortunately, I don't have control over the Openstack version and must use the one that is supplied on my private cluster/cloud.

[#4]

Hi @Aaron Bossert,

Is it possible you are using Keystone v3 credential? If yes, could you recreate your credential as v2 and try with it? We had an issue with Keystone v3 credential recently on some OS deployment.

[#5]

@mmolnar, thanks for the suggestion...unfortunately, my private cloud does not seem to have the v2 credentials enabled as I get errors when trying to create a new credential as v2.

[#6]

It does not work with keystone v2 credential as well.
Once I workaround the problem for downloading the image ( Q217819 ), it still fails with:

Cannot use the specified Ambari stack: StackRepoDetails{stack='HDP-2.6'; utils='HDP-UTILS-1.1.0.22'}. Error: An internal system exception occurred: Could not load url from http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-292.xml.  public-repo-1.hortonworks.com

[#7]

I found out that my Ambari server node to could communicate externaly due to /etc/resolv.conf

# cat /etc/resolv.conf
; Created by cloud-init on instance boot automatically, do not edit. 
;
; generated by /etc/dhcp/dhclient-enter-hooks
search openstacklocal
nameserver 127.0.0.1

I added a valid DNS server and the HDP cluster was able to be deployed !

[#8]

@Renaud Manus, I have circled back to this and now am using cloudbreak 2.9.0. I am not even able to create a credential in cloudbreak now...neither v2 or v3 seems to work. I have confirmed that my openstack credentials work and are properly configured by using them with the openstack CLI (python client) on my Cloudbreak server. I am seeing error messages related to the URL being requested within the Openstack logs:

10.10.0.98 - - [27/Mar/2019:15:45:28 -0400] "POST /v3/tokens HTTP/1.1" 404 233 "-" "Jersey/2.26 (HttpUrlConnection 10.0.2)"
10.10.0.98 - - [27/Mar/2019:15:46:35 -0400] "POST /v3/tokens HTTP/1.1" 404 233 "-" "Jersey/2.26 (HttpUrlConnection 10.0.2)"
10.10.0.98 - - [27/Mar/2019:15:47:24 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 6992 "-" "Jersey/2.26 (HttpUrlConnection 10.0.2)"
10.10.0.98 - - [27/Mar/2019:15:50:12 -0400] "POST /v3/auth/tokens HTTP/1.1" 201 6992 "-" "Jersey/2.26 (HttpUrlConnection 10.0.2)"

It looks like when I try to create a v3 credential it is hitting the right URL, but is giving a fairly useless "NOT FOUND" error on the cloudbreak side. With the v2 credential, I get the same "NOT FOUND" error in the cloudbreak logs, but in the Openstack logs, it looks like the correct URL is getting hit (/v3/auth/tokens vs /v3/tokens).

I am running out of things to try and hope someone can take a look at this to see what might be going on. Given that the first time around, @Renaud Manus seemed to be able to get it working with the V2 API, I am hoping that this can work...