Support Questions

story0117 · ‎01-25-2017

Hi All.

I would like to create a Hadoop cluster using Cloudbreak and ambari blueprint.

But, I can't anythig...

cloudbreak_1   | 2017-xx-xx xx:xx:xx,xxx [reactorDispatcher-42] send:41 INFO  c.s.c.s.n.HttpNotificationSender - [owner:xxxxxxx-xxxx-xxxxx-xxx-xxxxxxx] [type:STACK] [id:19] [name:hogehoge] Could not send notification to the specified endpoint: 'https://ec2-xx-xxx-xxx-xx.ap-northeast-1.compute.amazonaws.com/notifications' Cause: org.apache.http.conn.HttpHostConnectException: Connect to ec2-xx-xxx-xxx-xx.ap-northeast-1.compute.amazonaws.com:443 [ec2-xx-xxx-xxx-xx.ap-northeast-1.compute.amazonaws.com/xx.xxx.xxx.xx] failed: Connection refused (Connection refused)
cloudbreak_1   | 2017-xx-xx xx:xx:xx,xxx [taskScheduler-1] send:41 INFO  c.s.c.s.n.HttpNotificationSender - [owner:spring] [type:springLog] [id:] [name:] Could not send notification to the specified endpoint: 'https://ec2-xx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com/notifications' Cause: org.apache.http.conn.HttpHostConnectException: Connect to ec2-xx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com:443 [ec2-xx-xxx-xxx-xxx.ap-northeast-1.compute.amazonaws.com/xx.xxx.xxx.xxx] failed: Connection timed out (Connection timed out)

What could be reason?

Regards,

story0117 · ‎01-25-2017

Cluster Status is as follows.

Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.

sgowda · ‎01-25-2017

@ikawana

Can you please login into the Ambari URL and check the whether all the services and up and running ? If there are any failures please click on background ops ( ops link next to the cluster name in the ambari home page ) to look for error. Below is the sample screenshot of how the background ops look like.

story0117 · ‎01-25-2017

Thanks @Santhosh B Gowda

Actually, I don't have an Ambari Web UI yet.

After I pressed the create cluster button, I installed Ambari Server,

I was thinking that Ambari Web UI could be seen on the installed server.

Am I misunderstanding?

I'm trying to install Ambari under bellow.

msereg · ‎01-25-2017

@ikawanaThe first 2 log messages only show that the backend couldn't send push notifications to the UI, so you may need to refresh the page to see changes, but that won't cause any errors on the backend.

Based on the cluster status, it seems that the instance was successfully started on AWS and the ambari-agent was started as well, but the blueprint installation couldn't be finished.

Can you send us the cloudbreak logs that contain this error message? "Cluster installation failed to complete..."

And if you click on the cluster on the Cloudbreak UI, can you see the Ambari address displayed? If not can you reach the Ambari server web UI by going to https://<public-ip-of-ambari-node>/ambari?

story0117 · ‎01-25-2017

Thanks @Marton Sereg

Below is error message like "Cluster installation failed..."

54140 /cbreak_cloudbreak_1 | 2017-01-25 10:40:16,205 [reactorDispatcher-42] buildAmbariCluster:261 ERROR c.s.c.s.c.f.AmbariClusterConnector - [owner:12a1db55-ef2e-4b8c-aed1-c724e885f00e] [type:STACK] [id:19] [name:hdp-ambari-ui] Error while building the Ambari cluster. Message Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later., throwable: {}
54141 /cbreak_cloudbreak_1 | com.sequenceiq.cloudbreak.core.ClusterException: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.
/cbreak_cloudbreak_1 | 2017-01-25 10:40:16,277 [reactorDispatcher-42] updateClusterStatusByStackId:387 DEBUG c.s.c.s.c.AmbariClusterService - [owner:12a1db55-ef2e-4b8c-aed1-c724e885f00e] [type:STACK] [id:19] [name:hdp-ambari-ui] Updating cluster status. stackId: 19, status: CREATE_FAILED, statusReason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.
/cbreak_cloudbreak_1 | 2017-01-25 10:40:16,364 [reactorDispatcher-42] fireEventAndLog:24 DEBUG c.s.c.c.f.s.FlowMessageService - [owner:12a1db55-ef2e-4b8c-aed1-c724e885f00e] [type:STACK] [id:19] [name:hdp-ambari-ui] AMBARI_CLUSTER_CREATE_FAILED [STACK_FLOW_STEP].
/cbreak_cloudbreak_1 | 2017-01-25 10:40:16,364 [reactorDispatcher-42] fireCloudbreakEvent:51 INFO  c.s.c.s.e.DefaultCloudbreakEventService - [owner:12a1db55-ef2e-4b8c-aed1-c724e885f00e] [type:STACK] [id:19] [name:hdp-ambari-ui] Firing Cloudbreak event: CloudbreakEventData{entityId=19, eventType='CREATE_FAILED', eventMessage='Ambari cluster could not be created. Reason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.'}
54167 /cbreak_cloudbreak_1 | 2017-01-25 10:40:16,365 [reactorDispatcher-42] accept:33 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:12a1db55-ef2e-4b8c-aed1-c724e885f00e] [type:STACK] [id:19] [name:hdp-ambari-ui] Handling cloudbreak event: Event{id=null, headers=null, replyTo=null, key=CLOUDBREAK_EVENT, data=CloudbreakEventData{entityId=19, eventType='CREATE_FAILED', eventMessage='Ambari cluster could not be created. Reason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.'}}
54174 /cbreak_cloudbreak_1 | 2017-01-25 10:40:16,365 [reactorDispatcher-42] accept:36 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Persisting data: CloudbreakEventData{entityId=19, eventType='CREATE_FAILED', eventMessage='Ambari cluster could not be created. Reason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.'}
54178 /cbreak_cloudbreak_1 | 2017-01-25 10:40:16,369 [reactorDispatcher-42] createStackEvent:66 DEBUG c.s.c.s.e.DefaultCloudbreakEventService - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Creating stack event from: CloudbreakEventData{entityId=19, eventType='CREATE_FAILED', eventMessage='Ambari cluster could not be created. Reason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.'}
54189 /cbreak_cloudbreak_1 | 2017-01-25 10:40:16,420 [reactorDispatcher-42] createStackEvent:71 INFO  c.s.c.s.e.DefaultCloudbreakEventService - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Created stack event: CloudbreakEvent{id=307, eventType='CREATE_FAILED', eventTimestamp=Wed Jan 25 10:40:16 UTC 2017, eventMessage='Ambari cluster could not be created. Reason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.', owner='12a1db55-ef2e-4b8c-aed1-c724e885f00e', account='seq1234567', cloud='AWS', region='ap-northeast-1', blueprintName='hdp-singlenode-2.5', instanceGroup='null', blueprintId=15', st        ackId=19', clusterId=18', clusterName=hdp-ambari-ui}
54190 /cbreak_cloudbreak_1 | 2017-01-25 10:40:16,421 [reactorDispatcher-42] accept:38 INFO  c.s.c.s.e.CloudbreakEventHandler - [owner:undefined] [type:CLOUDBREAKEVENTDATA] [id:undefined] [name:undefined] Sending notification with data: CloudbreakEvent{id=307, eventType='CREATE_FAILED', eventTimestamp=Wed Jan 25 10:40:16 UTC 2017, eventMessage='Ambari cluster could not be created. Reason: Cluster installation failed to complete, please check the Ambari UI for more details. You can try to reinstall the cluster with a different blueprint or fix the failures in Ambari and sync the cluster with Cloudbreak later.', owner='12a1db55-ef2e-4b8c-aed1-c724e885f00e', account='seq1234567', cloud='AWS', region='ap-northeast-1', blueprintName='hdp-singlenode-2.5', instanceGroup='null', blueprintId=15', stackId=        19', clusterId=18', clusterName=hdp-ambari-ui}

Actually, I can’t move on to the next screen by going to https:///ambari?>/ambari? "I can not move on to the next screen any more..."

sgowda · ‎01-25-2017

@ikawana

you don't need to install the Ambari server manually, cloudbreak will take care of installing it for you. If you click on the cluster, it will take you to a screen where Ambari UI link is displayed. Does that link work ?

Below is the screenshot of cluster details. This details screen has the Ambari Server Address

story0117 · ‎01-25-2017

@Santhosh B Gowda

It Doesn't works even if I click the "details" link...

story0117 · ‎01-26-2017

I am trying Cloudbreak today, but I can't resolve it yet...

Actually, When executing "cbd logs cloudbreak" command, the following error appears.

cloudbreak_1   | 2017-01-26 02:33:14,170 [http-nio-8080-exec-3] loadBlueprints:90 ERROR c.s.c.s.b.BlueprintLoaderService - [owner:5260b843-5700-487d-a297-3d3f80c14a54] [type:springLog] [id:] [name:] Blueprint is not available for 'java.io.FileNotFoundException: class path resource [defaults/blueprints/hdp-ephemeral.bp] cannot be opened because it does not exist' user.

$docker logs xxxxxxxx

xxxxx is docker container of hortonworks/cloudbreak-uaa-db:v3.6.0.

Below is the container logs.

ERROR:  duplicate key value violates unique constraint "groups_unique_key"
DETAIL:  Key (displayname, identity_zone_id)=(scim.write, uaa) already exists.
STATEMENT:  insert into groups ( id,displayName,description,created,lastModified,version,identity_zone_id ) values ($1,$2,$3,$4,$5,$6,$7)

msereg · ‎01-26-2017

@ikawana

the first exception is about a default blueprint cannot be found. It is caused by an improperly set default value, but it won't cause an issue.

the second exception shouldn't cause this kind of failure either.

So rather please provide me the following things:

- cbd/cloudbreak version and where is it deployed (locally, on aws, etc)

- please attach the full cloudbreak logs (docker logs -f cbreak_cloudbreak_1)

- are you trying to deploy the cluster with the default network settings that create a new VPC for the cluster?

- if this error message happens, you should already see your VMs running on EC2. Can you see those VMs in "running" state on the AWS console? Can you SSH to those instances? If yes do you see ambari-server running on the instance? What is the output of `sudo ambari-server status`?

story0117 · ‎01-27-2017

@Marton Sereg

: - cbd/cloudbreak version and where is it deployed (locally, on aws, etc)

Below is cdb/cloudbreak version.

And, It would like to deploy on AWS EC2(actually, I use community AMI(ami-cfc1bea8))

local version:1.6.2
latest release:1.6.2
docker images:
 hortonworks/haveged:1.1.0
 hortonworks/socat:1.0.0
 hortonworks/cloudbreak-uaa-db:v3.6.0
 hortonworks/cloudbreak-uaa:3.6.0
 hortonworks/cloudbreak-server-db:1.2.0
 hortonworks/cloudbreak:1.6.2
 hortonworks/cloudbreak-auth:1.6.2
 hortonworks/cloudbreak-web:1.6.2
 hortonworks/cloudbreak-autoscale-db:1.2.0
 hortonworks/cloudbreak-autoscale:1.6.2

: - please attach the full cloudbreak logs (docker logs -f cbreak_cloudbreak_1)

This logs is so large.

2017-01-27 01:51:59,956 [main] updateDefaultBlueprints:53 INFO  c.s.c.s.b.BlueprintLoaderService - [owner:spring] [type:springLog] [id:] [name:] Blueprint 3 is a default blueprint with name 'hdp-streaming-cluster'. Updating with the new config.
2017-01-27 01:51:59,969 [main] updateDefaultBlueprints:63 INFO  c.s.c.s.b.BlueprintLoaderService - [owner:spring] [type:springLog] [id:] [name:] Updating default blueprint with name 'hdp-ephemeral' wasn't successful because default blueprint file couldn't read: {}.
java.io.FileNotFoundException: class path resource [defaults/blueprints/hdp-ephemeral.bp] cannot be opened because it does not exist
        at org.springframework.core.io.ClassPathResource.getInputStream(ClassPathResource.java:172)
        at com.sequenceiq.cloudbreak.util.FileReaderUtils.readFileFromClasspath(FileReaderUtils.java:36)
        at com.sequenceiq.cloudbreak.service.blueprint.BlueprintUtils.readDefaultBlueprintFromFile(BlueprintUtils.java:23)
        at com.sequenceiq.cloudbreak.service.blueprint.BlueprintLoaderService.updateDefaultBlueprints(BlueprintLoaderService.java:49)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:354)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:305)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:133)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:408)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:772)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766)
        at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:307)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180)
        at com.sequenceiq.cloudbreak.CloudbreakApplication.main(CloudbreakApplication.java:22)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54)
        at java.lang.Thread.run(Thread.java:745)
2017-01-27 01:52:03,862 [main] validateSelectors:45 DEBUG c.s.c.r.i.ClusterPlatformInitializer - [owner:spring] [type:springLog] [id:] [name:] There are 14 handlers suitable for registering

:- are you trying to deploy the cluster with the default network settings that create a new VPC for the cluster?

I tried default network.

story0117 · ‎01-27-2017

@Marton Sereg

: - if this error message happens, you should already see your VMs running on EC2. Can you see those VMs in "running" state on the AWS console? Can you SSH to those instances? If yes do you see ambari-server running on the instance? What is the output of `sudo ambari-server status`?

I tried it over and over, so now the following error has appeared. How can we solve this?

I check the stack with aws cloudformation list-stacks, Then, aws cloudformation delete-stack --stack-name I deleted with the xxxxx command, but I could not delete the stack.

Caused by: com.sequenceiq.cloudbreak.cloud.exception.CloudConnectorException: AWS CloudFormation stack reached an error state: CREATE_FAILED reason: The maximum number of addresses has been reached.

msereg · ‎01-27-2017

@ikawana

"The maximum number of addresses has been reached" means that you've reached your quota of Elastic IPs on AWS. That is probably because you did not delete your CloudFormation stacks and have a few AWS resources stuck. You can terminate your clusters from the Cloudbreak UI - that deletes all the resources from AWS even if the cluster create failed before.

If you don't want to attach the full logs here you can send it to me via email (msereg hortonworks com), or please find the part with the original error message: ("Cluster installation failed to complete, please check the Ambari UI for more details") and attach the context of that - the previous messages that lead to the error message.

Support Questions

I can't create Cloudbreak/Ambari blueprint cluster in AWS