Support Questions
Find answers, ask questions, and share your expertise

How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Explorer

We're trying to take advantage of the custom AMI patch that @rdoktorics described in this post in our environment in order to use RHEL instead of Amazon Linux. However, we get this error when trying to use a Red Hat image:

Failed to connect ssh: Exhausted available authentication methods

The deployment of the cluster eventually times out and fails completely.

Can someone please describe why this might be happening?

Is there any way to detail the bootstrapping steps required to prepare an RHEL instance for Cloudbreak to deploy on?

Best,

Michael

18 REPLIES 18

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

@mjrhee

I may have misunderstood your question. Do you mean install the CloudBreak management software on a VM? Or are you referring to the VMs that CloudBreak creates as part of the cluster deployment process?

You can deploy CloudBreak on your own VM. Have you seen this: http://sequenceiq.com/cloudbreak-docs/latest/

You can install the Cloudbreak Deployer on your own VM/host manually. Once Cloudbreak Deployer is installed, use it to set up the Cloudbreak Application. We suggest that you install the Cloudbreak Application as close to your desired HDP clusters as possible. For example, if you plan to launch clusters on AWS, install the Cloudbreak Application on AWS.

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Explorer

Hi Michael,

Yes, I think you've misunderstood me. What I'm trying to do is use a completely different AMI on AWS instead of the default that Cloudbreak offers. I want to start this from a scratch RHEL base image and customize it with some packages in addition to whatever Cloudbreak needs to deploy HDP. I'm looking for some guidance on what kinds of bootstrapping is required.

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Perhaps this is what you are looking for?

http://sequenceiq.com/cloudbreak-docs/latest/onprem/

Once you have the CloudBreak deployer setup, you should be able to create an AMI from the instance.

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Explorer

I've been able to work past the ssh issue by installing the Docker cert-tool on the server. But now I see a new error related to the SaltOrchestrator:

cloudbreak_1 | 2016-08-10 04:15:09,840 [reactorDispatcher-42] isBootstrapApiAvailable:186 INFO  c.s.c.o.s.SaltOrchestrator - [owner:spring] [type:springLog] [id:] [name:] Failed to connect to bootstrap app MessageBodyReader not found for media type=text/html, type=class com.sequenceiq.cloudbreak.orchestrator.model.GenericResponse, genericType=class com.sequenceiq.cloudbreak.orchestrator.model.GenericResponse.

The error appears to be linked to a failed HTTP request to "http://127.0.0.1:7070/saltboot/health" on nginx.

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Contributor

I'd check what the default AMI the cloudbreak uses is, spawn an EC2 instance from that, apply your customization, create a new AMI and then instruct cloudbreak to use your new private AMI.

This way you don't have to worry about what cloudbreak needs, you're starting from cloudbreak's default config and customizing forward from there.

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

New Contributor

Hi Eric,

The problem is the default AMI is based on Amazon Linux. It's not consistent to other cloud provider. What I want to achieve is to have a Centos based AMI.

Regards,

Wendell

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Expert Contributor

It's a little bit complicated, the bootstrap app which CB failed to connect to is a little application written in golang. You can find it's source code here: https://github.com/sequenceiq/salt-bootstrap you need this application to start during the VM boot and listen on port 7070. Cloudbreak will SSH into the instance during boot with a temp SSH key using 'cloudbreak' as user then it will launch an nginx with some mappings and one of them is the salt-bootstrap app. Once it's done CB removes the temp SSH key. Once the nginx and the go application is running it will send it's HTTP requests to the app to bootstrap the machines. Basically CB tells the IPs to the bootstrap app to launch the salt master and minions.

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Explorer

Thank you, @khorvath. That all makes sense and is what I suspected. Here is as much of the bootstrapping that I could figure out based on the error messages coming out of Cloudbreak and what I could find on the default base image:

  • Docker cert-tool
  • nginx
  • salt 2015.5.8 (Lithium)
  • user-data-helper.sh script
  • salt-bootstrap (application that you mentioned)

With all these in place, I can get past the "Failed to connect to bootstrap app MessageBodyReader" error, but then run into an issue connecting to the salt-bootstrap api (IP is address X'd out):

cloudbreak_1 | 2016-08-10 15:31:32,692 [reactorDispatcher-42] createClient:48 INFO  o.t.m.e.s.TerracottaClusteredInstanceFactory - [owner:spring] [type:springLog] [id:] [name:] Constructing jax rs client for config: server cert: /certs/stack-16/ca.pem, client cert: /certs/stack-16/cert.pem, debug: false
cloudbreak_1 | 2016-08-10 15:31:32,699 [reactorDispatcher-42] createClient:74 INFO  o.t.m.e.s.TerracottaClusteredInstanceFactory - [owner:spring] [type:springLog] [id:] [name:] Jax rs client has been constructed: org.glassfish.jersey.client.JerseyClient@5967b56e, sslContext: javax.net.ssl.SSLContext@2060098c
cloudbreak_1 | 2016-08-10 15:31:32,731 [reactorDispatcher-42] isBootstrapApiAvailable:186 INFO  c.s.c.o.s.SaltOrchestrator - [owner:spring] [type:springLog] [id:] [name:] Failed to connect to bootstrap app org.apache.http.conn.HttpHostConnectException: Connect to XX.XX.XX.XX:9443 [/XX.XX.XX.XX] failed: Connection refused

I suspect it isn't running in nginx like it's supposed to, but that's as much as I can tell. From what I've seen, nginx starts with all the mappings you mentioned, but I still get this error. Is there something else I'm missing?

Would love to hear your thoughts.

Best,

Michael

Re: How to Bootstrap a Custom AMI on AWS for Cloudbreak 1.3?

Expert Contributor

Here's the nginx conf: https://github.com/sequenceiq/cloudbreak/blob/master/core/src/main/resources/init/host/nginx.conf

and the script we run during the temp SSH connection: https://github.com/sequenceiq/cloudbreak/blob/master/core/src/main/resources/init/host/tls-setup.sh so if you install the salt-bootstrap app:

: ${CLOUDBREAK_BOOTSTRAP_VERSION:=0.1.2}

curl -Lo /tmp/shared/salt-bootstrap_${CLOUDBREAK_BOOTSTRAP_VERSION}_Linux_x86_64.tgz https://github.com/sequenceiq/salt-bootstrap/releases/download/v${CLOUDBREAK_BOOTSTRAP_VERSION}/salt...
  tar -zxf /tmp/shared/salt-bootstrap_${CLOUDBREAK_BOOTSTRAP_VERSION}_Linux_x86_64.tgz -C /usr/sbin/

with the following systemd config:

[Install]
WantedBy=multi-user.target


[Unit]
Description=Salt Bootstrap Service
After=network.target network.service cloud-final.service waagent.service google-startup-scripts.service
Wants=cloud-final.service waagent.service google-startup-scripts.service


[Service]
Restart=on-failure
TimeoutSec=15s
ExecStart=/usr/sbin/salt-bootstrap
Environment='SALTBOOT_PORT=7070'


chmod +x /usr/sbin/salt-bootstrap
    systemctl enable salt-bootstrap

The app will start on port 7070 and if CB launchen nginx with the conf above it will map it to /saltboot then you should be good to go.