Cloudera Manager bootstrap fails for UBUNTU

[#1]

When creating a new cloudera manager via the cloudera director it fails with the following error under Ubuntu (which is listed as supported in the documentation)

[2014-11-17 03:07:04] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 172.31.1.150 was automatically accepted
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: Filesystem Size Used Avail Use% Mounted on
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: /dev/xvda1 59G 827M 56G 2% /
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 4.0K 0 4.0K 0% /sys/fs/cgroup
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: udev 492M 12K 492M 1% /dev
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: tmpfs 100M 328K 99M 1% /run
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 5.0M 8.0K 5.0M 1% /run/lock
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 497M 0 497M 0% /run/shm
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 100M 0 100M 0% /run/user
[2014-11-17 03:07:05] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: << None{}
[2014-11-17 03:07:05] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...
[2014-11-17 03:07:05] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UBUNTU
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(BootstrapClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
[2014-11-17 03:07:05] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...

[2014-11-17 03:07:06] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UBUNTU
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(BootstrapClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
[2014-11-17 03:07:06] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...
[2014-11-17 03:07:06] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UBUNTU
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(BootstrapClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
[2014-11-17 03:07:06] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...

[#2 · Accepted solution]

Thanks for trying out Director! The Director Server and Client software will run on all supported platforms (as listed in the CDH and CM requirements page). However Director can only deploy clusters with nodes that run Red Hat 6.4, CentOS6.4 and 6.5. Ubuntu will be supported in due course but for now please use one of the above mentioned OSes.

[#3]

Deploying centos 6.4 seems to be working as expected. Do you know when ubuntu will be supported? 

[#4]

I guess I spoke too soon. The manager is deployed according to the UI, but ther is no process listening on port 7180 so connecting to the manager fails. Futhermore bootstraping a new cluster also fails with the following error:

[2014-11-17 05:47:31] ERROR [pipeline-thread-38] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
javax.ws.rs.client.ClientException: org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:548) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:534) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:545) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at com.sun.proxy.$Proxy151.hostInstallCommand(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:91) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:81) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/
:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
Caused by: org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
... 16 common frames omitted
Caused by: java.net.ConnectException: ConnectException invoking http://172.31.9.89:7180/api/v6/cm/commands/hostInstall: Connection refused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_33]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.6.0_33]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.6.0_33]
at java.lang.reflect.Constructor.newInstance(Constructor.java:534) ~[na:1.6.0_33]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ~[cxf-api-2.7.5.jar!/:2.7.5]
... 19 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_33]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) ~[na:1.6.0_33]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) ~[na:1.6.0_33]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) ~[na:1.6.0_33]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385) ~[na:1.6.0_33]
at java.net.Socket.connect(Socket.java:546) ~[na:1.6.0_33]
at sun.net.NetworkClient.doConnect(NetworkClient.java:173) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:409) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:240) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.New(HttpClient.java:321) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.New(HttpClient.java:338) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:979) ~[na:1.6.0_33]
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:168) ~[cxf-rt-transports-http-2.
7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1282) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1233) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:195) ~[cxf-rt-transports-http-2.7.5.ja
r!/:2.7.5]
at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1295) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]

... 22 common frames omitted

[#5]

We are actively working on adding more platforms but I don't have a timeline to share as of now. Sorry about that.

Now that you have used Director, did you have any feedback about the product or anything you'd like to see in future versions?

[#6]

The error is connection refused:
Caused by: java.net.ConnectException: ConnectException invoking http://172.31.9.89:7180/api/v6/cm/commands/hostInstall: Connection refused

Are you able to connect to 172.31.9.89:7180 from host where Director is installed? To function correctly, both Director and the cluster it is building should be part of the same subnet so that Director can easily access all the boxes. Could you confirm this by running

# telnet 172.31.9.89 7180

on the host running Director?

[#7]

As I understand it 7180 is the manager port and the IP im seeing in the logs corresponds with the manager IP. All of my security groups are setup such that all cluster instances are allowed to talk to the director and manager on any port. After creating another cluster as a test (I didn't want to leave my instance running over night) I've run in to the same issue. Logging in to the manager IP address and checking netstat (`netstat -tulpn`) for open connections I do not see anything listening on port 7180.

Here is the relevant netstat output for the director and manager nodes.

Director:

# sudo netstat -tulpn | grep java
tcp 0 0 0.0.0.0:7189 0.0.0.0:* LISTEN 13081/java

Manager:

# sudo netstat -tulpn | grep java
tcp 0 0 0.0.0.0:7186 0.0.0.0:* LISTEN 11280/java
tcp 0 0 0.0.0.0:7187 0.0.0.0:* LISTEN 4119/java
tcp 0 0 0.0.0.0:10101 0.0.0.0:* LISTEN 4097/java
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 11280/java

On the manager node I've noticed that the cloudera-scm-server process does not appear to start correctly.

# /etc/init.d/cloudera-scm-server status
cloudera-scm-server dead but pid file exists

After restarting this service and waiting several minutes the port finally appears open in netstat. I will attempt to launch another cluster now that I have verified the manager service is listening. If I can provide any further logs or details please let me know.

For what its worth I am launching these instances as t2.micro since I'm trying to evaluate the product and I'm hesitant to spend more money launching test clusters until I know that it works.

[#8]

t2.micro is definitely too small to run all the services. You will immediately get memory exceptions and the JVMs will fail to start for various services (not only Cloudera Manager). t2.micro is also too small to run Director itself. 

For a small development type of setup I would recommend m1.medium or larger (it depends on how much data you want to process with the cluster).

If your goal is to use Hue and experiment with various products have you considered Cloudera Live?

[#9]

The director actually doesn't perform terribly on a t2.micro and I don't recall getting any exceptions or OOM errors. The manager and cluster nodes were the issue. We're already running a CDH 5 cluster with Spark. I'm interested in using director as a replacement to our current methods of creating/deploying/scaling/managing clusters. Will try creating a more realistic cluster now.