Created on 11-16-2014 08:43 PM - edited 09-16-2022 02:13 AM
When creating a new cloudera manager via the cloudera director it fails with the following error under Ubuntu (which is listed as supported in the documentation)
[2014-11-17 03:07:04] WARN [reader] - c.c.l.sshj.TrustAnyHostKeyVerifier: Host key for 172.31.1.150 was automatically accepted
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: Filesystem Size Used Avail Use% Mounted on
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: /dev/xvda1 59G 827M 56G 2% /
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 4.0K 0 4.0K 0% /sys/fs/cgroup
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: udev 492M 12K 492M 1% /dev
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: tmpfs 100M 328K 99M 1% /run
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 5.0M 8.0K 5.0M 1% /run/lock
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 497M 0 497M 0% /run/shm
[2014-11-17 03:07:05] INFO [io-thread-1] - ssh:172.31.1.150: none 100M 0 100M 0% /run/user
[2014-11-17 03:07:05] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: << None{}
[2014-11-17 03:07:05] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...
[2014-11-17 03:07:05] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UBUNTU
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(BootstrapClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
[2014-11-17 03:07:05] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...
[2014-11-17 03:07:06] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UBUNTU
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(BootstrapClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
[2014-11-17 03:07:06] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...
[2014-11-17 03:07:06] ERROR [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
java.lang.UnsupportedOperationException: Operating system type not supported: UBUNTU
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.futureInstallClouderaManagerRepositories(BootstrapClouderaManager.java:131) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:82) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.deployment.BootstrapClouderaManager.run(BootstrapClouderaManager.java:44) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job2.runUnchecked(Job2.java:31) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
[2014-11-17 03:07:06] INFO [pipeline-thread-1] - c.c.l.p.DatabasePipelineRunner: >> BootstrapClouderaManager/2 [CreateDeploymentContext{environment=Environment{name='Testing', provider=InstanceProviderConfig{typ ...
Created 11-16-2014 08:52 PM
Created 11-16-2014 08:52 PM
Created 11-16-2014 09:46 PM
Deploying centos 6.4 seems to be working as expected. Do you know when ubuntu will be supported?
Created 11-16-2014 09:53 PM
I guess I spoke too soon. The manager is deployed according to the UI, but ther is no process listening on port 7180 so connecting to the manager fails. Futhermore bootstraping a new cluster also fails with the following error:
[2014-11-17 05:47:31] ERROR [pipeline-thread-38] - c.c.l.p.DatabasePipelineRunner: Attempt to execute job failed
javax.ws.rs.client.ClientException: org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:548) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:534) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:545) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at com.sun.proxy.$Proxy151.hostInstallCommand(Unknown Source) ~[na:na]
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:91) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.bootstrap.cluster.BootstrapClouderaManagerAgent$HostInstall.run(BootstrapClouderaManagerAgent.java:81) ~[launchpad-bootstrap-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.job.Job3.runUnchecked(Job3.java:32) ~[launchpad-pipeline-1.0.1.jar!/:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner$1.call(DatabasePipelineRunner.java:229) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[guava-retrying-1.0.6.jar!/:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) ~[guava-retrying-1.0.6.jar!/:na]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.attemptMultipleJobExecutionsWithRetries(DatabasePipelineRunner.java:213) ~[launchpad-pipeline-database-1.0.1.jar!/
:1.0.1]
at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:132) ~[launchpad-pipeline-database-1.0.1.jar!/:1.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_33]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_33]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_33]
Caused by: org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543) ~[cxf-rt-frontend-jaxrs-2.7.5.jar!/:2.7.5]
... 16 common frames omitted
Caused by: java.net.ConnectException: ConnectException invoking http://172.31.9.89:7180/api/v6/cm/commands/hostInstall: Connection refused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.6.0_33]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.6.0_33]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.6.0_33]
at java.lang.reflect.Constructor.newInstance(Constructor.java:534) ~[na:1.6.0_33]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) ~[cxf-api-2.7.5.jar!/:2.7.5]
... 19 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.6.0_33]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) ~[na:1.6.0_33]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) ~[na:1.6.0_33]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) ~[na:1.6.0_33]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385) ~[na:1.6.0_33]
at java.net.Socket.connect(Socket.java:546) ~[na:1.6.0_33]
at sun.net.NetworkClient.doConnect(NetworkClient.java:173) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:409) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:240) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.New(HttpClient.java:321) ~[na:1.6.0_33]
at sun.net.www.http.HttpClient.New(HttpClient.java:338) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801) ~[na:1.6.0_33]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:979) ~[na:1.6.0_33]
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:168) ~[cxf-rt-transports-http-2.
7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1282) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1233) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:195) ~[cxf-rt-transports-http-2.7.5.ja
r!/:2.7.5]
at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:69) ~[cxf-api-2.7.5.jar!/:2.7.5]
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1295) ~[cxf-rt-transports-http-2.7.5.jar!/:2.7.5]
... 22 common frames omitted
Created 11-16-2014 09:54 PM
Created 11-16-2014 10:01 PM
Created 11-17-2014 08:11 AM
As I understand it 7180 is the manager port and the IP im seeing in the logs corresponds with the manager IP. All of my security groups are setup such that all cluster instances are allowed to talk to the director and manager on any port. After creating another cluster as a test (I didn't want to leave my instance running over night) I've run in to the same issue. Logging in to the manager IP address and checking netstat (`netstat -tulpn`) for open connections I do not see anything listening on port 7180.
Here is the relevant netstat output for the director and manager nodes.
Director:
# sudo netstat -tulpn | grep java
tcp 0 0 0.0.0.0:7189 0.0.0.0:* LISTEN 13081/java
Manager:
# sudo netstat -tulpn | grep java
tcp 0 0 0.0.0.0:7186 0.0.0.0:* LISTEN 11280/java
tcp 0 0 0.0.0.0:7187 0.0.0.0:* LISTEN 4119/java
tcp 0 0 0.0.0.0:10101 0.0.0.0:* LISTEN 4097/java
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 11280/java
On the manager node I've noticed that the cloudera-scm-server process does not appear to start correctly.
# /etc/init.d/cloudera-scm-server status
cloudera-scm-server dead but pid file exists
After restarting this service and waiting several minutes the port finally appears open in netstat. I will attempt to launch another cluster now that I have verified the manager service is listening. If I can provide any further logs or details please let me know.
For what its worth I am launching these instances as t2.micro since I'm trying to evaluate the product and I'm hesitant to spend more money launching test clusters until I know that it works.
Created 11-17-2014 11:24 AM
t2.micro is definitely too small to run all the services. You will immediately get memory exceptions and the JVMs will fail to start for various services (not only Cloudera Manager). t2.micro is also too small to run Director itself.
For a small development type of setup I would recommend m1.medium or larger (it depends on how much data you want to process with the cluster).
If your goal is to use Hue and experiment with various products have you considered Cloudera Live?
Created 11-17-2014 11:33 AM
The director actually doesn't perform terribly on a t2.micro and I don't recall getting any exceptions or OOM errors. The manager and cluster nodes were the issue. We're already running a CDH 5 cluster with Spark. I'm interested in using director as a replacement to our current methods of creating/deploying/scaling/managing clusters. Will try creating a more realistic cluster now.