Member since
03-08-2017
3
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3184 | 03-08-2017 09:27 PM |
03-08-2017
09:27 PM
1 Kudo
I was able to spend more time looking into logs and figured that this is due to services not able to start. CM instance ran out of memory and went to unstable state. Increasing the memory resolved this issue.
... View more
03-08-2017
07:46 PM
While looking in the logs, I found that it is because the Cloudera Manager is going down with in few minutes of starting. Issues with Service Monitor and Host monitor. Not able to access CM after that. 2017-03-09 03:10:16,919 WARN 1216842385@scm-web-3:com.cloudera.server.web.cmf.StatusProvider: Failed to submit task for getting status from SERVICE_MONITORING com.cloudera.cmon.MgmtServiceNotRunningException: SERVICE_MONITORING is not running at com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:145) at com.cloudera.server.web.cmf.StatusProvider$SubjectStatusCustomFuture.<init>(StatusProvider.java:618) at com.cloudera.server.web.cmf.StatusProvider.getStatus(StatusProvider.java:1043) at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getServicesByType(ServiceManagerDaoImpl.java:581) at com.cloudera.api.dao.impl.ServiceManagerDaoImpl.getMgmtService(ServiceManagerDaoImpl.java:608) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:193) at com.cloudera.api.dao.impl.ManagerDaoBase.access$100(ManagerDaoBase.java:76) at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:234) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:124) at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:158) at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:269) at com.sun.proxy.$Proxy125.getMgmtService(Unknown Source) at com.cloudera.api.v1.impl.MgmtServiceResourceImpl.readService(MgmtServiceResourceImpl.java:42) at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl$RoleConfigGroupsResourceWrapper.<init>(MgmtServiceResourceV3Impl.java:38) at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl.getRoleConfigGroupsResource(MgmtServiceResourceV3Impl.java:26) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256) at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
... View more
03-08-2017
12:52 PM
I was able to start cloudera director successfully. successfully able to create a cloudera manager instance. Able to login and etc. When I setup a cluster with 1 master 1 gateway and 3 datanodes. Bootstrap failed. Was able to allocate all new instances. At the end of the set up, where it run firstRun it fails with connection refused. Below is the error. [2017-03-07 01:15:53.217 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-03-07 01:15:53.278 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [60, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ... [2017-03-07 01:15:53.465 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null} [2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Command CreateHiveDatabase with ID 60 completed successfully. Details: ApiCommand{id=60, name=CreateHiveDatabase, startTime=Tue Mar 07 01:15:52 UTC 2017, endTime=Tue Mar 07 01:15:55 UTC 2017, active=false, success=true, resultMessage=Created Hive Metastore Database., serviceRef=ApiServiceRef{peerName=null, clusterName=DevCluster, serviceName=CD-HIVE-NJZFPYye}, roleRef=null, hostRef=null, parent=null} [2017-03-07 01:16:08.525 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-03-07 01:16:08.563 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> InvokeFirstRunClusterCommandV7/3 [Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredentials{ ... [2017-03-07 01:16:08.609 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Calling firstRun on cluster DevCluster [2017-03-07 01:16:08.951 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-03-07 01:16:08.960 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> SetStatusJob/1 [Waiting for firstRun on cluster DevCluster] [2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.launchpad.pipeline.AbstractJob: Waiting for firstRun on cluster DevCluster [2017-03-07 01:16:08.961 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: << None{} [2017-03-07 01:16:08.971 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ... [2017-03-07 01:16:09.039 +0000] INFO [pipeline-thread-3] - c.c.l.b.UnboundedWaitForApiCommand: Waiting for ApiCommand{id=61, name=First Run, startTime=Tue Mar 07 01:16:08 UTC 2017, endTime=null, active=true, success=null, resultMessage=null, serviceRef=null, roleRef=null, hostRef=null, parent=null} [2017-03-07 01:16:24.048 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed javax.ws.rs.ProcessingException: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused) at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:596) at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:578) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:748) at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:231) at com.sun.proxy.$Proxy236.readCommand(Unknown Source) at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudera.api.ext.ClouderaManagerClientProxy.invoke(ClouderaManagerClientProxy.java:115) at com.sun.proxy.$Proxy236.readCommand(Unknown Source) at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.pollWhileActive(UnboundedWaitForApiCommand.java:113) at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:53) at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand.run(UnboundedWaitForApiCommand.java:33) at com.cloudera.launchpad.pipeline.job.Job4.runUnchecked(Job4.java:33) at com.cloudera.launchpad.pipeline.job.Job4$$FastClassBySpringCGLIB$$54178504.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:97) at com.cloudera.launchpad.pipeline.PipelineJobProfiler$1.call(PipelineJobProfiler.java:67) at com.codahale.metrics.Timer.time(Timer.java:101) at com.cloudera.launchpad.pipeline.PipelineJobProfiler.profileJobRun(PipelineJobProfiler.java:63) at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:629) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:618) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) at com.cloudera.launchpad.bootstrap.UnboundedWaitForApiCommand$$EnhancerBySpringCGLIB$$9cd7874c.runUnchecked(<generated>) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:164) at com.cloudera.launchpad.pipeline.util.PipelineRunner$JobCallable.call(PipelineRunner.java:135) at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) at com.github.rholder.retry.Retryer.call(Retryer.java:160) at com.cloudera.launchpad.pipeline.util.PipelineRunner.attemptMultipleJobExecutionsWithRetries(PipelineRunner.java:104) at com.cloudera.launchpad.pipeline.DatabasePipelineRunner.run(DatabasePipelineRunner.java:128) at com.cloudera.launchpad.ExceptionHandlingRunnable.run(ExceptionHandlingRunnable.java:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: ConnectException invoking http://10.142.0.3:7180/api/v6/commands/61: Connection refused (Connection refused) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1377) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1361) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:651) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:666) at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:746) ... 42 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1181) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:332) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1578) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1607) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1551) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1348) ... 48 common frames omitted [2017-03-07 01:16:34.049 +0000] INFO [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: >> UnboundedWaitForApiCommand/4 [61, Environment{name='Dev', provider=InstanceProviderConfig{type='google'}, credentials=SshCredenti ... [2017-03-07 01:16:34.055 +0000] ERROR [pipeline-thread-3] - c.c.l.pipeline.util.PipelineRunner: Attempt to execute job failed
... View more
Labels:
- Labels:
-
Apache Hive
-
Cloudera Manager
-
Gateway