Hello all,
I have an installation of Cloudera Express and our admins have performed OpenStack upgrade during the last weekend. After the upgrade and reboot of the whole (virtualized) system, once I wanted to start Cloudera Manager server again (by service cloudera-scm-server start), I have encountered this error issue below, taken from:
/opt/cloudera/cloudera-manager/cm-5.7.1/log/cloudera-scm-server/cloudera-scm-server.log
I have Cloudera Express 5.7.1 on CentOS 6.6 with Java 1.8.0_60, Python 2.6.6 and using external MySQL 5.1.73. I had also all levels of TLS for Cloudera Manager set up before the upgrade, now it looks like the error in the keystore. I just want to add that no IP adresses used in the appropriate certificates didn't change, so I think this is not affecting it.
Can you please help me? Many thanks in advance!
2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent connections will use TLS
2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent TLS certificates will be validated.
2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent RPC connections will use port: 7182
2016-09-07 09:12:53,391 INFO MainThread:org.mortbay.log: jetty-6.1.26.cloudera.4
2016-09-07 09:12:53,447 WARN MainThread:org.mortbay.log: failed SslSelectChannelConnector@0.0.0.0:7182: java.security.UnrecoverableKeyException: Cannot recover key
2016-09-07 09:12:53,448 WARN MainThread:org.mortbay.log: failed Server@1e454228: java.security.UnrecoverableKeyException: Cannot recover key
2016-09-07 09:12:53,448 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2016-09-07 09:12:53,448 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.security.UnrecoverableKeyException: Cannot recover key
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
at com.cloudera.server.cmf.Main.startAgentServer(Main.java:571)
at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
at com.cloudera.server.cmf.Main.run(Main.java:619)
at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:146)
at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:96)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetKey(JavaKeyStore.java:70)
at java.security.KeyStore.getKey(KeyStore.java:1023)
at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133)
at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70)
at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256)
at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:651)
at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
... 4 more
2016-09-07 09:12:53,869 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KAFKA-2.0.2-1.2.0.2.p0.5-jessie.parcel
2016-09-07 09:12:53,924 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KEYTRUSTEE-5.8.0-5.KEYTRUSTEE5.8.0.p0.21-jessie.parcel
2016-09-07 09:12:53,924 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KEYTRUSTEE-5.8.0-5.KEYTRUSTEE5.8.0.p0.21-sles12.parcel
2016-09-07 09:12:55,019 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/gplextras5/parcels/manifest.json
2016-09-07 09:12:55,309 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-5.8.0-1.cdh5.8.0.p0.42-sles12.parcel
2016-09-07 09:12:55,309 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-5.8.0-1.cdh5.8.0.p0.42-jessie.parcel
2016-09-07 09:13:02,948 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully.
2016-09-07 09:13:48,121 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy88.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1282)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 11 more
2016-09-07 09:13:54,899 INFO Thread-11:org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'rootContext': startup date [Wed Sep 07 09:12:24 CEST 2016]; parent: org.springframework.context.support.GenericApplicationContext@682b2fa
2016-09-07 09:13:54,905 INFO Thread-11:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2401856: defining beans [contextApplicationContextProvider,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,sessionRegistry,passwordEncoder,predefinedPlots,workAggregatesConfigListener,predefinedViews,monitoringTypesInitializer,metricSchemaManagerBean,viewFactory,metricSchemaGeneration,csdRegistryImpl,csdLocalRepository,mdlRegistry,csdTranslationManager,csdManager,validatorConfiguration,parameterFactory,securityUtils,sslHelper,dssdToggleListener.PostCommit,dssdToggleListener.PreCommit,oozieLoadBalancerConfigUpdateListener,HBaseIndexerAuthenticationConfigUpdateListener,scmParamTrackerStoreImpl,dynamicServiceHandlerFactory,runnerDescriptorProcessFactory,configWriterFactory,auxConfigGeneratorFactory,peerConfigGeneratorFactory,compatibilityFactory,providesFactory,configGeneratorFactory,kerberosPrincProvider,processStalenessInterceptor,processStalenessDetector,configHelper,processHelper,paramResolver,releaseDetector,zkServerInitListener,solrAuthenticationConfigUpdateListener,solrLoadBalancerConfigUpdateListener,HBaseZkConfigUpdateListener,hbaseRestServerSecurityListener,hbaseThriftServerSecurityListener,callableFactory,commandManager,stalenessChecker,commandStorage,diagnosticsDataUploadHelper,cmfSchedulerImpl,scheduleManagerImpl,dirtyParametersListener,descriptorFactory,clientProtocolImpl,idleSessionManagerImpl,sessionServiceImpl,hostTemplateManagerImpl,actionablesProviderImpl,cloudStatusDeterminer,heartbeatCheckerImpl,beanConfiguration,serviceDataProviderBean,scmDbValueStore,clouderaManagerMetricsForwarder,firehoseRequestService,operationalReportsDisabledListener,scmActive,embeddedDbManager,licenseManagerImpl,navigatorDisabledListener,cmServerState,userSettingTransactionManagerImpl,currentUserManagerImpl,trialEventAuditor,authorizer,operationsManagerImpl,cmUpgradeHelper,licensedFeatureManager,trialEventStalenessCheckTrigger,jythonObjectFactoryImpl,pythonInterpreterFactory,logSearchEventsCollectorImpl,agentLogFetcherImpl,serverLogFetcherImpl,ServerLogSearchResponse,parcelManagerImpl,parcelStatusProviderImpl,parcelDependencyManagerImpl,localParcelManagerImpl,parcelInstallerImpl,parcelDownloaderImpl,parcelUpdateService,periodicParcelTasks,parcelRepoConfigUpdateListener,agentParcelProviderImpl,prototypeFactory,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,rulesEngine,builtInServiceTypes,builtInRoleTypes,builtInNamesForCrossEntityAggregateMetrics,builtInMetricEntityAttributes,builtInMetricEntityTypes,uniqueFieldValidator,validServiceDependencyValidator,uniqueServiceTypeValidator,uniqueRoleTypeValidator,existingServiceTypeValidator,existingRoleTypeValidator,expressionValidator,autoConfigSharesValidValidator,messageInterpolator,sdlParser,mdlParser,parcelParser,alternativesParser,permissionsParser,manifestParser,stringInterpolator,serviceDescriptorValidatorWithoutDependencyCheck,serviceDescriptorValidatorWithDependencyCheck,serviceMonitoringDefinitionsDescriptorValidator,descriptorVisitor,referenceValidator,parcelDescriptorValidator,alternativesDescriptorValidator,permissionsDescriptorValidator,manifestDescriptorValidator,defaultValidatorConfiguration,springConstraintValidatorFactory,validatorFactoryBean,metricNameFormatValidator,nameForCrossEntityAggregateFormatValidator,objectMapper,getObjectMapper,agentAsyncClient,newHeartbeatRequester,commandRequestsBean,getSupportedLocale,newServiceHandlerRegistry,newEventStoreClientFactory,newAutoUpgradeHandlerRegistry,newUpgradeHandlerRegistry,newAgentResultFetcher,newCmfEntityManager,newDatabaseSizeGauge,newCdhExecutorFactory,databaseExecutor]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@3d34d211
2016-09-07 09:13:54,909 INFO Thread-11:com.cloudera.server.cmf.components.ScmActive: ScmActive shutting down.
2016-09-07 09:13:54,916 INFO metric-schema-updater:com.cloudera.cmon.components.MetricSchemaManager: Breaking from sleep in schema update thread.
