Cloudera Manager server - Cannot recover key

surovecv — Fri, 16 Sep 2022 10:38:11 GMT

Hello all,

I have an installation of Cloudera Express and our admins have performed OpenStack upgrade during the last weekend. After the upgrade and reboot of the whole (virtualized) system, once I wanted to start Cloudera Manager server again (by service cloudera-scm-server start), I have encountered this error issue below, taken from:

/opt/cloudera/cloudera-manager/cm-5.7.1/log/cloudera-scm-server/cloudera-scm-server.log

I have Cloudera Express 5.7.1 on CentOS 6.6 with Java 1.8.0_60, Python 2.6.6 and using external MySQL 5.1.73. I had also all levels of TLS for Cloudera Manager set up before the upgrade, now it looks like the error in the keystore. I just want to add that no IP adresses used in the appropriate certificates didn't change, so I think this is not affecting it.

Can you please help me? Many thanks in advance!

2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent connections will use TLS
2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent TLS certificates will be validated.
2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent RPC connections will use port: 7182
2016-09-07 09:12:53,391 INFO MainThread:org.mortbay.log: jetty-6.1.26.cloudera.4
2016-09-07 09:12:53,447 WARN MainThread:org.mortbay.log: failed SslSelectChannelConnector@0.0.0.0:7182: java.security.UnrecoverableKeyException: Cannot recover key
2016-09-07 09:12:53,448 WARN MainThread:org.mortbay.log: failed Server@1e454228: java.security.UnrecoverableKeyException: Cannot recover key
2016-09-07 09:12:53,448 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2016-09-07 09:12:53,448 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.security.UnrecoverableKeyException: Cannot recover key
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
        at com.cloudera.server.cmf.Main.startAgentServer(Main.java:571)
        at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
        at com.cloudera.server.cmf.Main.run(Main.java:619)
        at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
        at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
        at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:146)
        at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:96)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetKey(JavaKeyStore.java:70)
        at java.security.KeyStore.getKey(KeyStore.java:1023)
        at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133)
        at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70)
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256)
        at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:651)
        at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at org.mortbay.jetty.Server.doStart(Server.java:235)
        at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
        at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
        ... 4 more
2016-09-07 09:12:53,869 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KAFKA-2.0.2-1.2.0.2.p0.5-jessie.parcel
2016-09-07 09:12:53,924 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KEYTRUSTEE-5.8.0-5.KEYTRUSTEE5.8.0.p0.21-jessie.parcel
2016-09-07 09:12:53,924 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KEYTRUSTEE-5.8.0-5.KEYTRUSTEE5.8.0.p0.21-sles12.parcel
2016-09-07 09:12:55,019 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/gplextras5/parcels/manifest.json
2016-09-07 09:12:55,309 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-5.8.0-1.cdh5.8.0.p0.42-sles12.parcel
2016-09-07 09:12:55,309 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-5.8.0-1.cdh5.8.0.p0.42-jessie.parcel
2016-09-07 09:13:02,948 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully.
2016-09-07 09:13:48,121 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
        at com.sun.proxy.$Proxy88.writeMetrics(Unknown Source)
        at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
        at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
        at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
        at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
        ... 11 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
        at sun.net.www.http.HttpClient.New(HttpClient.java:308)
        at sun.net.www.http.HttpClient.New(HttpClient.java:326)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1282)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
        at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
        at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
        at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
        at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
        at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
        ... 11 more
2016-09-07 09:13:54,899 INFO Thread-11:org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'rootContext': startup date [Wed Sep 07 09:12:24 CEST 2016]; parent: org.springframework.context.support.GenericApplicationContext@682b2fa
2016-09-07 09:13:54,905 INFO Thread-11:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2401856: defining beans [contextApplicationContextProvider,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,sessionRegistry,passwordEncoder,predefinedPlots,workAggregatesConfigListener,predefinedViews,monitoringTypesInitializer,metricSchemaManagerBean,viewFactory,metricSchemaGeneration,csdRegistryImpl,csdLocalRepository,mdlRegistry,csdTranslationManager,csdManager,validatorConfiguration,parameterFactory,securityUtils,sslHelper,dssdToggleListener.PostCommit,dssdToggleListener.PreCommit,oozieLoadBalancerConfigUpdateListener,HBaseIndexerAuthenticationConfigUpdateListener,scmParamTrackerStoreImpl,dynamicServiceHandlerFactory,runnerDescriptorProcessFactory,configWriterFactory,auxConfigGeneratorFactory,peerConfigGeneratorFactory,compatibilityFactory,providesFactory,configGeneratorFactory,kerberosPrincProvider,processStalenessInterceptor,processStalenessDetector,configHelper,processHelper,paramResolver,releaseDetector,zkServerInitListener,solrAuthenticationConfigUpdateListener,solrLoadBalancerConfigUpdateListener,HBaseZkConfigUpdateListener,hbaseRestServerSecurityListener,hbaseThriftServerSecurityListener,callableFactory,commandManager,stalenessChecker,commandStorage,diagnosticsDataUploadHelper,cmfSchedulerImpl,scheduleManagerImpl,dirtyParametersListener,descriptorFactory,clientProtocolImpl,idleSessionManagerImpl,sessionServiceImpl,hostTemplateManagerImpl,actionablesProviderImpl,cloudStatusDeterminer,heartbeatCheckerImpl,beanConfiguration,serviceDataProviderBean,scmDbValueStore,clouderaManagerMetricsForwarder,firehoseRequestService,operationalReportsDisabledListener,scmActive,embeddedDbManager,licenseManagerImpl,navigatorDisabledListener,cmServerState,userSettingTransactionManagerImpl,currentUserManagerImpl,trialEventAuditor,authorizer,operationsManagerImpl,cmUpgradeHelper,licensedFeatureManager,trialEventStalenessCheckTrigger,jythonObjectFactoryImpl,pythonInterpreterFactory,logSearchEventsCollectorImpl,agentLogFetcherImpl,serverLogFetcherImpl,ServerLogSearchResponse,parcelManagerImpl,parcelStatusProviderImpl,parcelDependencyManagerImpl,localParcelManagerImpl,parcelInstallerImpl,parcelDownloaderImpl,parcelUpdateService,periodicParcelTasks,parcelRepoConfigUpdateListener,agentParcelProviderImpl,prototypeFactory,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,rulesEngine,builtInServiceTypes,builtInRoleTypes,builtInNamesForCrossEntityAggregateMetrics,builtInMetricEntityAttributes,builtInMetricEntityTypes,uniqueFieldValidator,validServiceDependencyValidator,uniqueServiceTypeValidator,uniqueRoleTypeValidator,existingServiceTypeValidator,existingRoleTypeValidator,expressionValidator,autoConfigSharesValidValidator,messageInterpolator,sdlParser,mdlParser,parcelParser,alternativesParser,permissionsParser,manifestParser,stringInterpolator,serviceDescriptorValidatorWithoutDependencyCheck,serviceDescriptorValidatorWithDependencyCheck,serviceMonitoringDefinitionsDescriptorValidator,descriptorVisitor,referenceValidator,parcelDescriptorValidator,alternativesDescriptorValidator,permissionsDescriptorValidator,manifestDescriptorValidator,defaultValidatorConfiguration,springConstraintValidatorFactory,validatorFactoryBean,metricNameFormatValidator,nameForCrossEntityAggregateFormatValidator,objectMapper,getObjectMapper,agentAsyncClient,newHeartbeatRequester,commandRequestsBean,getSupportedLocale,newServiceHandlerRegistry,newEventStoreClientFactory,newAutoUpgradeHandlerRegistry,newUpgradeHandlerRegistry,newAgentResultFetcher,newCmfEntityManager,newDatabaseSizeGauge,newCdhExecutorFactory,databaseExecutor]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@3d34d211
2016-09-07 09:13:54,909 INFO Thread-11:com.cloudera.server.cmf.components.ScmActive: ScmActive shutting down.
2016-09-07 09:13:54,916 INFO metric-schema-updater:com.cloudera.cmon.components.MetricSchemaManager: Breaking from sleep in schema update thread.

Re: Cloudera Manager server - Cannot recover key

surovecv — Wed, 07 Sep 2016 10:18:31 GMT

So we have found the problem, sorry for early post, but maybe for somebody else it can be helpful:

For all the aliases stored in keystores must be used the same password, as said here:

http://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_sg_tls_browser.html

Set -keypass to the same value as -storepass. Cloudera Manager assumes that the same password is used to access both the key and the keystore, and therefore, does not support separate values for -keypass and -storepass.

We changed these passwords so they are the same now, changed the passwords also in our external MySQL database and then restart the Cloudera Manager server and it worked fine.

question Cloudera Manager server - Cannot recover key in Archives of Support Questions (Read Only)

Cloudera Manager server - Cannot recover key

Re: Cloudera Manager server - Cannot recover key