- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Cloudera Manager server - Cannot recover key
Created on ‎09-07-2016 02:23 AM - edited ‎09-16-2022 03:38 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello all,
I have an installation of Cloudera Express and our admins have performed OpenStack upgrade during the last weekend. After the upgrade and reboot of the whole (virtualized) system, once I wanted to start Cloudera Manager server again (by service cloudera-scm-server start), I have encountered this error issue below, taken from:
/opt/cloudera/cloudera-manager/cm-5.7.1/log/cloudera-scm-server/cloudera-scm-server.log
I have Cloudera Express 5.7.1 on CentOS 6.6 with Java 1.8.0_60, Python 2.6.6 and using external MySQL 5.1.73. I had also all levels of TLS for Cloudera Manager set up before the upgrade, now it looks like the error in the keystore. I just want to add that no IP adresses used in the appropriate certificates didn't change, so I think this is not affecting it.
Can you please help me? Many thanks in advance!
2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent connections will use TLS 2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent TLS certificates will be validated. 2016-09-07 09:12:53,340 INFO MainThread:com.cloudera.server.cmf.Main: Agent RPC connections will use port: 7182 2016-09-07 09:12:53,391 INFO MainThread:org.mortbay.log: jetty-6.1.26.cloudera.4 2016-09-07 09:12:53,447 WARN MainThread:org.mortbay.log: failed SslSelectChannelConnector@0.0.0.0:7182: java.security.UnrecoverableKeyException: Cannot recover key 2016-09-07 09:12:53,448 WARN MainThread:org.mortbay.log: failed Server@1e454228: java.security.UnrecoverableKeyException: Cannot recover key 2016-09-07 09:12:53,448 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener. 2016-09-07 09:12:53,448 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed. org.apache.avro.AvroRuntimeException: java.security.UnrecoverableKeyException: Cannot recover key at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89) at com.cloudera.server.cmf.Main.startAgentServer(Main.java:571) at com.cloudera.server.cmf.Main.startAvro(Main.java:483) at com.cloudera.server.cmf.Main.run(Main.java:619) at com.cloudera.server.cmf.Main.main(Main.java:217) Caused by: java.security.UnrecoverableKeyException: Cannot recover key at sun.security.provider.KeyProtector.recover(KeyProtector.java:328) at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:146) at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineGetKey(KeyStoreDelegator.java:96) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetKey(JavaKeyStore.java:70) at java.security.KeyStore.getKey(KeyStore.java:1023) at sun.security.ssl.SunX509KeyManagerImpl.<init>(SunX509KeyManagerImpl.java:133) at sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:70) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:256) at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:651) at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87) ... 4 more 2016-09-07 09:12:53,869 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KAFKA-2.0.2-1.2.0.2.p0.5-jessie.parcel 2016-09-07 09:12:53,924 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KEYTRUSTEE-5.8.0-5.KEYTRUSTEE5.8.0.p0.21-jessie.parcel 2016-09-07 09:12:53,924 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : KEYTRUSTEE-5.8.0-5.KEYTRUSTEE5.8.0.p0.21-sles12.parcel 2016-09-07 09:12:55,019 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 404 URI: http://archive.cloudera.com/gplextras5/parcels/manifest.json 2016-09-07 09:12:55,309 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-5.8.0-1.cdh5.8.0.p0.42-sles12.parcel 2016-09-07 09:12:55,309 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Invalid parcel in manifest : CDH-5.8.0-1.cdh5.8.0.p0.42-jessie.parcel 2016-09-07 09:13:02,948 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully. 2016-09-07 09:13:48,121 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics. java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy88.writeMetrics(Unknown Source) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312) at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) ... 11 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1168) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1104) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:998) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:932) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1282) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257) at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72) ... 11 more 2016-09-07 09:13:54,899 INFO Thread-11:org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'rootContext': startup date [Wed Sep 07 09:12:24 CEST 2016]; parent: org.springframework.context.support.GenericApplicationContext@682b2fa 2016-09-07 09:13:54,905 INFO Thread-11:org.springframework.beans.factory.support.DefaultListableBeanFactory: Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2401856: defining beans [contextApplicationContextProvider,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,sessionRegistry,passwordEncoder,predefinedPlots,workAggregatesConfigListener,predefinedViews,monitoringTypesInitializer,metricSchemaManagerBean,viewFactory,metricSchemaGeneration,csdRegistryImpl,csdLocalRepository,mdlRegistry,csdTranslationManager,csdManager,validatorConfiguration,parameterFactory,securityUtils,sslHelper,dssdToggleListener.PostCommit,dssdToggleListener.PreCommit,oozieLoadBalancerConfigUpdateListener,HBaseIndexerAuthenticationConfigUpdateListener,scmParamTrackerStoreImpl,dynamicServiceHandlerFactory,runnerDescriptorProcessFactory,configWriterFactory,auxConfigGeneratorFactory,peerConfigGeneratorFactory,compatibilityFactory,providesFactory,configGeneratorFactory,kerberosPrincProvider,processStalenessInterceptor,processStalenessDetector,configHelper,processHelper,paramResolver,releaseDetector,zkServerInitListener,solrAuthenticationConfigUpdateListener,solrLoadBalancerConfigUpdateListener,HBaseZkConfigUpdateListener,hbaseRestServerSecurityListener,hbaseThriftServerSecurityListener,callableFactory,commandManager,stalenessChecker,commandStorage,diagnosticsDataUploadHelper,cmfSchedulerImpl,scheduleManagerImpl,dirtyParametersListener,descriptorFactory,clientProtocolImpl,idleSessionManagerImpl,sessionServiceImpl,hostTemplateManagerImpl,actionablesProviderImpl,cloudStatusDeterminer,heartbeatCheckerImpl,beanConfiguration,serviceDataProviderBean,scmDbValueStore,clouderaManagerMetricsForwarder,firehoseRequestService,operationalReportsDisabledListener,scmActive,embeddedDbManager,licenseManagerImpl,navigatorDisabledListener,cmServerState,userSettingTransactionManagerImpl,currentUserManagerImpl,trialEventAuditor,authorizer,operationsManagerImpl,cmUpgradeHelper,licensedFeatureManager,trialEventStalenessCheckTrigger,jythonObjectFactoryImpl,pythonInterpreterFactory,logSearchEventsCollectorImpl,agentLogFetcherImpl,serverLogFetcherImpl,ServerLogSearchResponse,parcelManagerImpl,parcelStatusProviderImpl,parcelDependencyManagerImpl,localParcelManagerImpl,parcelInstallerImpl,parcelDownloaderImpl,parcelUpdateService,periodicParcelTasks,parcelRepoConfigUpdateListener,agentParcelProviderImpl,prototypeFactory,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,rulesEngine,builtInServiceTypes,builtInRoleTypes,builtInNamesForCrossEntityAggregateMetrics,builtInMetricEntityAttributes,builtInMetricEntityTypes,uniqueFieldValidator,validServiceDependencyValidator,uniqueServiceTypeValidator,uniqueRoleTypeValidator,existingServiceTypeValidator,existingRoleTypeValidator,expressionValidator,autoConfigSharesValidValidator,messageInterpolator,sdlParser,mdlParser,parcelParser,alternativesParser,permissionsParser,manifestParser,stringInterpolator,serviceDescriptorValidatorWithoutDependencyCheck,serviceDescriptorValidatorWithDependencyCheck,serviceMonitoringDefinitionsDescriptorValidator,descriptorVisitor,referenceValidator,parcelDescriptorValidator,alternativesDescriptorValidator,permissionsDescriptorValidator,manifestDescriptorValidator,defaultValidatorConfiguration,springConstraintValidatorFactory,validatorFactoryBean,metricNameFormatValidator,nameForCrossEntityAggregateFormatValidator,objectMapper,getObjectMapper,agentAsyncClient,newHeartbeatRequester,commandRequestsBean,getSupportedLocale,newServiceHandlerRegistry,newEventStoreClientFactory,newAutoUpgradeHandlerRegistry,newUpgradeHandlerRegistry,newAgentResultFetcher,newCmfEntityManager,newDatabaseSizeGauge,newCdhExecutorFactory,databaseExecutor]; parent: org.springframework.beans.factory.support.DefaultListableBeanFactory@3d34d211 2016-09-07 09:13:54,909 INFO Thread-11:com.cloudera.server.cmf.components.ScmActive: ScmActive shutting down. 2016-09-07 09:13:54,916 INFO metric-schema-updater:com.cloudera.cmon.components.MetricSchemaManager: Breaking from sleep in schema update thread.
Created ‎09-07-2016 03:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So we have found the problem, sorry for early post, but maybe for somebody else it can be helpful:
For all the aliases stored in keystores must be used the same password, as said here:
http://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_sg_tls_browser.html
- Set -keypass to the same value as -storepass. Cloudera Manager assumes that the same password is used to access both the key and the keystore, and therefore, does not support separate values for -keypass and -storepass.
We changed these passwords so they are the same now, changed the passwords also in our external MySQL database and then restart the Cloudera Manager server and it worked fine.
Created ‎09-07-2016 03:18 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So we have found the problem, sorry for early post, but maybe for somebody else it can be helpful:
For all the aliases stored in keystores must be used the same password, as said here:
http://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_sg_tls_browser.html
- Set -keypass to the same value as -storepass. Cloudera Manager assumes that the same password is used to access both the key and the keystore, and therefore, does not support separate values for -keypass and -storepass.
We changed these passwords so they are the same now, changed the passwords also in our external MySQL database and then restart the Cloudera Manager server and it worked fine.
