Reply
New Contributor
Posts: 4
Registered: ‎02-16-2016

How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

After I upgraded Cloudera into version 5.5, I realized Cloudera hides hive metastore password in hive-site.xml.

See hive-site.xml password ******** in path /var/run/cloudera-scm-agent/process/ below:

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive1</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>********</value>
  </property>

Based on latest Cloudera 5.5 Documentation page 24, it says:

 

Starting with Cloudera Manager and CDH 5.5, passwords will no longer be accessible in cleartext through the Cloudera Manager UI or in the configuration files stored on disk. For components such as HDFS, HBase, Hive, etc. that use core Hadoop, the feature has been implemented by using Hadoop's CredentialProvider interface to encrypt and store passwords inside a secure creds.jceks keystore file

 

The problem is I want to change my current Postgresql database into MySQL and a step for doing that is Postgresql SQL dump. Without having the password (I forget the password), I wont be able to move the data into MySQL. How to retrieve the password from creds.jceks file?

 

creds.localjceks.PNG

 

The content of creds.localjceks is encrypted.

 

Thanks

Anto

 

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

Hi Anto,

You can retrieve the password using the API:
https://cloudera.github.io/cm_api/

Look for a configuration property on the Hive Service.
https://cloudera.github.io/cm_api/apidocs/v11/path__clusters_-clusterName-_services_-serviceName-_co...

You can also use a DB admin account to create your postgresql dump. If you are using the postgresql that comes with CM, that admin username and password is located in /var/lib/cloudera-scm-server-db/data/generated_password.txt (or something similar, that's from memory).
New Contributor
Posts: 4
Registered: ‎02-16-2016

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

Hi Darren,

 

Thanks for the quick reply. Since my intention was to do postgresql dump, I opted to 2nd way by using DB Admin because I never use Cloudera Rest API before. It works! However, I would like to address the 2nd step so anyone who has the same problem as me can solve it immediately. Make sure to connect to correct Postgresql instance set by Cloudera (default port 7432 in CM). Otherwise we will connect to default postgresql installation 5432. I use command psql -h localhost -d scm -p 7432 -U scm -W. Make sure from localhost, since the default configuration is for localhost connection only.

 

Explorer
Posts: 10
Registered: ‎12-14-2015

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

I have a variation of this issue with CDH 5.5.3. 

 

I know the password for Hive userid but metastore server do not start and fails with error message:

Hive Metastore Startup Errors:

Role failed to start due to error com.cloudera.cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks.

 

Any recommendation? 

 

Thanks

Sudhir

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

Is there any interesting message in your CM server log? /var/log/cloudera-scm-server/cloudera-scm-server.log
Explorer
Posts: 10
Registered: ‎12-14-2015

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

I see following:

 

2016-04-15 20:17:57,603 WARN 190754513@agentServer-966:com.cloudera.server.cmf.AgentProtocolImpl: (1 skipped) Received optimized heartbeat from eeclxvm44.unx.sas.com even though we have no previous state. Master was probably restarted between requests. The next heartbeat will be complete.
2016-04-15 20:18:00,723 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.
2016-04-15 20:18:02,484 INFO 809128795@scm-web-3567:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=43, name=hive} Role: DbRole{id=346, name=hive-HIVEMETASTORE-27e3bb97ca1870af148e09312e24f3c9, hostName=eeclxvm48.na.sas.com}
2016-04-15 20:18:02,558 WARN 809128795@scm-web-3567:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfiguration(AbstractRoleHandler.java:768)
at com.cloudera.cmf.service.hive.BaseHiveRoleHandler.makeProcess(BaseHiveRoleHandler.java:78)
at com.cloudera.cmf.service.GenericBringUpRoleCommand$RoleBringUpCmdWork.doWork(GenericBringUpRoleCommand.java:218)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:164)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:98)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:118)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:836)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:791)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:786)
at com.cloudera.server.cmf.components.OperationsManagerImpl.executeRoleCommand(OperationsManagerImpl.java:1737)
at com.cloudera.server.web.cmf.InstancesController.roleInstancesDispatch(InstancesController.java:412)

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

There should be more to that stack trace. Can you include it?
Explorer
Posts: 10
Registered: ‎12-14-2015

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

sorry about that. Please find full stack the moment I tried to start metastore server on cm console.

 

 


2016-04-15 20:51:14,191 INFO 190754513@agentServer-966:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbHost.name (#10) has changed]}.
2016-04-15 20:51:15,409 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT1.206S
2016-04-15 20:51:22,929 INFO 1755936940@scm-web-3611:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Restart BasicCmdArgs{args=[]}. Service: DbService{id=43, name=hive} Role: DbRole{id=346, name=hive-HIVEMETASTORE-27e3bb97ca1870af148e09312e24f3c9, hostName=eeclxvm48.na.sas.com}
2016-04-15 20:51:22,942 INFO 1755936940@scm-web-3611:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=43, name=hive} Role: DbRole{id=346, name=hive-HIVEMETASTORE-27e3bb97ca1870af148e09312e24f3c9, hostName=eeclxvm48.na.sas.com}
2016-04-15 20:51:23,003 WARN 1755936940@scm-web-3611:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfiguration(AbstractRoleHandler.java:768)
at com.cloudera.cmf.service.hive.BaseHiveRoleHandler.makeProcess(BaseHiveRoleHandler.java:78)
at com.cloudera.cmf.service.GenericBringUpRoleCommand$RoleBringUpCmdWork.doWork(GenericBringUpRoleCommand.java:218)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:164)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:98)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:118)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:836)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:791)
at com.cloudera.cmf.command.flow.CmdWorkCtx.execRoleCmd(CmdWorkCtx.java:140)
at com.cloudera.cmf.command.flow.work.ExecRoleCmdWork.doWork(ExecRoleCmdWork.java:56)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:164)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:98)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:118)
at com.cloudera.cmf.service.AbstractRestartCommands$GenericRestartRoleCommand.executeImpl(AbstractRestartCommands.java:187)
at com.cloudera.cmf.service.AbstractRoleCommand.execute(AbstractRoleCommand.java:64)
at com.cloudera.cmf.service.AbstractRoleCommand.execute(AbstractRoleCommand.java:24)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:836)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:791)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:786)
at com.cloudera.server.cmf.components.OperationsManagerImpl.executeRoleCommand(OperationsManagerImpl.java:1737)
at com.cloudera.server.web.cmf.InstancesController.roleInstancesDispatch(InstancesController.java:412)
at com.cloudera.server.web.cmf.InstancesController$$FastClassByCGLIB$$17ed6885.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.cloudera.server.web.cmf.InstancesController$$EnhancerByCGLIB$$947eb503.roleInstancesDispatch(<generated>)
at sun.reflect.GeneratedMethodAccessor2202.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: com.cloudera.cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks
at com.cloudera.cmf.service.config.JceksConfigFileGenerator.generate(JceksConfigFileGenerator.java:74)
at com.cloudera.cmf.service.HandlerUtil.emitConfigFiles(HandlerUtil.java:123)
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfiguration(AbstractRoleHandler.java:766)
... 98 more
Caused by: java.security.KeyStoreException
at com.sun.crypto.provider.JceKeyStore.engineSetKeyEntry(JceKeyStore.java:283)
at java.security.KeyStore.setKeyEntry(KeyStore.java:909)
at com.cloudera.cmf.service.config.JceksConfigFileGenerator.generate(JceksConfigFileGenerator.java:64)
... 100 more
2016-04-15 20:51:23,013 INFO 1755936940@scm-web-3611:com.cloudera.cmf.service.GenericBringUpRoleCommand: BringUp failed for command (223208) on service hive for role 346/hive-HIVEMETASTORE-27e3bb97ca1870af148e09312e24f3c9; final role status STOPPED, process running status false
2016-04-15 20:51:23,013 INFO 1755936940@scm-web-3611:com.cloudera.cmf.service.GenericBringUpRoleCommand: BringUp comm

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

What version of java are you running? What OS?

The exception is coming from Java classes, so it could be some strange JDK bug.

Does Hive have any particularly strange characters in passwords? Or empty passwords?
Explorer
Posts: 10
Registered: ‎12-14-2015

Re: How to Retrieve Hive Metastore DB Password on Cloudera 5.5 (creds.localjceks, creds.jceks)

Darren - 

 

I tried to change password (both empty etc and it did not work). 

 

Basically (for some odd reason) cloudera-scm-agent was not able to pick right $java path (all though till that point all java seems to work) to generate local.ecjks file.  I tried to run a certain $ hadoop credential list which led to be invalid $JAVA_HOME. I fixed /etc/profile and restarted cloudera-manager, agent and entire cluster. I restarted hive metastore server which generated creds.localjceks 

 

The problem is fixed and metastore server is working. 

 

One add thing is - how did all java get executed till then (as can be seen in log files) and then when it came to generating local.ecjks file it did not find. Looks as if there are 2 processes in play where first one found correct java path and second one getting $JAVA_HOME from /etc/profile did not. Any thoughts..

 

Sudhir