@bgooley    I tried to restart and there is no process directory getscreated for Navigator Metastore.   I had tail the cloudera agent log for server where Navigator Metadata is running. Cloudera agent throws below logs when i restarted Navigator Metastore.   Cloudera Agent log:   [25/Jun/2019 08:30:32 +0000] 2813189 CP Server Thread-5 _cplogging INFO 10.26.132.165 - - [25/Jun/2019:08:30:32] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0" [25/Jun/2019 08:30:34 +0000] 2813189 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for conf.cloudera.spark2_on_yarn with count 0 pipelines names []. [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin __init__ INFO Read metadata config for role GATEWAY. Metadata server url: https://a301-7781-0339.ldn.swissbank.com:7187 [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: [] [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for conf.cloudera.spark2_on_yarn with count 0 pipelines names []. [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_thread INFO Log entry poll interval: 5 [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_thread INFO Navigator server url: https://a301-7781-0339.ldn.swissbank.com:7187 [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin inotify_event_processor INFO Setting watch for directories: /var/log/hive/lineage [25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_logs INFO 9 navigator log files with prefix hive-HIVESERVER2-220e59e0745b944070fe360a2cc717c7 for role hive-HIVESERVER2-220e59e0745b944070fe360a2cc717c7 exist in navigator log directory /var/log/hive/lineage     Cloudera Sever throws below logs.   2019-06-25 08:33:07,341 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259) 2019-06-25 08:33:07,372 INFO 439059796@scm-web-6908:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=2, name=mgmt} Role: DbRole{id=324, name=mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7, hostName=a301-8744-7259.ldn.swissbank.com} 2019-06-25 08:33:07,372 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-06-25 08:33:07,372 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Start a role 2019-06-25 08:33:07,884 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution java.util.NoSuchElementException at com.google.common.collect.Iterators$1.next(Iterators.java:77) at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325) at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271) at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43) at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99) at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.getEnvironmentForRole(BaseMgmtRoleHandler.java:123) at com.cloudera.cmf.service.AbstractDaemonRoleHandler.getEnvironment(AbstractDaemonRoleHandler.java:347) at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.makeProcess(BaseMgmtRoleHandler.java:73) at com.cloudera.cmf.service.GenericBringUpRoleCommand$RoleBringUpCmdWork.doWork(GenericBringUpRoleCommand.java:203) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:943) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:898) at com.cloudera.cmf.command.flow.CmdWorkCtx.execRoleCmd(CmdWorkCtx.java:145) at com.cloudera.cmf.command.flow.work.ExecRoleCmdWork.doWork(ExecRoleCmdWork.java:58) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117) at com.cloudera.cmf.service.AbstractRestartCommands$GenericRestartRoleCommand.executeImpl(AbstractRestartCommands.java:187) at com.cloudera.cmf.service.AbstractRoleCommand.execute(AbstractRoleCommand.java:64) at com.cloudera.cmf.service.AbstractRoleCommand.execute(AbstractRoleCommand.java:24) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:943) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:898) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:893) at com.cloudera.server.cmf.components.OperationsManagerImpl.executeRoleCommand(OperationsManagerImpl.java:1811) at com.cloudera.server.web.cmf.InstancesController.internalRoleInstancesDispatch(InstancesController.java:469) at com.cloudera.server.web.cmf.InstancesController.access$100(InstancesController.java:76) at com.cloudera.server.web.cmf.InstancesController$3.apply(InstancesController.java:437) at com.cloudera.server.web.cmf.InstancesController$3.apply(InstancesController.java:434) at com.cloudera.server.web.cmf.InvocationUtil$1.call(InvocationUtil.java:36) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:108) at com.cloudera.server.web.cmf.InvocationUtil.invoke(InvocationUtil.java:32) at com.cloudera.server.web.cmf.InstancesController.roleInstancesDispatch(InstancesController.java:433) at com.cloudera.server.web.cmf.InstancesController$$FastClassByCGLIB$$17ed6885.invoke(<generated>) at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) at com.cloudera.server.web.cmf.InstancesController$$EnhancerByCGLIB$$16b0d4df.roleInstancesDispatch(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2019-06-25 08:33:07,885 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Role failed to start due to error null. 2019-06-25 08:33:08,032 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Role 324/mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7 Process cloudera-mgmt-NAVIGATORMETASERVER does not exist - unable to update status 2019-06-25 08:33:08,032 INFO 439059796@scm-web-6908:com.cloudera.cmf.model.DbCommand: Command 7269(Start) has completed. finalstate:FINISHED, success:false, msg:Failed to start role. 2019-06-25 08:33:08,032 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Failed to start role. 2019-06-25 08:33:08,032 INFO 439059796@scm-web-6908:com.cloudera.cmf.model.DbCommand: Command 7268(Restart) has completed. finalstate:FINISHED, success:false, msg:Failed to start role. 2019-06-25 08:33:08,447 INFO 439059796@scm-web-6908:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbConfig.valueForDb (#814) has changed]}. 2019-06-25 08:33:08,448 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence 2019-06-25 08:33:08,448 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Configuration Staleness Check 2019-06-25 08:33:09,456 INFO avro-servlet-hb-processor-2:com.cloudera.server.common.AgentAvroServlet: (23 skipped) AgentAvroServlet: heartbeat processing stats: average=1136ms, min=719ms, max=2947ms. 2019-06-25 08:33:10,359 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Running staleness check with FULL_CHECK for 44/74 roles. 2019-06-25 08:33:10,432 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Total staleness check tasks: 77   - Vijay M ... View more

@lwang    Below 2 Directories have generated.   drwxr-x--x. 3 root root 180 Jun 21 12:34 572-MGMT.NAVIGATOR-test-db-connection drwxr-x--x. 3 root root 180 Jun 21 12:34 571-MGMT.NAVIGATORMETASERVER-test-db-connection   I checked in both directories for logs, Below snap for it. But its not helping much.   Directory: 571-MGMT.NAVIGATORMETASERVER-test-db-connection   root@abc% pwd /run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection/logs root@abc% ls -lrt total 8 -rw-r--r--. 1 root root 3676 Jun 20 12:25 stderr.log -rw-r--r--. 1 root root 394 Jun 20 12:25 stdout.log   root@abc% cat stderr.log Thu Jun 20 12:25:58 BST 2019 + date + date Thu Jun 20 12:25:58 BST 2019 ++ dirname /usr/lib64/cmf/service/dbconnection/test_db_connection.sh + cloudera_config=/usr/lib64/cmf/service/dbconnection ++ cd /usr/lib64/cmf/service/dbconnection/../common ++ pwd + cloudera_config=/usr/lib64/cmf/service/common + . /usr/lib64/cmf/service/common/cloudera-config.sh ++ export JAVA_HOME=/app/bds/java ++ JAVA_HOME=/app/bds/java ++ export 'JAVA_TOOL_OPTIONS=-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf' ++ JAVA_TOOL_OPTIONS='-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf' ++ export KRB5_CONFIG=/etc/krb5_bds.conf ++ KRB5_CONFIG=/etc/krb5_bds.conf ++ set -x + locate_java_home + locate_java_home_no_verify + JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6') + local JAVA6_HOME_CANDIDATES + OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk') + local OPENJAVA6_HOME_CANDIDATES + JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle') + local JAVA7_HOME_CANDIDATES + OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk') + local OPENJAVA7_HOME_CANDIDATES + JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle') + local JAVA8_HOME_CANDIDATES + OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk') + local OPENJAVA8_HOME_CANDIDATES + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk') + local MISCJAVA_HOME_CANDIDATES + case ${BIGTOP_JAVA_MAJOR} in + JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]}) + '[' -z /app/bds/java ']' + verify_java_home + '[' -z /app/bds/java ']' + echo JAVA_HOME=/app/bds/java + source_parcel_environment + '[' '!' -z '' ']' + JAVA=/app/bds/java/bin/java + [[ -z /usr/share/cmf ]] + MGMT_CLASSPATH='/usr/share/cmf/lib/*' + JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar + MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' ++ pwd + MGMT_CLASSPATH='/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' + MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true ' + exec /app/bds/java/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf   root@abc% cat stdout.log Thu Jun 20 12:25:58 BST 2019 Current working directory: /run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection Launching one-off process: /usr/lib64/cmf/service/dbconnection/test_db_connection.sh db.properties Thu Jun 20 12:25:58 BST 2019 JAVA_HOME=/app/bds/java [ main] DbCommandExecutor INFO Successfully connected to database.         Directory: /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection/logs   root@abc% pwd /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection/logs root@abc% ls -lrt total 8 -rw-r--r--. 1 root root 3656 Jun 20 12:25 stderr.log -rw-r--r--. 1 root root 384 Jun 20 12:25 stdout.log     root@abc% cat stderr.log Thu Jun 20 12:25:58 BST 2019 + date + date Thu Jun 20 12:25:58 BST 2019 ++ dirname /usr/lib64/cmf/service/dbconnection/test_db_connection.sh + cloudera_config=/usr/lib64/cmf/service/dbconnection ++ cd /usr/lib64/cmf/service/dbconnection/../common ++ pwd + cloudera_config=/usr/lib64/cmf/service/common + . /usr/lib64/cmf/service/common/cloudera-config.sh ++ export JAVA_HOME=/app/bds/java ++ JAVA_HOME=/app/bds/java ++ export 'JAVA_TOOL_OPTIONS=-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf' ++ JAVA_TOOL_OPTIONS='-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf' ++ export KRB5_CONFIG=/etc/krb5_bds.conf ++ KRB5_CONFIG=/etc/krb5_bds.conf ++ set -x + locate_java_home + locate_java_home_no_verify + JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6') + local JAVA6_HOME_CANDIDATES + OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk') + local OPENJAVA6_HOME_CANDIDATES + JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle') + local JAVA7_HOME_CANDIDATES + OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk') + local OPENJAVA7_HOME_CANDIDATES + JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle') + local JAVA8_HOME_CANDIDATES + OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk') + local OPENJAVA8_HOME_CANDIDATES + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk') + local MISCJAVA_HOME_CANDIDATES + case ${BIGTOP_JAVA_MAJOR} in + JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]}) + '[' -z /app/bds/java ']' + verify_java_home + '[' -z /app/bds/java ']' + echo JAVA_HOME=/app/bds/java + source_parcel_environment + '[' '!' -z '' ']' + JAVA=/app/bds/java/bin/java + [[ -z /usr/share/cmf ]] + MGMT_CLASSPATH='/usr/share/cmf/lib/*' + JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar + MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' ++ pwd + MGMT_CLASSPATH='/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' + MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true ' + exec /app/bds/java/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf     root@abc% cat stdout.log Thu Jun 20 12:25:58 BST 2019 Current working directory: /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection Launching one-off process: /usr/lib64/cmf/service/dbconnection/test_db_connection.sh db.properties Thu Jun 20 12:25:58 BST 2019 JAVA_HOME=/app/bds/java [ main] DbCommandExecutor INFO Successfully connected to database. ... View more

@bgooley    I need your help once again.   This is for Hue connecting to hiveservers2 instances using haproxy load balancer.   Below is haproxy configuration for hiveserver2 to connect using hue, Kindly revert if any mistake in below configuration.   frontend hivejdbc_front bind *:10003 mode tcp option tcplog timeout client 720m timeout server 720m default_backend hive-hue #--------------------------------------------------------------------- # source balancing between the various backends #--------------------------------------------------------------------- backend hive-hue balance source mode tcp server hs2_1 a301-9941-0809.ldn.swissbank.com:10001 check server hs2_2 a301-9941-1309.ldn.swissbank.com:10001 check   - Update hue config property Hue Service Advanced Configuration Snippet (Safety Valve) for hue_safety_valve.ini from CM UI and added below into it, Kindly confirm if its not correct.   [beeswax] hive_server_host=a301-9941-0727.ldn.swissbank.com hive_server_port=10003     Also help me with haproxy config for any ODBC connectivity to hive from any BI tools.   - Vijay M ... View more

@bgooley   Can you help me with some example for user_attribute_mapping='{"uid":"username"}'  ?   My requirement is:   Post integrating with SAML when user try to access Hue UI, If IDP returns UID of user post authentication but when user logged in Hue UI shows Username which i want to be displayed.   Can you help me to understand about below property which says   user_attribute_mapping  - Map user attributed provided by IDP to Hue attributes   What is Hue attributes?   - Vijay M ... View more

@bgooley   After setting up Hiveserver2 HA and Impala using haproxy does   1. Any configuration needs to be done in Hue 2. While Connecting to Hive and impala through Hue any additional configuration in haproxy require?   - Vijay M ... View more

@bgooley   As suggested i have checked for SAN recreated certificate with SAN name defined in it with hostname of hapoxy server.   Checked and verified, haproxy forwarded the requests to both hiveserver2 instace, verified through logs.   Few questions:   1. What are different type of balance algorithm in haproxy like mentioned below ? balance source   2. What the difference between source, leastconn, roundrobin, etc?   - Vijay M ... View more

@bgooley   Did some more troubleshooting.    Updated alogrithm for load balance to roundrobin from source as mentioned below in haproxy configuration and started to get errors in hiveserver2 logs.   frontend hiveserver2_front bind *:10001 option tcplog mode tcp default_backend hiveserver2 backend hiveserver2 mode tcp balance roundrobin server hs2_1 a301-8883-0447.gdzd.ubs.net:10000 check server hs2_2 a301-8883-2675.gdzd.ubs.net:10000 check   When tried to connect from beeline from any server my 2nd connection always to server xyz and connection getting failed and hiveserver2 logs of xyz server throws below error.   2018-11-19 08:48:05,964 ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-44]: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:794) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:791) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1904) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:791) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 16 more       Kindly suggest ?     - Vijay M ... View more

  @bgooley   I have updated haproxy configuration as you provided. Below snap for the same. global log 127.0.0.1 local2 pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # stats socket /tmp/haproxy # tune.ssl.default-dh-param 2048 defaults mode http log global option httplog option dontlognull # option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 10m timeout server 10m timeout check 10s maxconn 3000   listen admin bind *:8000 stats enable   frontend hiveserver2_front bind *:10001 option tcplog mode tcp default_backend hiveserver2 backend hiveserver2 mode tcp balance source server hs2_1 abc:10000 server hs2_2 xyz:10000 restarted haproxy and it started with current timestamp. Check and verified. Below find configuration info haproxy service and 1st hiveserver instance running on server : abc 2nd hiveserver instance running on server : xyz Scenario 1: Both hiveserver2 instance up and running. Connecting from beeline from server abc, throws below error. beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0) Connecting from beeline from server xyz is successful, Below snap for the same. beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM scan complete in 2ms Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://abc:1> show databases; Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable. Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0) 0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 1: jdbc:hive2://abc:1> show databases; INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds INFO : OK +----------------+--+ | database_name | +----------------+--+ | default | | test1 | | test | +----------------+--+   Scenario 2: hiveserver2 instance stopped on server xyz Connecting from beeline from server abc, throws below error. beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0)   Connecting from beeline from server xyz is successful, Below snap for the same. beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM scan complete in 2ms Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@REALM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://abc:1> show databases; Unexpected end of file when reading from HS2 server. The root cause might be too many concurrent connections. Please ask the administrator to check the number of active connections, and adjust hive.server2.thrift.max.worker.threads if applicable. Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0) 0: jdbc:hive2://abc:1> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connected to: Apache Hive (version 1.1.0-cdh5.15.1) Driver: Hive JDBC (version 1.1.0-cdh5.15.1) Transaction isolation: TRANSACTION_REPEATABLE_READ 1: jdbc:hive2://abc:1> show databases; INFO : Compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.757 seconds INFO : Executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896): show databases INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20181119073535_1c5b4e65-5007-4629-99e6-ab45f32c4896); Time taken: 0.66 seconds INFO : OK +----------------+--+ | database_name | +----------------+--+ | default | | test1 | | test | +----------------+--+   Scenario 3: hiveserver2 instance stopped on server abc Connecting from beeline from server abc, throws below error. beeline> !connect jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Connecting to jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@REALM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://abc:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=*****;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching abc found. (state=08S01,code=0) Connecting from beeline from server xyz is failed, Below snap for the same. Request from haproxy seems not getting forwarded to xyzs server when abc is not available. beeline> !connect jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM scan complete in 1ms Connecting to jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://a301-8883-0447.gdzd.ubs.net:10001/default;ssl=true;sslTrustStore=/app/bds/security/pki/cloudera_truststore.jks;sslTrustPassword=Cldkeystore1;principal=hive/_HOST@BDS-DR.UBS.COM: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake (state=08S01,code=0)     Kindly check for all above scenario and suggest. - VIjay M ... View more