Skip to main content
VijayM
VijayM
Silver ⭐️⭐️⭐️
June 20, 2019
Solved

Cloudera Navigator Metadata Server not starting

Hello team,

 

Cloudera Navigator Metadata Server failed to start, no log file getting updated in /var/log/cloudera-scm-navigator for metaddata server.

 

Only Cloudera server shows below errors.

 

CDH version: 5.16.1

 

Kindly suggest?

 

Error:

 

2019-06-20 13:49:21,099 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)
2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=2, name=mgmt} Role: DbRole{id=324, name=mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7, hostName=a301-8744-7259.ldn.swissbank.com}
2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Start a role
2019-06-20 13:49:21,389 WARN CommandPusher:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.util.NoSuchElementException
at com.google.common.collect.Iterators$1.next(Iterators.java:77)
at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325)
at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271)
at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43)
at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99)
at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.getEnvironmentForRole(BaseMgmtRoleHandler.java:123)
at com.cloudera.cmf.service.AbstractDaemonRoleHandler.getEnvironment(AbstractDaemonRoleHandler.java:347)
at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.makeProcess(BaseMgmtRoleHandler.java:73)
at com.cloudera.cmf.service.GenericBringUpRoleCommand$RoleBringUpCmdWork.doWork(GenericBringUpRoleCommand.java:203)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:943)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:898)
at com.cloudera.cmf.command.flow.CmdWorkCtx.execRoleCmd(CmdWorkCtx.java:145)
at com.cloudera.cmf.command.flow.work.ExecRoleCmdWork.doWork(ExecRoleCmdWork.java:58)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.work.ScatterCmdWork.doWork(ScatterCmdWork.java:86)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
at com.cloudera.cmf.command.flow.CmdWorkCtx.execSvcCmd(CmdWorkCtx.java:159)
at com.cloudera.cmf.command.flow.work.ExecSvcCmdWork.doWork(ExecSvcCmdWork.java:91)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.SeqCmdWork.update(SeqCmdWork.java:180)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:180)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.flow.SeqFlowCmd.update(SeqFlowCmd.java:249)
at com.cloudera.cmf.command.CmdWorkCommand.update(CmdWorkCommand.java:144)
at com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:456)
at com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:270)
at com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:189)
at com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:157)
2019-06-20 13:49:21,389 ERROR CommandPusher:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7187 Role failed to start due to error null.

bgooley
bgooleyAnswer
Gold ⭐️
July 9, 2019

@VijayM,

 

I was out on PTO for a couple weeks so I just saw your comment.  This looks like a pretty unusual issue.

 

Based on the fact that the agent never seems to know that it should start the role, this is a problem that is occurring in Cloudera Manager only (before the command is to be started by the agent).

 

The stack trace and error messages show us something interesting:

 

(1)

 

CM sees that you want to start Navigator Metadata Server:

 

2019-06-25 08:33:07,341 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)

 

(2)

 

An exception occurs right away:

 

2019-06-25 08:33:07,884 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.util.NoSuchElementException
at com.google.common.collect.Iterators$1.next(Iterators.java:77)
at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325)
at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271)
at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43)
at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99)

 

(3)

 

Cloudera Manager fails the command start with:

 

2019-06-25 08:33:07,885 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Role failed to start due to error null.
2019-06-25 08:33:08,032 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Role 324/mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7 Process cloudera-mgmt-NAVIGATORMETASERVER does not exist - unable to update status

 

CONCLUSION:

 

Based on my review of the stack trace and Cloudera Manager code, the cause of this issue is that TLS/SSL Client Truststore File Location is configured with a path, but Cloudera Manager Server TLS/SSL Client Trust Store Password is empty in your Cloudera Management Service Configuration.

 

The Navigator code will try to pull the values for each and the line where it fails is where it attempts to read the Trust Store password.

 

I removed the password for my truststore and was able to reproduce.

 

FIX:

 

Configure Cloudera Manager Server TLS/SSL Client Trust Store Password with the password for your truststore file.

 

Let us know if that helps!

View in thread

7 replies

lwang
Platinum ⭐️
lwang
Platinum ⭐️
June 20, 2019

Hi @VijayM ,

 

Could you please check if there are any updated logs under /var/run/cloudera-scm-agent/process/xxx-cloudera-mgmt-NAVIGATORMETASERVER/logs where xxx is the latest/highest number. Usually you will find the stdout and stderr log from that folder to give you some advices on why the role can not start.

 

Thanks and hope this helps,

Li

Translate
Li Wang, Technical Solution Manager
VijayM
VijayMAuthor
Silver ⭐️⭐️⭐️
June 24, 2019

@lwang 

 

Below 2 Directories have generated.

 

drwxr-x--x. 3 root root 180 Jun 21 12:34 572-MGMT.NAVIGATOR-test-db-connection
drwxr-x--x. 3 root root 180 Jun 21 12:34 571-MGMT.NAVIGATORMETASERVER-test-db-connection

 

I checked in both directories for logs, Below snap for it. But its not helping much.

 

Directory: 571-MGMT.NAVIGATORMETASERVER-test-db-connection

 

root@abc% pwd
/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection/logs

root@abc% ls -lrt
total 8
-rw-r--r--. 1 root root 3676 Jun 20 12:25 stderr.log
-rw-r--r--. 1 root root 394 Jun 20 12:25 stdout.log

 

root@abc% cat stderr.log
Thu Jun 20 12:25:58 BST 2019
+ date
+ date
Thu Jun 20 12:25:58 BST 2019
++ dirname /usr/lib64/cmf/service/dbconnection/test_db_connection.sh
+ cloudera_config=/usr/lib64/cmf/service/dbconnection
++ cd /usr/lib64/cmf/service/dbconnection/../common
++ pwd
+ cloudera_config=/usr/lib64/cmf/service/common
+ . /usr/lib64/cmf/service/common/cloudera-config.sh
++ export JAVA_HOME=/app/bds/java
++ JAVA_HOME=/app/bds/java
++ export 'JAVA_TOOL_OPTIONS=-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ JAVA_TOOL_OPTIONS='-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ export KRB5_CONFIG=/etc/krb5_bds.conf
++ KRB5_CONFIG=/etc/krb5_bds.conf
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z /app/bds/java ']'
+ verify_java_home
+ '[' -z /app/bds/java ']'
+ echo JAVA_HOME=/app/bds/java
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ JAVA=/app/bds/java/bin/java
+ [[ -z /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /app/bds/java/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf

 

root@abc% cat stdout.log
Thu Jun 20 12:25:58 BST 2019
Current working directory: /run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection
Launching one-off process: /usr/lib64/cmf/service/dbconnection/test_db_connection.sh db.properties
Thu Jun 20 12:25:58 BST 2019
JAVA_HOME=/app/bds/java
[ main] DbCommandExecutor INFO Successfully connected to database.

 

 

 

 

Directory: /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection/logs

 

root@abc% pwd
/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection/logs
root@abc% ls -lrt
total 8
-rw-r--r--. 1 root root 3656 Jun 20 12:25 stderr.log
-rw-r--r--. 1 root root 384 Jun 20 12:25 stdout.log

 

 

root@abc% cat stderr.log
Thu Jun 20 12:25:58 BST 2019
+ date
+ date
Thu Jun 20 12:25:58 BST 2019
++ dirname /usr/lib64/cmf/service/dbconnection/test_db_connection.sh
+ cloudera_config=/usr/lib64/cmf/service/dbconnection
++ cd /usr/lib64/cmf/service/dbconnection/../common
++ pwd
+ cloudera_config=/usr/lib64/cmf/service/common
+ . /usr/lib64/cmf/service/common/cloudera-config.sh
++ export JAVA_HOME=/app/bds/java
++ JAVA_HOME=/app/bds/java
++ export 'JAVA_TOOL_OPTIONS=-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ JAVA_TOOL_OPTIONS='-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ export KRB5_CONFIG=/etc/krb5_bds.conf
++ KRB5_CONFIG=/etc/krb5_bds.conf
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z /app/bds/java ']'
+ verify_java_home
+ '[' -z /app/bds/java ']'
+ echo JAVA_HOME=/app/bds/java
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ JAVA=/app/bds/java/bin/java
+ [[ -z /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /app/bds/java/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf

 

 

root@abc% cat stdout.log
Thu Jun 20 12:25:58 BST 2019
Current working directory: /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection
Launching one-off process: /usr/lib64/cmf/service/dbconnection/test_db_connection.sh db.properties
Thu Jun 20 12:25:58 BST 2019
JAVA_HOME=/app/bds/java
[ main] DbCommandExecutor INFO Successfully connected to database.

Translate
bgooley
Gold ⭐️
bgooley
Gold ⭐️
June 24, 2019

@VijayM ,

 

It appears that you collected the "test-db-connection" command for Navigator which succeeded.

Based on the Cloudera command, the Navigator Metadata server role is on a301-8744-7259.ldn.swissbank.com, so make sure that you were looking in /var/run/cloudera-scm-agent/process for the METASERVER process.  If you don't see any, then it is likely the agent will tell us more of the story of what happened.

 

In general, this is the series of events that happens when you attempt to start a role via Cloudera Manager:

 

  1. Cloudera Manager sends a message to the agent on port 9000 asking it to heartbeat immediately.
  2. The agent sends a heartbeat to Cloudera Manager on port 7182
  3. Cloudera Manager compiles a heartbeat response including any files that need to be used to start and run the process
  4. The agent lays out the files in a new process directory
  5. The agent signals the supervisord process to start the role with a specified shell script
  6. The supervisor executes the startup shell script to start the role

 

Since it appears you do not have any process directories, there must be a problem before the agent even lays down the process configuration files.

If that is the case, the agent log, on the host where the Metadata Server role is configured to run should have some clues.

 

Please share /var/log/cloudera-scm-agent/cloudera-scm-agent.log and let us know the estimated time you tried to start the Metadata Server.

Translate
Terms & ConditionsAccessibility statement