Cloudera Navigator Metadata Server not starting

[#1]

Hello team,

Cloudera Navigator Metadata Server failed to start, no log file getting updated in /var/log/cloudera-scm-navigator for metaddata server.

Only Cloudera server shows below errors.

CDH version: 5.16.1

Kindly suggest?

Error:

2019-06-20 13:49:21,099 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)
2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=2, name=mgmt} Role: DbRole{id=324, name=mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7, hostName=a301-8744-7259.ldn.swissbank.com}
2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-06-20 13:49:21,127 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Start a role
2019-06-20 13:49:21,389 WARN CommandPusher:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.util.NoSuchElementException
at com.google.common.collect.Iterators$1.next(Iterators.java:77)
at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325)
at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271)
at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43)
at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99)
at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.getEnvironmentForRole(BaseMgmtRoleHandler.java:123)
at com.cloudera.cmf.service.AbstractDaemonRoleHandler.getEnvironment(AbstractDaemonRoleHandler.java:347)
at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.makeProcess(BaseMgmtRoleHandler.java:73)
at com.cloudera.cmf.service.GenericBringUpRoleCommand$RoleBringUpCmdWork.doWork(GenericBringUpRoleCommand.java:203)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:943)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:898)
at com.cloudera.cmf.command.flow.CmdWorkCtx.execRoleCmd(CmdWorkCtx.java:145)
at com.cloudera.cmf.command.flow.work.ExecRoleCmdWork.doWork(ExecRoleCmdWork.java:58)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.work.ScatterCmdWork.doWork(ScatterCmdWork.java:86)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845)
at com.cloudera.cmf.command.flow.CmdWorkCtx.execSvcCmd(CmdWorkCtx.java:159)
at com.cloudera.cmf.command.flow.work.ExecSvcCmdWork.doWork(ExecSvcCmdWork.java:91)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.SeqCmdWork.update(SeqCmdWork.java:180)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:180)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.flow.SeqFlowCmd.update(SeqFlowCmd.java:249)
at com.cloudera.cmf.command.CmdWorkCommand.update(CmdWorkCommand.java:144)
at com.cloudera.server.cmf.CommandPusherThread.handleCommandInternal(CommandPusherThread.java:456)
at com.cloudera.server.cmf.CommandPusherThread.handleCommandBatched(CommandPusherThread.java:270)
at com.cloudera.server.cmf.CommandPusherThread.innerLoop(CommandPusherThread.java:189)
at com.cloudera.server.cmf.CommandPusherThread.run(CommandPusherThread.java:157)
2019-06-20 13:49:21,389 ERROR CommandPusher:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7187 Role failed to start due to error null.

[#2]

Hi @VijayM ,

Could you please check if there are any updated logs under /var/run/cloudera-scm-agent/process/xxx-cloudera-mgmt-NAVIGATORMETASERVER/logs where xxx is the latest/highest number. Usually you will find the stdout and stderr log from that folder to give you some advices on why the role can not start.

Thanks and hope this helps,

Li

[#3]

@lwang 

Below 2 Directories have generated.

drwxr-x--x. 3 root root 180 Jun 21 12:34 572-MGMT.NAVIGATOR-test-db-connection
drwxr-x--x. 3 root root 180 Jun 21 12:34 571-MGMT.NAVIGATORMETASERVER-test-db-connection

I checked in both directories for logs, Below snap for it. But its not helping much.

Directory: 571-MGMT.NAVIGATORMETASERVER-test-db-connection

root@abc% pwd
/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection/logs

root@abc% ls -lrt
total 8
-rw-r--r--. 1 root root 3676 Jun 20 12:25 stderr.log
-rw-r--r--. 1 root root 394 Jun 20 12:25 stdout.log

root@abc% cat stderr.log
Thu Jun 20 12:25:58 BST 2019
+ date
+ date
Thu Jun 20 12:25:58 BST 2019
++ dirname /usr/lib64/cmf/service/dbconnection/test_db_connection.sh
+ cloudera_config=/usr/lib64/cmf/service/dbconnection
++ cd /usr/lib64/cmf/service/dbconnection/../common
++ pwd
+ cloudera_config=/usr/lib64/cmf/service/common
+ . /usr/lib64/cmf/service/common/cloudera-config.sh
++ export JAVA_HOME=/app/bds/java
++ JAVA_HOME=/app/bds/java
++ export 'JAVA_TOOL_OPTIONS=-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ JAVA_TOOL_OPTIONS='-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ export KRB5_CONFIG=/etc/krb5_bds.conf
++ KRB5_CONFIG=/etc/krb5_bds.conf
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z /app/bds/java ']'
+ verify_java_home
+ '[' -z /app/bds/java ']'
+ echo JAVA_HOME=/app/bds/java
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ JAVA=/app/bds/java/bin/java
+ [[ -z /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /app/bds/java/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf

root@abc% cat stdout.log
Thu Jun 20 12:25:58 BST 2019
Current working directory: /run/cloudera-scm-agent/process/571-MGMT.NAVIGATORMETASERVER-test-db-connection
Launching one-off process: /usr/lib64/cmf/service/dbconnection/test_db_connection.sh db.properties
Thu Jun 20 12:25:58 BST 2019
JAVA_HOME=/app/bds/java
[ main] DbCommandExecutor INFO Successfully connected to database.

Directory: /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection/logs

root@abc% pwd
/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection/logs
root@abc% ls -lrt
total 8
-rw-r--r--. 1 root root 3656 Jun 20 12:25 stderr.log
-rw-r--r--. 1 root root 384 Jun 20 12:25 stdout.log

root@abc% cat stderr.log
Thu Jun 20 12:25:58 BST 2019
+ date
+ date
Thu Jun 20 12:25:58 BST 2019
++ dirname /usr/lib64/cmf/service/dbconnection/test_db_connection.sh
+ cloudera_config=/usr/lib64/cmf/service/dbconnection
++ cd /usr/lib64/cmf/service/dbconnection/../common
++ pwd
+ cloudera_config=/usr/lib64/cmf/service/common
+ . /usr/lib64/cmf/service/common/cloudera-config.sh
++ export JAVA_HOME=/app/bds/java
++ JAVA_HOME=/app/bds/java
++ export 'JAVA_TOOL_OPTIONS=-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ JAVA_TOOL_OPTIONS='-Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf'
++ export KRB5_CONFIG=/etc/krb5_bds.conf
++ KRB5_CONFIG=/etc/krb5_bds.conf
++ set -x
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')
+ local JAVA6_HOME_CANDIDATES
+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')
+ local OPENJAVA6_HOME_CANDIDATES
+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')
+ local JAVA7_HOME_CANDIDATES
+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')
+ local OPENJAVA7_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})
+ '[' -z /app/bds/java ']'
+ verify_java_home
+ '[' -z /app/bds/java ']'
+ echo JAVA_HOME=/app/bds/java
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ JAVA=/app/bds/java/bin/java
+ [[ -z /usr/share/cmf ]]
+ MGMT_CLASSPATH='/usr/share/cmf/lib/*'
+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar
+ MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
++ pwd
+ MGMT_CLASSPATH='/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*'
+ MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true '
+ exec /app/bds/java/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection:/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties
Picked up JAVA_TOOL_OPTIONS: -Doracle.jdbc.thinLogonCapability=o3 -Djava.security.krb5.conf=/etc/krb5_bds.conf

root@abc% cat stdout.log
Thu Jun 20 12:25:58 BST 2019
Current working directory: /run/cloudera-scm-agent/process/572-MGMT.NAVIGATOR-test-db-connection
Launching one-off process: /usr/lib64/cmf/service/dbconnection/test_db_connection.sh db.properties
Thu Jun 20 12:25:58 BST 2019
JAVA_HOME=/app/bds/java
[ main] DbCommandExecutor INFO Successfully connected to database.

[#4]

@VijayM ,

It appears that you collected the "test-db-connection" command for Navigator which succeeded.

Based on the Cloudera command, the Navigator Metadata server role is on a301-8744-7259.ldn.swissbank.com, so make sure that you were looking in /var/run/cloudera-scm-agent/process for the METASERVER process.  If you don't see any, then it is likely the agent will tell us more of the story of what happened.

In general, this is the series of events that happens when you attempt to start a role via Cloudera Manager:

1. Cloudera Manager sends a message to the agent on port 9000 asking it to heartbeat immediately.
2. The agent sends a heartbeat to Cloudera Manager on port 7182
3. Cloudera Manager compiles a heartbeat response including any files that need to be used to start and run the process
4. The agent lays out the files in a new process directory
5. The agent signals the supervisord process to start the role with a specified shell script
6. The supervisor executes the startup shell script to start the role

Since it appears you do not have any process directories, there must be a problem before the agent even lays down the process configuration files.

If that is the case, the agent log, on the host where the Metadata Server role is configured to run should have some clues.

Please share /var/log/cloudera-scm-agent/cloudera-scm-agent.log and let us know the estimated time you tried to start the Metadata Server.

[#5]

@bgooley 

I tried to restart and there is no process directory getscreated for Navigator Metastore.

I had tail the cloudera agent log for server where Navigator Metadata is running. Cloudera agent throws below logs when i restarted Navigator Metastore.

Cloudera Agent log:

[25/Jun/2019 08:30:32 +0000] 2813189 CP Server Thread-5 _cplogging INFO 10.26.132.165 - - [25/Jun/2019:08:30:32] "GET /heartbeat HTTP/1.1" 200 2 "" "NING/1.0"
[25/Jun/2019 08:30:34 +0000] 2813189 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for conf.cloudera.spark2_on_yarn with count 0 pipelines names [].
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin __init__ INFO Read metadata config for role GATEWAY. Metadata server url: https://a301-7781-0339.ldn.swissbank.com:7187
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_plugin INFO Piplines updated for Metadata Plugin: []
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for conf.cloudera.spark2_on_yarn with count 0 pipelines names [].
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_thread INFO Log entry poll interval: 5
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_thread INFO Navigator server url: https://a301-7781-0339.ldn.swissbank.com:7187
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin inotify_event_processor INFO Setting watch for directories: /var/log/hive/lineage
[25/Jun/2019 08:30:37 +0000] 2813189 Metadata-Plugin navigator_logs INFO 9 navigator log files with prefix hive-HIVESERVER2-220e59e0745b944070fe360a2cc717c7 for role hive-HIVESERVER2-220e59e0745b944070fe360a2cc717c7 exist in navigator log directory /var/log/hive/lineage

Cloudera Sever throws below logs.

2019-06-25 08:33:07,341 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)
2019-06-25 08:33:07,372 INFO 439059796@scm-web-6908:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=2, name=mgmt} Role: DbRole{id=324, name=mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7, hostName=a301-8744-7259.ldn.swissbank.com}
2019-06-25 08:33:07,372 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-06-25 08:33:07,372 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Start a role
2019-06-25 08:33:07,884 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.util.NoSuchElementException
at com.google.common.collect.Iterators$1.next(Iterators.java:77)
at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325)
at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271)
at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43)
at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99)
at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.getEnvironmentForRole(BaseMgmtRoleHandler.java:123)
at com.cloudera.cmf.service.AbstractDaemonRoleHandler.getEnvironment(AbstractDaemonRoleHandler.java:347)
at com.cloudera.cmf.service.mgmt.BaseMgmtRoleHandler.makeProcess(BaseMgmtRoleHandler.java:73)
at com.cloudera.cmf.service.GenericBringUpRoleCommand$RoleBringUpCmdWork.doWork(GenericBringUpRoleCommand.java:203)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:943)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:898)
at com.cloudera.cmf.command.flow.CmdWorkCtx.execRoleCmd(CmdWorkCtx.java:145)
at com.cloudera.cmf.command.flow.work.ExecRoleCmdWork.doWork(ExecRoleCmdWork.java:58)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107)
at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177)
at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117)
at com.cloudera.cmf.service.AbstractRestartCommands$GenericRestartRoleCommand.executeImpl(AbstractRestartCommands.java:187)
at com.cloudera.cmf.service.AbstractRoleCommand.execute(AbstractRoleCommand.java:64)
at com.cloudera.cmf.service.AbstractRoleCommand.execute(AbstractRoleCommand.java:24)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommandHelper(ServiceHandlerRegistry.java:943)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:898)
at com.cloudera.cmf.service.ServiceHandlerRegistry.executeRoleCommand(ServiceHandlerRegistry.java:893)
at com.cloudera.server.cmf.components.OperationsManagerImpl.executeRoleCommand(OperationsManagerImpl.java:1811)
at com.cloudera.server.web.cmf.InstancesController.internalRoleInstancesDispatch(InstancesController.java:469)
at com.cloudera.server.web.cmf.InstancesController.access$100(InstancesController.java:76)
at com.cloudera.server.web.cmf.InstancesController$3.apply(InstancesController.java:437)
at com.cloudera.server.web.cmf.InstancesController$3.apply(InstancesController.java:434)
at com.cloudera.server.web.cmf.InvocationUtil$1.call(InvocationUtil.java:36)
at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32)
at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210)
at com.cloudera.server.common.RetryUtils.executeWithRetry(RetryUtils.java:108)
at com.cloudera.server.web.cmf.InvocationUtil.invoke(InvocationUtil.java:32)
at com.cloudera.server.web.cmf.InstancesController.roleInstancesDispatch(InstancesController.java:433)
at com.cloudera.server.web.cmf.InstancesController$$FastClassByCGLIB$$17ed6885.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.cloudera.server.web.cmf.InstancesController$$EnhancerByCGLIB$$16b0d4df.roleInstancesDispatch(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2019-06-25 08:33:07,885 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Role failed to start due to error null.
2019-06-25 08:33:08,032 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Role 324/mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7 Process cloudera-mgmt-NAVIGATORMETASERVER does not exist - unable to update status
2019-06-25 08:33:08,032 INFO 439059796@scm-web-6908:com.cloudera.cmf.model.DbCommand: Command 7269(Start) has completed. finalstate:FINISHED, success:false, msg:Failed to start role.
2019-06-25 08:33:08,032 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Failed to start role.
2019-06-25 08:33:08,032 INFO 439059796@scm-web-6908:com.cloudera.cmf.model.DbCommand: Command 7268(Restart) has completed. finalstate:FINISHED, success:false, msg:Failed to start role.
2019-06-25 08:33:08,447 INFO 439059796@scm-web-6908:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: com.cloudera.cmf.model.DbConfig.valueForDb (#814) has changed]}.
2019-06-25 08:33:08,448 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-06-25 08:33:08,448 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Configuration Staleness Check
2019-06-25 08:33:09,456 INFO avro-servlet-hb-processor-2:com.cloudera.server.common.AgentAvroServlet: (23 skipped) AgentAvroServlet: heartbeat processing stats: average=1136ms, min=719ms, max=2947ms.
2019-06-25 08:33:10,359 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Running staleness check with FULL_CHECK for 44/74 roles.
2019-06-25 08:33:10,432 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Total staleness check tasks: 77

- Vijay M

[#6]

@bgooley 

Any suggestion?

- Vijay M

[#7 · Accepted solution]

@VijayM,

I was out on PTO for a couple weeks so I just saw your comment.  This looks like a pretty unusual issue.

Based on the fact that the agent never seems to know that it should start the role, this is a problem that is occurring in Cloudera Manager only (before the command is to be started by the agent).

The stack trace and error messages show us something interesting:

(1)

CM sees that you want to start Navigator Metadata Server:

2019-06-25 08:33:07,341 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259)

(2)

An exception occurs right away:

2019-06-25 08:33:07,884 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution
java.util.NoSuchElementException
at com.google.common.collect.Iterators$1.next(Iterators.java:77)
at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325)
at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271)
at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43)
at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99)

(3)

Cloudera Manager fails the command start with:

2019-06-25 08:33:07,885 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Role failed to start due to error null.
2019-06-25 08:33:08,032 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Role 324/mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7 Process cloudera-mgmt-NAVIGATORMETASERVER does not exist - unable to update status

CONCLUSION:

Based on my review of the stack trace and Cloudera Manager code, the cause of this issue is that TLS/SSL Client Truststore File Location is configured with a path, but Cloudera Manager Server TLS/SSL Client Trust Store Password is empty in your Cloudera Management Service Configuration.

The Navigator code will try to pull the values for each and the line where it fails is where it attempts to read the Trust Store password.

I removed the password for my truststore and was able to reproduce.

FIX:

Configure Cloudera Manager Server TLS/SSL Client Trust Store Password with the password for your truststore file.

Let us know if that helps!

[#8]

@bgooley 

Cloudera Navigator Metastore needs TLS password to be set in CM UI otherwise it wont start.

Thanks for the help to fix the issue.

- Vijay M