Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎07-05-2019

CDH-6.1.0 failed to get out safe-mode on a namenode

[ Edited ]

Hello expert,



After crashing the cloudera master server (VM centos), I replace it by a back up. There is no data on the system actually, so I can play as I wish.

In the log "hadoop-cmf-hdfs-NAMENODE-lnxsrv-cloudera6-m.mnh.fr.log.out"

--------------------------------------

2019-07-05 15:21:35,116 INFO org.apache.hadoop.ipc.Server: IPC Server handler 22 on 8020, call Call#947 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 172.23.104.90:44933: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2019_07_05-15_21_35. Name node is in safe mode.

The reported blocks 1402 needs additional 171 blocks to reach the threshold 0,9990 of total blocks 1575.

The number of live datanodes 5 has reached the minimum number 1. Name node detected blocks with generation stamps in future. This means that Name node metadata is inconsistent. This can happen if Name node metadata files have been manually replaced. Exiting safe mode will cause loss of 252825 byte(s). Please restart name node with right metadata or use "hdfs dfsadmin -safemode forceExit" if you are certain that the NameNode was started with the correct FsImage and edit logs. If you encountered this during a rollback, it is safe to exit with -safemode forceExit. NamenodeHostName:lnxsrv-cloudera6-m.mnh.fr

2019-07-05 15:21:40,118 WARN org.apache.hadoop.security.LdapGroupsMapping: Failed to get groups for user hdfs (retry=0) by javax.naming.CommunicationException: dc16-m.mnh.fr:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

2019-07-05 15:21:40,124 WARN org.apache.hadoop.security.LdapGroupsMapping: Failed to get groups for user hdfs (retry=1) by javax.naming.CommunicationException: dc16-m.mnh.fr:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

2019-07-05 15:21:40,131 WARN org.apache.hadoop.security.LdapGroupsMapping: Failed to get groups for user hdfs (retry=2) by javax.naming.CommunicationException: dc16-m.mnh.fr:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target]

-----------------------------------------

The command "hdfs dfsadmin -safemode forceExit" didn't work, with

19/07/05 15:24:38 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] safemode: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "lnxsrv-cloudera6-m.mnh.fr/172.23.104.90"; destination host is: "lnxsrv-cloudera6-m.mnh.fr":8020;

------------------------------------

I regenerated the kerberos id, and it's the same.

 

So , if someone meet this problem and find the answer.

 

Regards

Alain