Can you share here any of the datanode logs, we can try to find what the problem in reaching out to the secondary namenode could be. ... View more

The issue seems to be in your secondary namenode: 2025-07-07 11:56:37,798 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Log not rolled. Name node is in safe mode. The reported blocks 0 has reached the threshold 0.9990 of total blocks 0. The number of live datanodes 0 needs an additional 1 live datanodes to reach the minimum number 1. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:dmidlkprdls01.svr.luc.edu It looks like the namenode can't communicate with your datanodes, hence it can't come out of safemode and crashes. Maybe there's some network problem that don't allow the communication between those 2 roles? Can you ping the secondary namenode from your datanodes and vice-versa? Are the required ports open on secondary namenode and datanodes? ... View more

I meant the namenode process logs. If you didn't customize the location, it should be under /var/log/hadood-hdfs, then you will see a bunch of logs.  Get the latest one that says NAMENODE (it's in caps) and if possible share it here.  Get them from both namenode servers please. ... View more

The logs from both namenode servers, to investigate why the checkpointing process is failing ... View more

Hello @jkoral  The log snippet you posted is not enough for us to identify the problem. Those messages about: Not enough replicas was chosen They are mostly harmless, and although annoying, don't pose a threat to the process. Is it possible for you to share the logs from both namenodes to check? ... View more

Hello @rizalt  Thanks for posting your questions on Cloudera Community forum! If I understood well, your space usage is high although it seems you don't have any data in hdfs at all. For confirmation, could you please run this command against hdfs (make sure to have a kerberos ticket if the cluster is kerberized: hdfs dfs -du -h / or show me from the browse directory page, the root folder. Finally, to answer your immediate question, please don't remove any data from  /hadoop/hdfs/data.  It is preferred to remove things using the proper tools.  We will walk you through it once we get the above information. Regards, JR ... View more

Hello @sayebogbon , Based on the error in the log you shared: opened_url = urlopen_with_retry_on_authentication_errors And the klist output showing this: Valid starting Expires 10/11/24 23:43:47 11/11/24 09:43:47 Looks like you need to regenerate the kerberos credentials for this host. To do so, please stop all services on this host. Then go to CM > Administration > Security > Kerberos credentials. In the search bar, type the hostname and select all the principals that appear, then click the regenerate selected button. If there are no problems, new credentials should be generated. Restart your services and let us know if that helps. ... View more

Hello Olivier, The error you got "org.apache.kudu.client.NonRecoverableException: Unauthorized action" happens when the account you use to take the backup don't have the proper permissions to access the kudu tables. Do you happen to use ranger to authorize access on kudu?  If so, then maybe the account you are using is not authorized on ranger.  It could be as well that you are running it with the wrong kerberos principal? Regards, Jason R ... View more