Hello @loridigia ,   It seems due to the outage there would be multiple ServerCrashProcedures created for the Regionservers. The Dead region severs with same names are different instances of the Region servers with a different epoch timestamp. As the Hbase Master was also down, it might be possible that it was not able to process the expiration of the Region servers. You might see some Crash procedures waiting to be finished under "Procedures & Locks" section of the Active Hbase Master Web UI. As you have already solved this issue in the past involving zookeeper. I guess you can try this : 1. Stop Hbase 2. Login to zookeeper using #hbase zkcli ( with a valid hbase ticket ) 3. Delete the /hbase-secure znode. rmr /hbase-secure 4. Sideline the entries under HDFS dir. hdfs dfs -mv /hbase/MasterProcWALs/*  /tmp. ( Not sure if this was done earlier ) 5. Start Hbase ... View more

Hello @KPG1 ,   The time taken to mark a datanode as stale is give by dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved. ... View more

Hello,   Based on the above test, I guess you are hitting HBASE-21852 which is still unresolved in the upstream. ... View more

@stale ,   It looks like a mismatch in the encryption types in your krb5.conf and the AD is causing this. Do check the below 2 Cloudera articles to see if that helps resolving this issue.   https://my.cloudera.com/knowledge/ERRORquotCaused-by-GSSException-Failure-unspecified-at-GSS-API?id=272836   https://my.cloudera.com/knowledge/ErrorquotCaused-by-Failure-unspecified-at-GSS-API-level?id=273436   ... View more

Hello @Grumash ,   I believe user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work. ... View more

Hi @enirys , You will need to add the host entries in the DNS record if freeipa is used to manage the DNS. You can compare the host entries from the other working Datanode in freeipa. Every node in a Data Lake, Data Hub, and a CDP data service should be configured to look up the FreeIPA DNS service for name resolution within the cluster. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/adding-host-entry ... View more