Hi @Meepoljd    As per the description of the issue, I believe you are hitting ZOOKEEPER-1622 . Cloudera has already released TSB-2022-577 for this. As mentioned in the TSB, the session IDs should be unique but due to this bug, two clients might use the same session ID and hence causing the unexpected session termination. ... View more

Hello @loridigia ,   It seems due to the outage there would be multiple ServerCrashProcedures created for the Regionservers. The Dead region severs with same names are different instances of the Region servers with a different epoch timestamp. As the Hbase Master was also down, it might be possible that it was not able to process the expiration of the Region servers. You might see some Crash procedures waiting to be finished under "Procedures & Locks" section of the Active Hbase Master Web UI. As you have already solved this issue in the past involving zookeeper. I guess you can try this : 1. Stop Hbase 2. Login to zookeeper using #hbase zkcli ( with a valid hbase ticket ) 3. Delete the /hbase-secure znode. rmr /hbase-secure 4. Sideline the entries under HDFS dir. hdfs dfs -mv /hbase/ MasterProcWALs/*  /tmp. ( Not sure if this was done earlier ) 5. Start Hbase ... View more

Hello @KPG1 ,   The time taken to mark a datanode as stale is give by  dfs.namenode.stale.datanode.interval, with a default of 30 seconds. If this is happening with a specific Datanode, you can check if there is any network issues between the Datanode and the Namenode or if the Datanode has any JVM pauses reported by checking the Datanode logs. As a bandaid, you can bump up the above parameter till the underlying problem is solved. ... View more

Hello,   Based on the above test, I guess you are hitting HBASE-21852 which is still unresolved in the upstream. ... View more

@stale ,   It looks like a mismatch in the encryption types in your krb5.conf and the AD is causing this. Do check the below 2 Cloudera articles to see if that helps resolving this issue.   https://my.cloudera.com/knowledge/ERRORquotCaused-by-GSSException-Failure-unspecified-at-GSS-API?id=272836   https://my.cloudera.com/knowledge/ErrorquotCaused-by-Failure-unspecified-at-GSS-API-level?id=273436   ... View more

Hello @Grumash ,   I believe  user=cdp_svc_fc_03 is the spark user which no longer exists. So when you are trying to move the file into the trash folder in the home dir, its failing to create the home dir. You need to create the home dir as the super user (hdfs), then chown it to cdp_svc_fc_03, then it should work. ... View more

Hi @enirys , You will need to add the host entries in the DNS record if freeipa is used to manage the DNS. You can compare the host entries from the other working Datanode in freeipa.  Every node in a Data Lake, Data Hub, and a CDP data service should be configured to look up the FreeIPA DNS service for name resolution within the cluster. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/adding-host-entry ... View more

Hello, Try to install the agent package on this host and let me know if that solves the issue. Make a copy of your /etc/cloudera-scm-agent/config.ini. Uninstall the cloudera-manager-agent package Install the cloudera-manager-agent package Copy the /etc/cloudera-scm-agent/config.ini back Start the cloudera-scm-agent service ... View more

Cell TTL is something that is defined at insertion time. So for an already existing data, you can perform a put of the same cell, which then will have the exact cell value on a new Hfile with the new TTL for the cell. ... View more

As HDFS-3743 still remains unresolved. I don't see any feature in the CM to copy the edits from the live journal node to the newly added journal node. Deploying the role using CM will add the role but the Active Namenode will report that its "out of sync" with this new journal node. So before starting the new journal node, copy the edits directory from one of the existing JournalNodes to the new one.   1 - Stop the New JN. 2 - Copy all the contents under /current of Journalnode folder from a working Journalnode to the new Journal node. 3 - Add the right permission to the newly copied folders with chown and chmod. 4 - Delete the edit_inprogress file (inside the new Journal node) such that the new Journalnode will start synchronising this file with the older Journal nodes.   ... View more

Ensure that at OS level the Namenode disk is mounted with read/write options ... View more

There is no direct method or property to disable tombstone marker. Major compacting the mob CF is the only way to go. There is  a mob cleaner running on master that removes unreferenced mob files: The period of the cleaner chore can be configured by setting hbase.master.mob.cleaner.period to a positive integer number of seconds. It defaults to running daily. You should not need to tune it unless you have a very aggressive TTL or a very high rate of MOB updates with a correspondingly high rate of non-MOB compactions. So you can check and set the above parameter value and  can manually trigger mob compaction by specifying the cf that includes mob data in hbase shell.   NOTE : MOBs are mostly for writing large cells that will rarely be updated or deleted but could have TTL configured for the cell or CF. ... View more