Hi @kkanchu  Can you please accept the answer if the above steps helps to change the log level of the HBase daemon on the fly without any restart. ... View more

Hi @Hilljairus  Once the Hbase is started we need to confirm weather the active and standby master states showing, until unless the system tables (meta, namespace, and ACLs) are assigned the active and standby states of the master will not get.   Based on the shared exception the namespace table is not assigned, please check the master logs and make sure the namespace table is assigned or not, if it is not assigned please assign the namespace table to fix the issue. ... View more

Hi @rahuledavalath  We can use the HBase replication process for the data copy between the two CDP clusters, like production to DR. We have multiple open documents for the setting Hbase replication please follow the docs for setting the replication between two cluster to synchronise the data. https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/hbase-backup-dr/topics/hbase-replication.html https://blog.cloudera.com/apache-hbase-replication-overview/     ... View more

@arunpoy  If we are using the CDH/CDP both timeout parameters(hbase.rpc.timeout & hbase.client.scanner.timeout.period) need to be added in both server-side and client-side in the below paths from the HBase configuration. HBase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml HBase Client Advanced Configuration Snippet (Safety Valve) for hbase-site.xml   Bothe the time-out parameters need to be added on the server and client-side and RPC(hbase.rpc.timeout) time-out needs to be set a bit higher than the client scanner time-out (hbase.client.scanner.timeout.period). ... View more

We can use the HBase master UI for enabling and disabling the TRACE/DEBUG/INFO without restarting any service.   ➤ To enable the TRACE/DEBUG logging please follow the below steps from the Hbase master UI for the master server. Step 1: Hbase Master UI -> click on log level-> add the property(org.apache.hadoop.ipc) on the second log box-> enter the DBUG in Level box-> click on Set log level radio button.   ➤ To enable the TRACE/DEBUG logging please follow the below steps from the Hbase master UI for the specific region server. Step 1: Hbase Master UI -> select specific region server-> click on log level-> add the property(org.apache.hadoop.ipc) on the second log box-> enter the DBUG in Level box-> click on Set log level radio button.   ➤ Once the completion of the above steps, just verifies whether the changes are effected or not with the below steps. Step 2: Hbase Master UI -> select specific region server/Master server-> click on log level-> add the property(org.apache.hadoop.ipc) on the fist log box-> click on the Get log level radio button, and verify the log level.   Or    ➤ hadoop daemonlog -setlevel <hbase regionserver host>:<port> org.apache.hadoop.hbase TRACE/DEBUG/INFO ... View more

Hi, @kras Thank you for writing back with your observation. Can you please check the below details as well? 1) When the Region Server JVM reports High CPU, Open "top" Command for the Region Server PID, 2) Use "Shift H" to open the Thread View of the PID. This would show the Threads within the Region Server JVM with CPU Usage, 3) Monitor the Thread View & Identify the Thread hitting the Max CPU Usage, 4) Take Thread Dump | JStack of Region Server PID & Compare the Thread with the "top" Thread View consuming the Highest CPU. 5) Check the CUP usage of the other services that are hosted on the Region Server host.   The above Process would allow you to identify the Thread contributing towards the CPU Usage. Compare the same with other Region Server & your Team can make a Conclusive Call to identify the reasoning for CPU Utilization. Howsoever Logs are reviewed, Narrowing the Focus of JVM review would assist in identifying the Cause. Review shared Link for additional reference.   Ref:  https://www.infoworld.com/article/3336222/java-challengers-6-thread-behavior-in-the-jvm.html  https://blogs.manageengine.com/application-performance-2/appmanager/2011/02/09/identify-java-code-co...  https://blog.jamesdbloom.com/JVMInternals.html   Thanks & Regards, Prathap Kumar. ... View more

1) check the region server logs is there “responseTooSlow” or “operationTooSlow” or any other WARN/ERROR messages. please provide log snippets. 2) if we are seeing the "responseTooSlow" on the region servers, please check the data node logs for the underlying issue from the data node logs. 3)In the data node logs please check we have below ERROR/WARN in the data node logs are not. Slow BlockReceiver write data to disk cost - This indicates that there was a delay in writing the block to the OS cache or disk. Slow BlockReceiver write packet to mirror took - This indicates that there was a delay in writing the block across the network Slow flushOrSync took/Slow manageWriterOsCache took - This indicates that there was a delay in writing the block to the OS cache or disk 4) If we have the above ERROR/WARN we need to check the infra team and OS vendor team to fix the underlying hardware issues to overcome issue.   There are many reasons this could happen including OS/Kernel bugs (update your system), swap, transparent huge pages, pauses by a hypervisor for the High CPU usage issues and you need to figure out which is causing the issue and need to fix it to overcome the issue. ... View more

There are many reasons this could happen including OS/Kernel bugs (update your system), swap, transparent huge pages, pauses by a hypervisor for the High latency issues.   1) As we are seeing the "responseTooSlow" on the region servers, please check the data node logs for the underlying issue from the data node logs. 2) In the data node logs please check we have below ERROR/WARN in the data node logs are not. Slow BlockReceiver write data to disk cost - This indicates that there was a delay in writing the block to the OS cache or disk. Slow BlockReceiver write packet to mirror took - This indicates that there was a delay in writing the block across the network Slow flushOrSync took/Slow manageWriterOsCache took - This indicates that there was a delay in writing the block to the OS cache or disk 3) If we have the above ERROR/WARN we need to check the infra team and OS vendor team to fix the underlying hardware issues to overcome issue.   ... View more