Hi,   I am getting error when I run kudu ksck command   Errors: ================== Corruption: master consensus error: there are master consensus conflicts FAILED Runtime error: ksck discovered errors   All 3 Masters & Tablet server are reported as healthy, Any assistance / pointer in fixing this error.   Thanks    ... View more

Hello, We are seeing a few Impala queries running for more than 3 hours in Impala. Upon checking we found that the query is in exception state however on the Cloudera Impala GUI it still showing it as running. Would like to know if there is a way to kill such type of queries at Impala level. We have implemented Impala ACL and set the Queue Timeout to 5 minutes yet some queries keep running. Our applications connect to Impala via JDBC and we also have some users who use Hue to run queries.   Any assistance/guidance is much appreciated.   ... View more

Hello, We are observing our Impala Catalogue Server’s process frequently gets exited / killed (This role encountered 1 unexpected exit(s) in the previous 5 minute(s).This included 1 exit(s) due to OutOfMemory errors. Critical threshold: any.) I was going through this article (https://community.cloudera.com/t5/Support-Questions/Cloudera-6-2-1-Impala-GC-Overhead-limit-Exceeded/td-p/293225) it seems to be Heap Memory related, would like to know if there is any way / calculations to find how much Heap should be allocated to avoid these issues.   CM / CDH 5.16.2 Java Heap Size of Catalog Server in Bytes = 15Gb   Appreciate any guidance in this regard. ... View more

Hello,   Is there a way to find which kudu table is stored in which tablet server ?   Thanks  Amn ... View more

Hello, I am getting the below error when trying to check logs via CM for Hue Kerberos Ticket Renewer.     CM / CDH - 6.3.3 I checked / /var/log/hue but did not find kt_renewer.log. Appreciate any assistance to fix this.   Thanks Amn ... View more

Hello, I need some assistance / guidance related to Impala Audit & Lineage Logs. In our cluster I  see Impala Audit Logs (enable_audit_event_log) Un-checked, yet on the node level we see logs getting generated. [service1@myserver audit]# ls -ltrh total 215M -rw-r--r-- 1 impala impala 4.2M May 27 14:45 impala_audit_event_log_1.0-1622078304372 -rw-r--r-- 1 impala impala 3.9M May 27 15:34 impala_audit_event_log_1.0-1622083537006 -rw-r--r-- 1 impala impala 4.3M May 27 16:44 impala_audit_event_log_1.0-1622086468134 -rw-r--r-- 1 impala impala 4.3M May 27 17:18 impala_audit_event_log_1.0-1622090691805 -rw-r--r-- 1 impala impala 4.3M May 27 18:26 impala_audit_event_log_1.0-1622092726630 -rw-r--r-- 1 impala impala 4.2M May 27 20:07 impala_audit_event_log_1.0-1622096799288 -rw-r--r-- 1 impala impala 4.0M May 27 21:17 impala_audit_event_log_1.0-1622102850640 -rw-r--r-- 1 impala impala 3.7M May 28 01:36 impala_audit_event_log_1.0-1622107032783 -rw-r--r-- 1 impala impala 2.7M May 28 14:34 impala_audit_event_log_1.0-1622122581046 -rw-r--r-- 1 impala impala 4.7M May 28 18:17 impala_audit_event_log_1.0-1622169268919 -rw-r--r-- 1 impala impala 4.3M May 28 19:34 impala_audit_event_log_1.0-1622182634220 -rw-r--r-- 1 impala impala 4.3M May 28 20:30 impala_audit_event_log_1.0-1622187283341 -rw-r--r-- 1 impala impala 4.3M May 28 20:58 impala_audit_event_log_1.0-1622190632083 -rw-r--r-- 1 impala impala 4.4M May 28 21:25 impala_audit_event_log_1.0-1622192312991   Could anyone please share some light as to why this is happening? I was under the impression that if this is checked only then logs would get captured, kindly clarify CM / CDH – 6.3.3   Thanks  Amn ... View more

Hello, I was trying to take report from HDFS Report and get the below message Upon checking /var/log/cloudera-scm-headlamp and see the following error    021-04-15 18:09:44,934 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:80) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:257) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:325) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:240) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:57) at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:494) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:74) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:221) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:211) at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:236) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:77) at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:72) ... 8 more Caused by: java.io.IOException: Unsupported layout version -64 at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75) at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:113) at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:70) ... 9 more     CM / CDH - 6.3.3 Appreciate any help / guidance in fixing this issue .   Thanks  Amn ... View more

Hello, Need some assistance / guidance on how we can reduce Non-HDFS Space. We see Non-HDFS Space of around 270 used, as we are facing space crunch, we would explore possibilities for getting non-hdfs space reduced. I have cleared all Yarn logs for the applications which were killed/ failed etc (our /data mountpoint houses dfs, yarn, kudu, impala), yet this does not solve our issue. Any assistance / guidance is much appreciated.   Thanks  Amn ... View more

Hello, We are getting alerts for Block Count on one of our data nodes as it has crossed the threshold of 10000. Since HDFS balancer did not fix the issue, the next thing I turned my focus to see if we are hitting small files issue. I was trying to put up a report via terminal script ( hdfs dfs -ls -R /tmp |grep ^- |awk '{if ($5 < 134217728) print $5, $8;}'| head -5 | column –t) but when I compare the result from the script output vs HDFS Report from Cloudera Manager I see a difference in the size of the same file.   Could anyone provide any guidance / assistance on this, or am I doing something wrong.   Thanks Amn ... View more

Hello, I need to increase the query timeout (backend_client_rpc_timeout_ms) from current 5 minutes to 30 minutes. Could anyone guide me from where in Impala Configurations I can do this change. I did some checking but cannot find any related, appreciate any assistance / guidance.   Thanks  Amn ... View more

Hello, We are seeing concerning alert on one of our data node related to File Descriptor (Concerning: Open file descriptors: 16,410. File descriptor limit: 32,768. Percentage in use: 50.08%. Warning threshold: 50.00%.)   Would appreciate any help/ guidance to fix this before it goes out of hand.   [user1@myserver ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1030544 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited   [user1@myserver ~]$ cat /proc/sys/fs/file-max 26161091 [user1@myserver ~]$ cat /proc/sys/fs/file-nr 80400 0 26161091   Thanks  Amn ... View more

Hello, On our data node, we are increasing getting alerts related to Data Node Pause Duration. So far, this is happening on a single data node out of nine data nodes. Following is the error captured from DN logs  2020-10-27 16:20:05,140 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1821ms GC pool 'ParNew' had collection(s): count=1 time=2075ms)   Current Java Heap Size of Data Node in Bytes is at 6GB   CM / CDH – 5.16.x   Any help is appreciated.   Regards Amn ... View more

Hello, How to update DNS part in host name, example from 192.168.0.1.test-1.pl to 192.168.0.1.test.co.pl so that we can open CM GUI via 192.168.0.1.test.co.pl:7180 and other web gui (hue, impala etc) with the new dns name test.co.pl.     Regards Anm ... View more