Member since
02-18-2019
70
Posts
0
Kudos Received
0
Solutions
01-18-2022
01:16 AM
@Tiger123 Appericate your assistacne, I see the below in the logs, 10:04:02.026 PM INFO cc:494 T 00000000000000000000000000000000 P c8d59ba15dbb4578900f597bb48bd9e0 [term 68 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865af50ae13e4cfabe5719b865d6716a" member_type: VOTER last_known_addr { host: "master1.com" port: 7051 } } peers { permanent_uuid: "c8d59ba15dbb4578900f597bb48ce9e0" member_type: VOTER last_known_addr { host: "master2.com" port: 7051 } } peers { permanent_uuid: "8be74245ecaf4b5baf18b24dbc730922" member_type: VOTER last_known_addr { host: "master3.com" port: 7051 } }
10:04:02.026 PM INFO cc:296
T 00000000000000000000000000000000 P c8d59ba15dbb4578900f597bb48bd9e0 [CANDIDATE]: Term 69 pre-election: Requested pre-vote from peers 865af50ae13e4cfabe5719b865d6716a (master1.com:7051), 8be74245ecaf4b5baf18b24dbc730922 (master3.com:7051)
10:04:02.028 PM INFO cc:310
T 00000000000000000000000000000000 P c8d59ba15dbb4578900f597bb48bd9e0 [CANDIDATE]: Term 69 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8d59ba15dbb4578900f597bb48bd9e0; no voters: 865af50ae13e4cfagh5719b865d6716a, 8be74245ecaf4b5baf18b24dbc730922
10:04:02.028 PM INFO cc:2592
T 00000000000000000000000000000000 P c8d59ba15dbb4578900f597bb48bd9e0 [term 68 FOLLOWER]: Leader pre-election lost for term 69. Reason: could not achieve majority Thanks
... View more
01-17-2022
10:43 PM
@Tiger123 Cannot attach ksck report due to company infosec policy. Regarding consensus matrix is the below you are looking for ?? All reported replicas are:
A = 865af50ae13e4cfagh5719b865d6716a
B = c8d59ba15dbb4578900f597bb48bd9e0
C = 8be74245ecaf4b5baf18b24dbc7318ea
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
A | A* B C | 68 | -1 | Yes
B | A B C | 68 | -1 | Yes
C | A* B C | 68 | -1 | Yes
... View more
01-17-2022
02:27 AM
Hi, I am getting error when I run kudu ksck command Errors:
==================
Corruption: master consensus error: there are master consensus conflicts
FAILED
Runtime error: ksck discovered errors All 3 Masters & Tablet server are reported as healthy, Any assistance / pointer in fixing this error. Thanks
... View more
Labels:
10-13-2021
09:59 PM
Hello, Whenever we have maintenance activity of our DB or DB Hosts, we have to bring our cluster down, I want to avoid this and was wondering if there is a way by which we could provide an Oracle VIP / Scan IP so that if one instance of the database is down Hadoop applications (Hive, Hue, Oozie, CM, etc which use DB) could connect to another instance, thus avoiding downtime. Would like some guidance/information on this. CM /CDH 6.3.3 Thanks
... View more
10-12-2021
12:47 AM
Hi @COE I am currently killing the query from Impala GUI, however, this becomes overkill when there are many queries in a state of exception. Was wondering if there could be a faster way to select all of such queries and kill them in one go.
... View more
10-05-2021
09:11 PM
Hello, We are seeing a few Impala queries running for more than 3 hours in Impala. Upon checking we found that the query is in exception state however on the Cloudera Impala GUI it still showing it as running. Would like to know if there is a way to kill such type of queries at Impala level. We have implemented Impala ACL and set the Queue Timeout to 5 minutes yet some queries keep running. Our applications connect to Impala via JDBC and we also have some users who use Hue to run queries. Any assistance/guidance is much appreciated.
... View more
Labels:
08-19-2021
12:35 AM
Hello, We are observing our Impala Catalogue Server’s process frequently gets exited / killed (This role encountered 1 unexpected exit(s) in the previous 5 minute(s).This included 1 exit(s) due to OutOfMemory errors. Critical threshold: any.) I was going through this article (https://community.cloudera.com/t5/Support-Questions/Cloudera-6-2-1-Impala-GC-Overhead-limit-Exceeded/td-p/293225) it seems to be Heap Memory related, would like to know if there is any way / calculations to find how much Heap should be allocated to avoid these issues. CM / CDH 5.16.2 Java Heap Size of Catalog Server in Bytes = 15Gb Appreciate any guidance in this regard.
... View more
Labels:
07-15-2021
01:49 AM
Hello, Is there a way to find which kudu table is stored in which tablet server ? Thanks Amn
... View more
Labels:
06-16-2021
12:22 AM
Hi @salimhussain Enable Impala Audit Event Generation property was already unchecked, for patching activity we had to bring down the cluster and start it back, so I hope that this counts as a restart of Impala too? If it does, then we still have logs getting generated. Thanks Amn
... View more
06-05-2021
02:38 AM
Hello @Sean464 Hue is already installed kt renewer service is also running fine/green. We noticed this post our upgrade from 5.16.x to 6.3.x So can I follow the steps mentioned by @nthomas? Thanks Amn
... View more
06-01-2021
08:47 PM
Hello, I am getting the below error when trying to check logs via CM for Hue Kerberos Ticket Renewer. CM / CDH - 6.3.3 I checked / /var/log/hue but did not find kt_renewer.log. Appreciate any assistance to fix this. Thanks Amn
... View more
Labels:
05-31-2021
12:55 AM
Hello, I need some assistance / guidance related to Impala Audit & Lineage Logs. In our cluster I see Impala Audit Logs (enable_audit_event_log) Un-checked, yet on the node level we see logs getting generated. [service1@myserver audit]# ls -ltrh
total 215M
-rw-r--r-- 1 impala impala 4.2M May 27 14:45 impala_audit_event_log_1.0-1622078304372
-rw-r--r-- 1 impala impala 3.9M May 27 15:34 impala_audit_event_log_1.0-1622083537006
-rw-r--r-- 1 impala impala 4.3M May 27 16:44 impala_audit_event_log_1.0-1622086468134
-rw-r--r-- 1 impala impala 4.3M May 27 17:18 impala_audit_event_log_1.0-1622090691805
-rw-r--r-- 1 impala impala 4.3M May 27 18:26 impala_audit_event_log_1.0-1622092726630
-rw-r--r-- 1 impala impala 4.2M May 27 20:07 impala_audit_event_log_1.0-1622096799288
-rw-r--r-- 1 impala impala 4.0M May 27 21:17 impala_audit_event_log_1.0-1622102850640
-rw-r--r-- 1 impala impala 3.7M May 28 01:36 impala_audit_event_log_1.0-1622107032783
-rw-r--r-- 1 impala impala 2.7M May 28 14:34 impala_audit_event_log_1.0-1622122581046
-rw-r--r-- 1 impala impala 4.7M May 28 18:17 impala_audit_event_log_1.0-1622169268919
-rw-r--r-- 1 impala impala 4.3M May 28 19:34 impala_audit_event_log_1.0-1622182634220
-rw-r--r-- 1 impala impala 4.3M May 28 20:30 impala_audit_event_log_1.0-1622187283341
-rw-r--r-- 1 impala impala 4.3M May 28 20:58 impala_audit_event_log_1.0-1622190632083
-rw-r--r-- 1 impala impala 4.4M May 28 21:25 impala_audit_event_log_1.0-1622192312991 Could anyone please share some light as to why this is happening? I was under the impression that if this is checked only then logs would get captured, kindly clarify CM / CDH – 6.3.3 Thanks Amn
... View more
Labels:
05-02-2021
07:45 PM
Hi @amk Hi, When I go to /var/lib/cloudera-scm-headlamp/hdfs/ I see the below listed : [nm1@myserver hdfs]# ls -ltr -rw-r----- 1 cloudera-scm cloudera-scm 710967007 Jan 29 23:35 fsimage -rw-r----- 1 cloudera-scm cloudera-scm 25680 Jan 29 23:44 aggregates drwxr-x--- 2 cloudera-scm cloudera-scm 8192 Jan 29 23:44 index -rw-r----- 1 cloudera-scm cloudera-scm 468901714 May 3 14:26 fsimage.tmp Are you suggesting that I move everything that is inside hdfs folder to tmp location that then start Rman server? Kindly confirm. Thanks Amn
... View more
04-15-2021
12:20 AM
Hello, I was trying to take report from HDFS Report and get the below message Upon checking /var/log/cloudera-scm-headlamp and see the following error 021-04-15 18:09:44,934 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:80)
at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:257)
at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:325)
at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:240)
at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:57)
at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:494)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:74)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:221)
at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:211)
at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:236)
at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:77)
at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:72)
... 8 more
Caused by: java.io.IOException: Unsupported layout version -64
at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:113)
at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:70)
... 9 more CM / CDH - 6.3.3 Appreciate any help / guidance in fixing this issue . Thanks Amn
... View more
Labels:
03-26-2021
04:25 AM
Hello, Need some assistance / guidance on how we can reduce Non-HDFS Space. We see Non-HDFS Space of around 270 used, as we are facing space crunch, we would explore possibilities for getting non-hdfs space reduced. I have cleared all Yarn logs for the applications which were killed/ failed etc (our /data mountpoint houses dfs, yarn, kudu, impala), yet this does not solve our issue. Any assistance / guidance is much appreciated. Thanks Amn
... View more
Labels:
03-23-2021
09:17 PM
Hello, We are getting alerts for Block Count on one of our data nodes as it has crossed the threshold of 10000. Since HDFS balancer did not fix the issue, the next thing I turned my focus to see if we are hitting small files issue. I was trying to put up a report via terminal script ( hdfs dfs -ls -R /tmp |grep ^- |awk '{if ($5 < 134217728) print $5, $8;}'| head -5 | column –t) but when I compare the result from the script output vs HDFS Report from Cloudera Manager I see a difference in the size of the same file. Could anyone provide any guidance / assistance on this, or am I doing something wrong. Thanks Amn
... View more
Labels:
03-16-2021
03:03 AM
Thanks @tjangid Just one doubt does it matter if we keep two dash (--backend_client_rpc_timeout_ms ) or (-backend_client_rpc_timeout_ms ), please confirm. Thanks Amn
... View more
03-15-2021
10:26 AM
Hello, I need to increase the query timeout (backend_client_rpc_timeout_ms) from current 5 minutes to 30 minutes. Could anyone guide me from where in Impala Configurations I can do this change. I did some checking but cannot find any related, appreciate any assistance / guidance. Thanks Amn
... View more
Labels:
03-08-2021
06:16 PM
Hello, We are seeing concerning alert on one of our data node related to File Descriptor (Concerning: Open file descriptors: 16,410. File descriptor limit: 32,768. Percentage in use: 50.08%. Warning threshold: 50.00%.) Would appreciate any help/ guidance to fix this before it goes out of hand. [user1@myserver ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1030544 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [user1@myserver ~]$ cat /proc/sys/fs/file-max 26161091 [user1@myserver ~]$ cat /proc/sys/fs/file-nr 80400 0 26161091 Thanks Amn
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Cloudera Manager
-
HDFS
11-10-2020
10:59 PM
Hello @PabitraDas, Appreciate your assistance, below is the block count on our DNs, as mentioned earlier we have allocated 6 GB JVM Heap for DN's and 10 GB Heap for NN & SNN. Do you suggest to increase DN Heap, or NN / SNN Heap as suggested by Shelton. Block Count: Node 1 = 7421379 Node 2 = 5569699 Node 3 = 6003009 Node 4 = 7444205 Node 5 = 8770674 Node 6 = 8849641 Node 7 = 8232779 Node 8 = 8354714 Node 9 = 8860602 Also, would greatly appreciate if you have any pointers / suggestions (scripts etc. ) to identify small file issue and possible remediation. Thanks Amn
... View more
11-05-2020
07:33 PM
@Shelton Apologies for the delay in replying. For my understanding, if possible, would you please explain how increasing NN Heap would fix DN Pause duration. Thanks in advance Amn
... View more
10-27-2020
12:41 AM
Hello @GangWar @Shelton Appericate your assistance, Following is the information available from NN WebUI- (23,326,719 files and directories, 22,735,340 blocks = 46,062,059 total filesystem object(s). Heap Memory used 5.47 GB of 10.6 GB Heap Memory. Max Heap Memory is 10.6 GB. Non Heap Memory used 120.51 MB of 122.7 MB Commited Non Heap Memory. Max Non Heap Memory is <unbounded>.) Could you please re-confirm whether I need to adjust the NN Heap Memory OR DN heap memory, as the issue is seen on data Node and that too only one data node other 8 seem to be running without any issues. Thanks Amn
... View more
10-26-2020
11:32 PM
Hello, On our data node, we are increasing getting alerts related to Data Node Pause Duration. So far, this is happening on a single data node out of nine data nodes. Following is the error captured from DN logs 2020-10-27 16:20:05,140 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1821ms GC pool 'ParNew' had collection(s): count=1 time=2075ms) Current Java Heap Size of Data Node in Bytes is at 6GB CM / CDH – 5.16.x Any help is appreciated. Regards Amn
... View more
Labels:
09-13-2020
07:33 PM
Hello, How to update DNS part in host name, example from 192.168.0.1.test-1.pl to 192.168.0.1.test.co.pl so that we can open CM GUI via 192.168.0.1.test.co.pl:7180 and other web gui (hue, impala etc) with the new dns name test.co.pl. Regards Anm
... View more
Labels:
07-19-2020
03:07 AM
Hi @GangWar Thanks for your reply.
... View more