About Amn_468

Amn_468 · ‎03-25-2020

Hi @StevenOD I tried to run re-balance tool but I get below error, Failed RPC negotiation. Trace: 0325 20:44:12.092074 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from XXX.XX.XXX.XXX:52183 0325 20:44:12.092167 (+ 93us) server_negotiation.cc:176] Beginning negotiation 0325 20:44:12.092170 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0325 20:44:12.096890 (+ 4720us) server_negotiation.cc:373] Connection header received 0325 20:44:12.098104 (+ 1214us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0325 20:44:12.098105 (+ 1us) server_negotiation.cc:412] Received NEGOTIATE request from client 0325 20:44:12.098128 (+ 23us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0325 20:44:12.098177 (+ 49us) server_negotiation.cc:197] Negotiated authn=SASL 0325 20:44:12.104531 (+ 6354us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0325 20:44:12.106114 (+ 1583us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0325 20:44:12.115849 (+ 9735us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0325 20:44:12.116299 (+ 450us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0325 20:44:12.116346 (+ 47us) server_negotiation.cc:581] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA384 0325 20:44:12.123359 (+ 7013us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from XXX.XX.XXX.XXX:52183: BlockingRecv error: failed to read from TLS socket: Cannot send after transport endpoint shutdown (error 108) Metrics: {"server-negotiator.queue_time_us":53} Thanks Amn

Amn_468 · ‎03-22-2020

Hi, We are repeatedly getting alerts for NN Pause duration (The health test result for NAME_NODE_PAUSE_DURATION has become bad: Average time spent paused was 44.5 second(s) (74.25%) per minute over the previous 5 minute(s). Critical threshold: 60.00%.) CM / CDH - 5.16.2 Current NN Heap Size - 4GB Block Used - 5TB Request some assistance in fixing this. Thanks Amn

Amn_468 · ‎03-19-2020

Any help on this one ?

Amn_468 · ‎03-17-2020

Hi, Do we need to stop or put all hadoop services in maintenance, during our DB patching activity or just putting Cloudera Manager in maintenance mode will suffice? Thanks Anm

Amn_468 · ‎03-01-2020

Hello, I would like to know if there a way to rebalance data in Kudu evenly across all kudu t-servers. Our Kudu deployment is as follows: 3 Kudu Masters 9 Tablet Servers kudu 1.7.0-cdh5.16.2/ CM 5.16.2 Data across these 9 T-Servers is not evenly distributed, out these 9 t-servers, I see data is more stored on 3 t-servers and not distributed evenly. I was going through some articles and found that currently there is no rebalance tool like HDFS (https://community.cloudera.com/t5/Support-Questions/Kudu-Tablet-Server-Data-Directories-rebalancing/td-p/79649), However if we Go to Clusters > Kudu > Click Actions I see Run Kudu Rebalancer Tool, would like to know, what is the purpose of this. Will this distribute data for overall Kudu or just Kudu Master or Kudu T-Servers too. Request some advice / assistance on the same. Thanks Amn

Amn_468 · ‎02-10-2020

Hi @kingpin Appreciate all your assistance Regards Amn

Amn_468 · ‎02-09-2020

Hi @kingpin Thanks for the assistance, wanted to confirm if restarting TS have any impact on Kudu service? Regards

Amn_468 · ‎01-23-2020

Hello, We have been getting alerts for Open file descriptors on Kudu T-servers “Concerning: Open file descriptors: 16,954. File descriptor limit: 32,768. Percentage in use: 51.74%. Warning threshold: 50.00%.” request assistance in resolving this. Also does restarting services on affected T-Servers have any performance impact on Kudu as whole. [root@myserver ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1546606 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1546606 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@myserver ~]# cat /proc/sys/fs/file-max 39233193 [root@myserver ~]# cat /proc/sys/fs/file-nr 35200 0 39233193 [root@myserver ~]# lsof -u kudu |wc -l 18326 Regards Amn

Amn_468 · ‎12-12-2019

@EricL Thanks for your reply, I did take a look at that at the first place, but did not find anything useful. Is there anything specific error to lookout for

Amn_468 · ‎12-10-2019

Hello All, Our Application has reported this error and all the application were stuck, after restarting all the applications they are back, now we are looking for rca I tried to find something but an hitting the wall, would be great if anyone can share some thoughts on it or guide on how to proceed further. Error in writeTable org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table my_table_xx_123. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; 19/12/10 21:21:22 INFO hive.metastore: Trying to connect to metastore with URI thrift://myserver.com:9083 19/12/10 21:21:22 ERROR dataloaderlogger: Error in writeTable org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table network_element_kpi_nrt. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; 19/12/10 21:21:22 WARN hive.metastore: Failed to connect to the MetaStore Server... 19/12/10 21:21:22 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 19/12/10 21:21:23 INFO hive.metastore: Trying to connect to metastore with URI thrift://myserver.com:9083 19/12/10 21:21:23 WARN hive.metastore: Failed to connect to the MetaStore Server... 19/12/10 21:21:23 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 19/12/10 21:21:24 INFO hive.metastore: Trying to connect to metastore with URI thrift://myserver.com:9083 19/12/10 21:21:24 WARN hive.metastore: Failed to connect to the MetaStore Server... 19/12/10 21:21:24 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 19/12/10 21:21:25 WARN client.HiveClientImpl: HiveClient got thrift exception, destroying client and retrying (0 tries remaining) org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table data_collection_logs. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1194) Thanks

Amn_468 · ‎11-28-2019

Hi Tim, Thanks for you reply, we ran couple of other queries and don't encounter this issue, we will resolve this as a one of case, appreciate your explanation in this regard.

Amn_468 · ‎11-27-2019

Hello, I am facing an issue with Impala queries, though the query status says 'finished' on Impala status page it still shows 'executing'. I was following Cloudera Article : https://my.cloudera.com/knowledge/Finished-Queries-show-as-Executing-in-the-Cloudera-Manager?id=71576 the difference in my case is queries were run through impala shell via jdbc connection. would the resolution outlined in the article be applicable, for queries submitted through impala jdbc connection? and secondly what would be the fix for such queries which are in hung state. Regards Amn

Amn_468 · ‎11-25-2019

Hi Robbiez, Strangely while checking on other Daemons I see different error, although the error received from CM is the same (The health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 0 error(s) seen monitoring executing queries, and 1 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any. ) whereas the logs paint a different picture, the new log I see is attached and I see the same in other daemons also although the file numbers are a bit different. [root@Server02 impalad]$ tail -n 100 impalad.Server02.impala.log.WARNING.20191120-115803.6341 at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738) at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:666) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:904) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:981) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:147) W1126 14:44:49.208544 6755 ShortCircuitCache.java:826] ShortCircuitCache(0x116b2697): could not load 1122093192_BP-196508081-172.18.208.227-1510815452859 due to InvalidToken exception. Java exception follows: org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/MYAPPLICATION.db/network/id=4/PRT_id=31268741116146977/part-00000_copy_293 Regards Amn

Amn_468 · ‎11-25-2019

Hi Attila, Appreciate your reply, not quite sure when you say 'how are clients accessing" as far as I know we use Impala to query kudu tables, if that makes sense…. still trying to learn the ropes of Kudu. Where and how can I check ? Regards Amn

Amn_468 · ‎11-24-2019

Hello All, We are observing below error on our Kudu Master server, CDH / CM = 5.16.2 Kudu 1.7.0 memory_limit_hard_bytes = 35GB W1125 14:29:26.059293 39728 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from KUDU MASTER IP:60098 dropped due to backpressure. The service queue is full; it has 50 items. W1125 16:12:31.944316 14678 negotiation.cc:313] Failed RPC negotiation. Trace: 1125 16:12:28.943331 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:33960 1125 16:12:28.943495 (+ 164us) server_negotiation.cc:176] Beginning negotiation 1125 16:12:28.943498 (+ 3us) server_negotiation.cc:365] Waiting for connection header 1125 16:12:28.944585 (+ 1087us) server_negotiation.cc:373] Connection header received 1125 16:12:31.944237 (+2999652us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:33960 Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1} W1125 16:32:01.273672 26481 negotiation.cc:313] Failed RPC negotiation. Trace: 1125 16:31:58.273342 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:34488 1125 16:31:58.273518 (+ 176us) server_negotiation.cc:176] Beginning negotiation 1125 16:31:58.273521 (+ 3us) server_negotiation.cc:365] Waiting for connection header 1125 16:31:58.274365 (+ 844us) server_negotiation.cc:373] Connection header received 1125 16:32:01.273565 (+2999200us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:34488 Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1} W1125 16:42:01.186538 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER1:49284 dropped due to backpressure. The service queue is full; it has 50 items. W1125 16:46:10.786643 39729 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER2:34404 dropped due to backpressure. The service queue is full; it has 50 items. W1125 17:40:18.836066 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER3:43694 dropped due to backpressure. The service queue is full; it has 50 items. Request some assistance in resolving / fixing this issue Regards Amn

Amn_468 · ‎11-20-2019

Hello, We have been receiving alerts for Impala [ The health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 0 error(s) seen monitoring executing queries, and 1 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any.] I was following Cloudera Article ( https://my.cloudera.com/knowledge/Impala--IMPALADQUERYMONITORINGSTATUS-has-become-bad?id=75434) but am unsure if this would apply to our environment (our is CDH 5.16.2 / CH 5.16.2), also the 2 Apache Jira mentioned in this Article is resolved. Below is the log from one of our Impala daemon. I have also check and there are no failed queries (screenshot attached) Current Query Monitoring Timeout = 1 minute / Query Monitoring Period = 1 minute. Increasing the values as mentioned in the article would postpone the alerts, would like to know if there is a fix for this issue. Impalad Logs [root@NTPDQ cloudera-scm-agent]# tail -n 100 cloudera-scm-agent.log result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 38, in _subprocess_with_timeout return subprocess_with_timeout(args, timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout raise Exception("timeout with args %s" % args) Exception: timeout with args ['ntpq', '-np'] [21/Nov/2019 17:01:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.90371108055 [21/Nov/2019 17:01:29 +0000] 5415 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) for requested version . [21/Nov/2019 17:03:18 +0000] 5415 MainThread agent WARNING Long HB processing time: 5.24657082558 [21/Nov/2019 17:05:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.19837498665 [21/Nov/2019 17:06:18 +0000] 5415 MainThread agent WARNING Long HB processing time: 5.16436910629 [21/Nov/2019 17:07:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.74405503273 [21/Nov/2019 17:07:29 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.06 mean:0.10 max:0.25 LIFE_MAX:45.16 [21/Nov/2019 17:10:24 +0000] 5415 MainThread agent WARNING Long HB processing time: 10.0939228535 [21/Nov/2019 17:11:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.39454698563 [21/Nov/2019 17:12:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.03050684929 [21/Nov/2019 17:13:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 5.31969809532 [21/Nov/2019 17:14:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.35476112366 [21/Nov/2019 17:15:27 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.7484540939 [21/Nov/2019 17:16:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.23533391953 [21/Nov/2019 17:17:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 23.2213561535 [21/Nov/2019 17:17:22 +0000] 5415 MainThread agent WARNING Delayed HB: 8s since last [21/Nov/2019 17:17:37 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.05 mean:0.09 max:0.95 LIFE_MAX:45.16 [21/Nov/2019 17:18:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 11.9228010178 [21/Nov/2019 17:19:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 12.488642931 [21/Nov/2019 17:20:27 +0000] 5415 MainThread agent WARNING Long HB processing time: 20.2072188854 [21/Nov/2019 17:20:27 +0000] 5415 MainThread agent WARNING Delayed HB: 5s since last [21/Nov/2019 17:21:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.10305094719 [21/Nov/2019 17:22:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.03495693207 [21/Nov/2019 17:23:00 +0000] 5415 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants user CPU lower than expected for process 29048: 1419823.60001, 245602.28 [21/Nov/2019 17:23:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.17436099052 [21/Nov/2019 17:24:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.09477305412 [21/Nov/2019 17:25:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.74135708809 [21/Nov/2019 17:26:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.5287861824 [21/Nov/2019 17:27:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.24944400787 [21/Nov/2019 17:27:44 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.06 mean:0.07 max:0.11 LIFE_MAX:45.16 [21/Nov/2019 17:29:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.15325498581 [21/Nov/2019 17:30:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.27527880669 [21/Nov/2019 17:31:17 +0000] 5415 MainThread agent WARNING Long HB processing time: 18.4421730042 [21/Nov/2019 17:31:17 +0000] 5415 MainThread agent WARNING Delayed HB: 3s since last [21/Nov/2019 17:31:24 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.10764193535 [21/Nov/2019 17:32:22 +0000] 5415 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) for requested version . [21/Nov/2019 17:32:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 19.3404970169 [21/Nov/2019 17:32:22 +0000] 5415 MainThread agent WARNING Delayed HB: 4s since last [21/Nov/2019 17:33:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 14.3549408913 [21/Nov/2019 17:34:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.6162810326 [21/Nov/2019 17:35:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.1593289375 [21/Nov/2019 17:36:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.8171610832 [21/Nov/2019 17:37:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 12.1159389019 [21/Nov/2019 17:37:52 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.06 mean:0.09 max:1.04 LIFE_MAX:45.16 [21/Nov/2019 17:38:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.9521288872 [21/Nov/2019 17:39:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 12.0477230549 [21/Nov/2019 17:39:58 +0000] 5415 Monitor-HostMonitor throttling_logger ERROR (2 skipped) Timeout with args ['ntpq', '-np'] None [21/Nov/2019 17:39:58 +0000] 5415 Monitor-HostMonitor throttling_logger ERROR (2 skipped) Failed to collect NTP metrics Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 48, in collect self.collect_ntpd() File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 66, in collect_ntpd result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 38, in _subprocess_with_timeout return subprocess_with_timeout(args, timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout raise Exception("timeout with args %s" % args) Exception: timeout with args ['ntpq', '-np'] [21/Nov/2019 17:40:29 +0000] 5415 ImpalaDaemonQueryMonitoring query_monitor ERROR Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log' Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 901, in _get_completed_query_profiles self._query_monitor.get_completed_queries(query_log_file) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 581, in get_completed_queries completed_query_report_limit) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 456, in get_completed_queries last_accessed_file_timestamp) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 265, in _get_completed_queries for event in streamer.stream(): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/event_streamer.py", line 134, in stream file_list = self.__sort_file_list(self.__filtered_file_list, ascending) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/event_streamer.py", line 162, in __sort_file_list handle = f.open_log_file() File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/file.py", line 41, in open_log_file handle = open(self.__path) IOError: [Errno 2] No such file or directory: '/var/log/impalad/profiles/impala_profile_log_1.1-1574308887326' [21/Nov/2019 17:40:29 +0000] 5415 MainThread agent WARNING Long HB processing time: 22.2102739811 [21/Nov/2019 17:40:29 +0000] 5415 MainThread agent WARNING Delayed HB: 7s since last [21/Nov/2019 17:41:24 +0000] 5415 MainThread agent WARNING Long HB processing time: 25.2082297802 [21/Nov/2019 17:41:24 +0000] 5415 MainThread agent WARNING Delayed HB: 10s since last [21/Nov/2019 17:42:23 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.1406769753 [21/Nov/2019 17:43:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 11.081389904 [21/Nov/2019 17:44:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 10.6936099529 [21/Nov/2019 17:45:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.0398440361 [21/Nov/2019 17:46:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.3374710083 [21/Nov/2019 17:47:18 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.1221549511 [21/Nov/2019 17:47:55 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:39 LIFE_MIN:0.00 min:0.06 mean:0.08 max:0.24 LIFE_MAX:45.16 [21/Nov/2019 17:48:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.35701394081 [21/Nov/2019 17:49:17 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.75842308998 [21/Nov/2019 17:50:23 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.1954960823 [21/Nov/2019 17:51:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.66306400299 [21/Nov/2019 17:52:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.91945910454 [21/Nov/2019 17:53:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.45511198044 [21/Nov/2019 17:54:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.34867501259 Thanks Amn

Amn_468 · ‎11-18-2019

I see all my Tablet Servers Healthy and the Summary by Table also shows them Healthy. Nothing in 'Recovering / Under-Replicated / Unavailable'

Amn_468 · ‎11-17-2019

@awong Checked all 9 tablet servers all are done bootstrapping, I see the same results as I posted in the previous screenshot although the numbers are different, but its all at 100% Regards Amn

Amn_468 · ‎11-17-2019

Hi Awong, Thanks for the quick reply, following is what I see, based on the screenshot what I understand is that the bootstrap process is completed, as it says 100%, also when I click on Details > toggle I see Under Last Status as either -Bootstrap complete. or No bootstrap required, opened a new log, for the corresponding Table Name. When I check the logs I still see the same previous error. anything else I can check?? Regards Amn

Amn_468 · ‎11-17-2019

Hi All, I am getting the below error on one of my Kudu tablet server, I have restarted table server services on this host yet when I check them I continue to get this error W1118 19:43:31.815698 33067 consensus_peers.cc:435] T 3292e490cf4843d994a45f9a4c7782c0 P cc36320dd81646d081a24203751c2a6a -> Peer 164c8bcafccc4fd0adfb6dfe7a2ff60e (MYSERVER.com:7050): Couldn't send request to peer 164c8bcafccc4fd0adfb6dfe7a2ff60e for tablet 3292e490cf4843d994a45f9a4c7782c0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. Retrying in the next heartbeat period. Already tried 389 times. Any help is much appreciated Regards Amn

Amn_468 · ‎10-08-2019

Hi Li, Thanks for all your help and assistance 🙂

Amn_468 · ‎10-06-2019

Hello All, We are facing issue with one of our clients cluster, we see some files over 2GB, and would like to know the purpose of these files, and if these could be deleted to make way for some space. 1. Eventserver [root@TestBed ~]# du -sh /var/lib/cloudera-scm-eventserver/v3/* |grep G 5.4G /var/lib/cloudera-scm-eventserver/v3/_1bx6l.fdt 1.2G /var/lib/cloudera-scm-eventserver/v3/_1hgcy.fdt 1.4G /var/lib/cloudera-scm-eventserver/v3/_1mz17.fdt 2. Cloudera-Scm-Headlamp [root@TestBed ~]# du -sh /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/* |grep G 2.3G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_2t.fdt 2.2G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_5m.fdt 2.1G /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/index/_8f.fdt Also is there a way to limit TS data in SMON & HMON, I understand that 10GB is the minimum requirement for these roles but what could be done if a clients has limited resources. Any help / guidance is appreciated Thanks

Amn_468 · ‎09-04-2019

Hi Eric, Thanks for your reply, post your reply, I have more questions than answers 🙂 1) Is there a way to find if impala deamon can handle additional connections (from default 64 to 128 and more) 2) Can we find the number of connections that are made in real time? (Apart from Active Frontend API Connections Chart) 3) How are connection closed (i.e. are they closed once an impala query is completed or is there some other process) 4) Any best practice recommendation for Swap Memory setting in Impala. Regards Amn

Amn_468 · ‎08-27-2019

Hello All, We have been receiving alerts of IMPALAD_FRONTEND_CONNECTIONS becoming bad "There are 0 (Beeswax pool) 55 (Hive Server 2 pool) active client connections, each pool has a configured maximum of 64" Impala Daemon Concurrent Client Connections Monitoring Percentage Thresholds Warning = 80 Critical = 95 Impala Daemon Max Client Connections = 64 CM - 5.16.2 CDH - 5.16.2 Will increasing Impala Daemon Max Client Connections reduce / stop these alerts? Any help is most welcome.

Amn_468 · ‎07-26-2019

Hi EricL, Thanks for your help.

Amn_468 · ‎07-25-2019

Hello, We have upgraded our cluster and are getting error " The Oozie Server build version(4.1.0-cdh5.16.2) and the Oozie Server shared library version(4.1.0-cdh5.14.2) do not match". We have done the following: Upgrade the Oozie SharedLib Go to the Oozie service If the Oozie service is stopped, start it: Select Actions > Start and click Start to confirm. Select Actions > Install Oozie SharedLib and click Install Oozie SharedLib to confirm. The service became Green from some time and then again its Red, kindly suggest / guide path to fix this

Amn_468 · ‎04-17-2019

Thank you all for your help

Amn_468 · ‎04-10-2019

Hi Harsh J / Tomas79, Thank you for your responses, instead of disabling Audit & Lineage Log collection, I would be more inclined if can I reduce the default settings of 5000 to 1000, would there be any negative consequences? Appreciate your insights / advice.

Amn_468 · ‎04-08-2019

Hello All, Could someone appraise me the significance of Impala Audit & Lineage Logs. We are facing space issue and would like to know whether we can delete some of those old log files, also the default setting for Impala Daemon Maximum Lineage Log File Size & Impala Daemon Maximum Audit Log File Size is 5000 can this be lowered to 1000 etc. Regards Wert

Online	Offline
Last Visited	‎03-25-2020 10:39 PM

Date Registered	‎02-18-2019 07:29 PM
Last Visited	‎03-25-2020 10:39 PM
Total Messages Posted	29

Re: Kudu T-Server Re-balancing

Name Node Pause duration

Re: Kudu T-Server Re-balancing

Hadoop DB question

Kudu T-Server Re-balancing

Re: Open file descriptors issue on Kudu

Re: Open file descriptors issue on Kudu

Open file descriptors issue on Kudu

Re: Hive Error

Hive Error

Re: Impala query troubles

Impala query troubles

Re: Impalad Query Monitoring Error

Re: Kudu Issue

Kudu Issue

Impalad Query Monitoring Error

Re: Kudu T-Server Error

Re: Kudu T-Server Error

Re: Kudu T-Server Error

Kudu T-Server Error

Re: Cloudera Manager Space Constraints

Cloudera Manager Space Constraints

Re: Impala Client Connections

Impala Client Connections

Re: Oozie error after upgrade

Oozie error after upgrade

Re: Impala Deamon Logs

Re: Impala Deamon Logs

Impala Deamon Logs