About Amn_468

Amn_468 · ‎09-13-2020

Hello, How to update DNS part in host name, example from 192.168.0.1.test-1.pl to 192.168.0.1.test.co.pl so that we can open CM GUI via 192.168.0.1.test.co.pl:7180 and other web gui (hue, impala etc) with the new dns name test.co.pl. Regards Anm

Amn_468 · ‎07-19-2020

Hi @GangWar Thanks for your reply.

Amn_468 · ‎07-17-2020

Hello, I need to reboot my Data node to fix an underlying network issue. OS team needs around 3 hours (approx.) to complete this activity, what would be the best approach: Decommission and recommission host / dn. Move DN to Offline Mode (feature in CM 5.16.x) Any help / guidance is appreciated. Regards Amn

Amn_468 · ‎07-05-2020

Hello, We are observing error on our Impala Query Status: InternalException: Error updating the catalog due to lock contention. When we see from Impala >Queries it appeares to be showing as Executing but when we check in query details we see error in query status. Session Type: HIVESERVER2 Impala Version: impalad version 2.12.0-cdh5.16.2 Query Type: DML Query State: EXCEPTION Any assistance is appreciated. Regards Amn.

Amn_468 · ‎06-17-2020

Hi @tjangid Thanks for your reply, in my previous post I incorrectly mentioned that we want to move from MIT Kerberos to AD, whereas, we currently have MIT Kerberos (local) working in our cluster and we need that to be integrated with AD. So basically I am looking to find / get some detailed steps / guides on how to get this done. I have come across some blogs regarding one-way cross-realm trust etc, and a bit confused on these. Appreciate any help in this regard Thanks

Amn_468 · ‎06-05-2020

Hello, In our Cluster we have MIT Kerberos authentication enabled, we would like to move to AD Authentication, would appreciate if someone could share best practices / documents / how to etc, on how to move forward on this and what changes would be required in order to achieve this mission. Regards Amn

Amn_468 · ‎05-14-2020

Hi@Madhur Appreciate your assistance, I am using CM, where would this setting be in CM for making the changes in Cluster level, and to confirm these values have to be passed in seconds ? Could you also provide steps / document outlining how to change this while submitting spark jobs.

Amn_468 · ‎05-14-2020

Hi @Madhur This is happening with all Spark jobs, there has been no changes in the code or cluster, also the failure is random.

Amn_468 · ‎05-13-2020

Hello All, We are running Spark jobs via yarn and its failing with the below error, any help / pointer to fix is much appericated. Shell output: main : command provided 1 main : run as user is TEST1 main : requested yarn user is TEST1 Writing to tmp file /data/8/yarn/nm/nmPrivate/application_1587389136999_0013/container_e56_1587389136999_0013_01_000477/container_e56_1587389136999_0013_01_000477.pid.tmp Writing to cgroup task files... Container exited with a non-zero exit code 1 org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply in 120 seconds. This timeout is controlled by spark.rpc.askTimeout at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216) at scala.util.Try$.apply(Try.scala:192) at scala.util.Failure.recover(Try.scala:216) at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:326) at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:326) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at org.spark_project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293) at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136) at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) at scala.concurrent.Promise$class.complete(Promise.scala:55) at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:63) at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:78) at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:55) at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:55) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:54) at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601) at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:106) at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599) at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) at scala.concurrent.Promise$class.tryFailure(Promise.scala:112) at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153) at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onFailure$1(NettyRpcEnv.scala:205) at org.apache.spark.rpc.netty.NettyRpcEnv$$anon$1.run(NettyRpcEnv.scala:239) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Cannot receive any reply in 120 seconds ... 8 more Regards Amn

Amn_468 · ‎04-09-2020

Hi, Wanted to know if SAN can be used to store kudu related data instead of a physical HDD on a server. Example: drwx------. 2 kudu kudu 151552 Mar 9 16:24 consensus-meta drwx------. 2 kudu kudu 4624384 Mar 9 16:20 data -rw-------. 1 kudu kudu 681 Nov 16 2017 instance drwx------. 2 kudu kudu 118784 Mar 9 16:23 tablet-meta drwx------. 1775 kudu kudu 110592 Mar 9 13:49 wals Thanks Amn

Amn_468 · ‎04-01-2020

Hi @venkatsambath Appericate your help, would like some assistance in configuring NN heap size, would like to know what would be the best way to move forward: Increase NN heap via Cloudera Manager (HDFS>Config> NN>Heap (HA Cluster)) OR Changing it via /etc/hadoop/conf/hadoop-env.sh (HADOOP_NAMENODE_OPTS) Kindly Advise Regards Amn

Amn_468 · ‎03-25-2020

Hi @StevenOD I tried to run re-balance tool but I get below error, Failed RPC negotiation. Trace: 0325 20:44:12.092074 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from XXX.XX.XXX.XXX:52183 0325 20:44:12.092167 (+ 93us) server_negotiation.cc:176] Beginning negotiation 0325 20:44:12.092170 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0325 20:44:12.096890 (+ 4720us) server_negotiation.cc:373] Connection header received 0325 20:44:12.098104 (+ 1214us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0325 20:44:12.098105 (+ 1us) server_negotiation.cc:412] Received NEGOTIATE request from client 0325 20:44:12.098128 (+ 23us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0325 20:44:12.098177 (+ 49us) server_negotiation.cc:197] Negotiated authn=SASL 0325 20:44:12.104531 (+ 6354us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0325 20:44:12.106114 (+ 1583us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0325 20:44:12.115849 (+ 9735us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0325 20:44:12.116299 (+ 450us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0325 20:44:12.116346 (+ 47us) server_negotiation.cc:581] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA384 0325 20:44:12.123359 (+ 7013us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from XXX.XX.XXX.XXX:52183: BlockingRecv error: failed to read from TLS socket: Cannot send after transport endpoint shutdown (error 108) Metrics: {"server-negotiator.queue_time_us":53} Thanks Amn

Amn_468 · ‎03-22-2020

Hi, We are repeatedly getting alerts for NN Pause duration (The health test result for NAME_NODE_PAUSE_DURATION has become bad: Average time spent paused was 44.5 second(s) (74.25%) per minute over the previous 5 minute(s). Critical threshold: 60.00%.) CM / CDH - 5.16.2 Current NN Heap Size - 4GB Block Used - 5TB Request some assistance in fixing this. Thanks Amn

Amn_468 · ‎03-19-2020

Any help on this one ?

Amn_468 · ‎03-17-2020

Hi, Do we need to stop or put all hadoop services in maintenance, during our DB patching activity or just putting Cloudera Manager in maintenance mode will suffice? Thanks Anm

Amn_468 · ‎03-01-2020

Hello, I would like to know if there a way to rebalance data in Kudu evenly across all kudu t-servers. Our Kudu deployment is as follows: 3 Kudu Masters 9 Tablet Servers kudu 1.7.0-cdh5.16.2/ CM 5.16.2 Data across these 9 T-Servers is not evenly distributed, out these 9 t-servers, I see data is more stored on 3 t-servers and not distributed evenly. I was going through some articles and found that currently there is no rebalance tool like HDFS (https://community.cloudera.com/t5/Support-Questions/Kudu-Tablet-Server-Data-Directories-rebalancing/td-p/79649), However if we Go to Clusters > Kudu > Click Actions I see Run Kudu Rebalancer Tool, would like to know, what is the purpose of this. Will this distribute data for overall Kudu or just Kudu Master or Kudu T-Servers too. Request some advice / assistance on the same. Thanks Amn

Amn_468 · ‎02-10-2020

Hi @kingpin Appreciate all your assistance Regards Amn

Amn_468 · ‎02-09-2020

Hi @kingpin Thanks for the assistance, wanted to confirm if restarting TS have any impact on Kudu service? Regards

Amn_468 · ‎01-23-2020

Hello, We have been getting alerts for Open file descriptors on Kudu T-servers “Concerning: Open file descriptors: 16,954. File descriptor limit: 32,768. Percentage in use: 51.74%. Warning threshold: 50.00%.” request assistance in resolving this. Also does restarting services on affected T-Servers have any performance impact on Kudu as whole. [root@myserver ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 1546606 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1546606 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [root@myserver ~]# cat /proc/sys/fs/file-max 39233193 [root@myserver ~]# cat /proc/sys/fs/file-nr 35200 0 39233193 [root@myserver ~]# lsof -u kudu |wc -l 18326 Regards Amn

Amn_468 · ‎12-12-2019

@EricL Thanks for your reply, I did take a look at that at the first place, but did not find anything useful. Is there anything specific error to lookout for

Amn_468 · ‎12-10-2019

Hello All, Our Application has reported this error and all the application were stuck, after restarting all the applications they are back, now we are looking for rca I tried to find something but an hitting the wall, would be great if anyone can share some thoughts on it or guide on how to proceed further. Error in writeTable org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table my_table_xx_123. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; 19/12/10 21:21:22 INFO hive.metastore: Trying to connect to metastore with URI thrift://myserver.com:9083 19/12/10 21:21:22 ERROR dataloaderlogger: Error in writeTable org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table network_element_kpi_nrt. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient; 19/12/10 21:21:22 WARN hive.metastore: Failed to connect to the MetaStore Server... 19/12/10 21:21:22 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 19/12/10 21:21:23 INFO hive.metastore: Trying to connect to metastore with URI thrift://myserver.com:9083 19/12/10 21:21:23 WARN hive.metastore: Failed to connect to the MetaStore Server... 19/12/10 21:21:23 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 19/12/10 21:21:24 INFO hive.metastore: Trying to connect to metastore with URI thrift://myserver.com:9083 19/12/10 21:21:24 WARN hive.metastore: Failed to connect to the MetaStore Server... 19/12/10 21:21:24 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 19/12/10 21:21:25 WARN client.HiveClientImpl: HiveClient got thrift exception, destroying client and retrying (0 tries remaining) org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table data_collection_logs. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1194) Thanks

Amn_468 · ‎11-28-2019

Hi Tim, Thanks for you reply, we ran couple of other queries and don't encounter this issue, we will resolve this as a one of case, appreciate your explanation in this regard.

Amn_468 · ‎11-27-2019

Hello, I am facing an issue with Impala queries, though the query status says 'finished' on Impala status page it still shows 'executing'. I was following Cloudera Article : https://my.cloudera.com/knowledge/Finished-Queries-show-as-Executing-in-the-Cloudera-Manager?id=71576 the difference in my case is queries were run through impala shell via jdbc connection. would the resolution outlined in the article be applicable, for queries submitted through impala jdbc connection? and secondly what would be the fix for such queries which are in hung state. Regards Amn

Amn_468 · ‎11-25-2019

Hi Robbiez, Strangely while checking on other Daemons I see different error, although the error received from CM is the same (The health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 0 error(s) seen monitoring executing queries, and 1 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any. ) whereas the logs paint a different picture, the new log I see is attached and I see the same in other daemons also although the file numbers are a bit different. [root@Server02 impalad]$ tail -n 100 impalad.Server02.impala.log.WARNING.20191120-115803.6341 at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:804) at org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:738) at org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:485) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:666) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:904) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:981) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:147) W1126 14:44:49.208544 6755 ShortCircuitCache.java:826] ShortCircuitCache(0x116b2697): could not load 1122093192_BP-196508081-172.18.208.227-1510815452859 due to InvalidToken exception. Java exception follows: org.apache.hadoop.security.token.SecretManager$InvalidToken: access control error while attempting to set up short-circuit access to /user/hive/warehouse/MYAPPLICATION.db/network/id=4/PRT_id=31268741116146977/part-00000_copy_293 Regards Amn

Amn_468 · ‎11-25-2019

Hi Attila, Appreciate your reply, not quite sure when you say 'how are clients accessing" as far as I know we use Impala to query kudu tables, if that makes sense…. still trying to learn the ropes of Kudu. Where and how can I check ? Regards Amn

Amn_468 · ‎11-24-2019

Hello All, We are observing below error on our Kudu Master server, CDH / CM = 5.16.2 Kudu 1.7.0 memory_limit_hard_bytes = 35GB W1125 14:29:26.059293 39728 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from KUDU MASTER IP:60098 dropped due to backpressure. The service queue is full; it has 50 items. W1125 16:12:31.944316 14678 negotiation.cc:313] Failed RPC negotiation. Trace: 1125 16:12:28.943331 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:33960 1125 16:12:28.943495 (+ 164us) server_negotiation.cc:176] Beginning negotiation 1125 16:12:28.943498 (+ 3us) server_negotiation.cc:365] Waiting for connection header 1125 16:12:28.944585 (+ 1087us) server_negotiation.cc:373] Connection header received 1125 16:12:31.944237 (+2999652us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:33960 Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":54,"threads_started":1} W1125 16:32:01.273672 26481 negotiation.cc:313] Failed RPC negotiation. Trace: 1125 16:31:58.273342 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 172.12.345.678:34488 1125 16:31:58.273518 (+ 176us) server_negotiation.cc:176] Beginning negotiation 1125 16:31:58.273521 (+ 3us) server_negotiation.cc:365] Waiting for connection header 1125 16:31:58.274365 (+ 844us) server_negotiation.cc:373] Connection header received 1125 16:32:01.273565 (+2999200us) negotiation.cc:304] Negotiation complete: Timed out: Server connection negotiation failed: server connection from 172.12.345.678:34488 Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1} W1125 16:42:01.186538 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER1:49284 dropped due to backpressure. The service queue is full; it has 50 items. W1125 16:46:10.786643 39729 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER2:34404 dropped due to backpressure. The service queue is full; it has 50 items. W1125 17:40:18.836066 39726 service_pool.cc:130] GetTableSchema request on kudu.master.MasterService from TSERVER3:43694 dropped due to backpressure. The service queue is full; it has 50 items. Request some assistance in resolving / fixing this issue Regards Amn

Amn_468 · ‎11-20-2019

Hello, We have been receiving alerts for Impala [ The health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 0 error(s) seen monitoring executing queries, and 1 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any.] I was following Cloudera Article ( https://my.cloudera.com/knowledge/Impala--IMPALADQUERYMONITORINGSTATUS-has-become-bad?id=75434) but am unsure if this would apply to our environment (our is CDH 5.16.2 / CH 5.16.2), also the 2 Apache Jira mentioned in this Article is resolved. Below is the log from one of our Impala daemon. I have also check and there are no failed queries (screenshot attached) Current Query Monitoring Timeout = 1 minute / Query Monitoring Period = 1 minute. Increasing the values as mentioned in the article would postpone the alerts, would like to know if there is a fix for this issue. Impalad Logs [root@NTPDQ cloudera-scm-agent]# tail -n 100 cloudera-scm-agent.log result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 38, in _subprocess_with_timeout return subprocess_with_timeout(args, timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout raise Exception("timeout with args %s" % args) Exception: timeout with args ['ntpq', '-np'] [21/Nov/2019 17:01:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.90371108055 [21/Nov/2019 17:01:29 +0000] 5415 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) for requested version . [21/Nov/2019 17:03:18 +0000] 5415 MainThread agent WARNING Long HB processing time: 5.24657082558 [21/Nov/2019 17:05:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.19837498665 [21/Nov/2019 17:06:18 +0000] 5415 MainThread agent WARNING Long HB processing time: 5.16436910629 [21/Nov/2019 17:07:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.74405503273 [21/Nov/2019 17:07:29 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.06 mean:0.10 max:0.25 LIFE_MAX:45.16 [21/Nov/2019 17:10:24 +0000] 5415 MainThread agent WARNING Long HB processing time: 10.0939228535 [21/Nov/2019 17:11:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.39454698563 [21/Nov/2019 17:12:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.03050684929 [21/Nov/2019 17:13:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 5.31969809532 [21/Nov/2019 17:14:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.35476112366 [21/Nov/2019 17:15:27 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.7484540939 [21/Nov/2019 17:16:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.23533391953 [21/Nov/2019 17:17:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 23.2213561535 [21/Nov/2019 17:17:22 +0000] 5415 MainThread agent WARNING Delayed HB: 8s since last [21/Nov/2019 17:17:37 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.05 mean:0.09 max:0.95 LIFE_MAX:45.16 [21/Nov/2019 17:18:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 11.9228010178 [21/Nov/2019 17:19:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 12.488642931 [21/Nov/2019 17:20:27 +0000] 5415 MainThread agent WARNING Long HB processing time: 20.2072188854 [21/Nov/2019 17:20:27 +0000] 5415 MainThread agent WARNING Delayed HB: 5s since last [21/Nov/2019 17:21:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.10305094719 [21/Nov/2019 17:22:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.03495693207 [21/Nov/2019 17:23:00 +0000] 5415 MonitorDaemon-Reporter throttling_logger INFO (59 skipped) Descendants user CPU lower than expected for process 29048: 1419823.60001, 245602.28 [21/Nov/2019 17:23:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.17436099052 [21/Nov/2019 17:24:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.09477305412 [21/Nov/2019 17:25:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.74135708809 [21/Nov/2019 17:26:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.5287861824 [21/Nov/2019 17:27:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.24944400787 [21/Nov/2019 17:27:44 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.06 mean:0.07 max:0.11 LIFE_MAX:45.16 [21/Nov/2019 17:29:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 6.15325498581 [21/Nov/2019 17:30:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.27527880669 [21/Nov/2019 17:31:17 +0000] 5415 MainThread agent WARNING Long HB processing time: 18.4421730042 [21/Nov/2019 17:31:17 +0000] 5415 MainThread agent WARNING Delayed HB: 3s since last [21/Nov/2019 17:31:24 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.10764193535 [21/Nov/2019 17:32:22 +0000] 5415 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) for requested version . [21/Nov/2019 17:32:22 +0000] 5415 MainThread agent WARNING Long HB processing time: 19.3404970169 [21/Nov/2019 17:32:22 +0000] 5415 MainThread agent WARNING Delayed HB: 4s since last [21/Nov/2019 17:33:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 14.3549408913 [21/Nov/2019 17:34:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.6162810326 [21/Nov/2019 17:35:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.1593289375 [21/Nov/2019 17:36:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.8171610832 [21/Nov/2019 17:37:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 12.1159389019 [21/Nov/2019 17:37:52 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.00 min:0.06 mean:0.09 max:1.04 LIFE_MAX:45.16 [21/Nov/2019 17:38:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.9521288872 [21/Nov/2019 17:39:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 12.0477230549 [21/Nov/2019 17:39:58 +0000] 5415 Monitor-HostMonitor throttling_logger ERROR (2 skipped) Timeout with args ['ntpq', '-np'] None [21/Nov/2019 17:39:58 +0000] 5415 Monitor-HostMonitor throttling_logger ERROR (2 skipped) Failed to collect NTP metrics Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 48, in collect self.collect_ntpd() File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 66, in collect_ntpd result, stdout, stderr = self._subprocess_with_timeout(args, self._timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/host/ntp_monitor.py", line 38, in _subprocess_with_timeout return subprocess_with_timeout(args, timeout) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/subprocess_timeout.py", line 94, in subprocess_with_timeout raise Exception("timeout with args %s" % args) Exception: timeout with args ['ntpq', '-np'] [21/Nov/2019 17:40:29 +0000] 5415 ImpalaDaemonQueryMonitoring query_monitor ERROR Error fetching completed queries from '/var/log/impalad/profiles/impala_profile_log' Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 901, in _get_completed_query_profiles self._query_monitor.get_completed_queries(query_log_file) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 581, in get_completed_queries completed_query_report_limit) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 456, in get_completed_queries last_accessed_file_timestamp) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.16.2-py2.7.egg/cmf/monitor/impalad/query_monitor.py", line 265, in _get_completed_queries for event in streamer.stream(): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/event_streamer.py", line 134, in stream file_list = self.__sort_file_list(self.__filtered_file_list, ascending) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/event_streamer.py", line 162, in __sort_file_list handle = f.open_log_file() File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/ClusterStatsLogStreaming-UNKNOWN-py2.7.egg/clusterstats/log/streaming/file.py", line 41, in open_log_file handle = open(self.__path) IOError: [Errno 2] No such file or directory: '/var/log/impalad/profiles/impala_profile_log_1.1-1574308887326' [21/Nov/2019 17:40:29 +0000] 5415 MainThread agent WARNING Long HB processing time: 22.2102739811 [21/Nov/2019 17:40:29 +0000] 5415 MainThread agent WARNING Delayed HB: 7s since last [21/Nov/2019 17:41:24 +0000] 5415 MainThread agent WARNING Long HB processing time: 25.2082297802 [21/Nov/2019 17:41:24 +0000] 5415 MainThread agent WARNING Delayed HB: 10s since last [21/Nov/2019 17:42:23 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.1406769753 [21/Nov/2019 17:43:21 +0000] 5415 MainThread agent WARNING Long HB processing time: 11.081389904 [21/Nov/2019 17:44:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 10.6936099529 [21/Nov/2019 17:45:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.0398440361 [21/Nov/2019 17:46:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.3374710083 [21/Nov/2019 17:47:18 +0000] 5415 MainThread agent WARNING Long HB processing time: 8.1221549511 [21/Nov/2019 17:47:55 +0000] 5415 MainThread heartbeat_tracker INFO HB stats (seconds): num:39 LIFE_MIN:0.00 min:0.06 mean:0.08 max:0.24 LIFE_MAX:45.16 [21/Nov/2019 17:48:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.35701394081 [21/Nov/2019 17:49:17 +0000] 5415 MainThread agent WARNING Long HB processing time: 7.75842308998 [21/Nov/2019 17:50:23 +0000] 5415 MainThread agent WARNING Long HB processing time: 13.1954960823 [21/Nov/2019 17:51:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.66306400299 [21/Nov/2019 17:52:20 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.91945910454 [21/Nov/2019 17:53:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.45511198044 [21/Nov/2019 17:54:19 +0000] 5415 MainThread agent WARNING Long HB processing time: 9.34867501259 Thanks Amn

Amn_468 · ‎11-18-2019

I see all my Tablet Servers Healthy and the Summary by Table also shows them Healthy. Nothing in 'Recovering / Under-Replicated / Unavailable'

Amn_468 · ‎11-17-2019

@awong Checked all 9 tablet servers all are done bootstrapping, I see the same results as I posted in the previous screenshot although the numbers are different, but its all at 100% Regards Amn

Amn_468 · ‎11-17-2019

Hi Awong, Thanks for the quick reply, following is what I see, based on the screenshot what I understand is that the bootstrap process is completed, as it says 100%, also when I click on Details > toggle I see Under Last Status as either -Bootstrap complete. or No bootstrap required, opened a new log, for the corresponding Table Name. When I check the logs I still see the same previous error. anything else I can check?? Regards Amn

Online	Offline
Last Visited	‎10-13-2020 06:29 AM

Date Registered	‎02-18-2019 07:29 PM
Last Visited	‎10-13-2020 06:29 AM
Total Messages Posted	42

changing dns name

Re: Maintenance approach for worker / data node

Maintenance approach for worker / data node

Impala Query Error

Re: Switch from MIT Kerberos auth to AD Auth

Switch from MIT Kerberos auth to AD Auth

Re: Spark jobs failing

Re: Spark jobs failing

Spark jobs failing

Kudu Data Storage on SAN

Re: Name Node Pause duration

Re: Kudu T-Server Re-balancing

Name Node Pause duration

Re: Kudu T-Server Re-balancing

Hadoop DB question

Kudu T-Server Re-balancing

Re: Open file descriptors issue on Kudu

Re: Open file descriptors issue on Kudu

Open file descriptors issue on Kudu

Re: Hive Error

Hive Error

Re: Impala query troubles

Impala query troubles

Re: Impalad Query Monitoring Error

Re: Kudu Issue

Kudu Issue

Impalad Query Monitoring Error

Re: Kudu T-Server Error

Re: Kudu T-Server Error

Re: Kudu T-Server Error