Hi, since a few weeks, we have regular warnings on our datanodes : "This role's host has been out of contact with Cloudera Manager for a concerning amount of time." cm-agent is consuming alot of CPU , in particular MonitorDaemon-R (cm-agent PID is 25598): top -H -b -n1 -p 25598 top - 14:08:21 up 22 days, 6:17, 1 user, load average: 5,10, 5,07, 4,98 Threads: 35 total, 2 running, 33 sleeping, 0 stopped, 0 zombie %Cpu(s): 20,1 us, 21,5 sy, 0,0 ni, 56,5 id, 0,0 wa, 0,0 hi, 1,9 si, 0,0 st KiB Mem : 26370784+total, 7788424 free, 38057140 used, 21786227+buff/cache KiB Swap: 31457276 total, 31349500 free, 107776 used. 22445827+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25833 root 20 0 3085980 407988 10016 R 87,5 0,2 124:55.80 MonitorDaemon-R 25598 root 20 0 3085980 407988 10016 S 0,0 0,2 28:07.22 cmagent 25748 root 20 0 3085980 407988 10016 S 0,0 0,2 2:49.93 cmagent 25754 root 20 0 3085980 407988 10016 S 0,0 0,2 0:14.20 Audit-Plugin 25755 root 20 0 3085980 407988 10016 S 0,0 0,2 0:13.98 Metadata-Plugin 25756 root 20 0 3085980 407988 10016 S 0,0 0,2 0:14.35 Profile-Plugin 25800 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.15 _TimeoutMonitor 25801 root 20 0 3085980 407988 10016 S 0,0 0,2 0:10.39 HTTPServer _sta 25802 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.00 CP Server Worke 25803 root 20 0 3085980 407988 10016 S 0,0 0,2 0:03.12 CP Server Worke 25804 root 20 0 3085980 407988 10016 S 0,0 0,2 0:04.83 CP Server Worke 25805 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.03 CP Server Worke 25806 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.01 CP Server Worke 25807 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.00 CP Server Worke 25808 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.00 CP Server Worke 25809 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.00 CP Server Worke 25810 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.00 CP Server Worke 25811 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.00 CP Server Worke 25831 root 20 0 3085980 407988 10016 S 0,0 0,2 0:09.61 Monitor-HostMon 25832 root 20 0 3085980 407988 10016 S 0,0 0,2 0:27.20 DnsResolutionMo 25834 root 20 0 3085980 407988 10016 S 0,0 0,2 2:49.89 MonitorDaemon-S 26279 root 20 0 3085980 407988 10016 S 0,0 0,2 3:02.54 WorkerThread 26629 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.36 __run_queue 26630 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.39 __run_queue 26631 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.46 __run_queue 26632 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.80 __run_queue 26637 root 20 0 3085980 407988 10016 S 0,0 0,2 4:24.79 GM KUDU_TSERVER 26639 root 20 0 3085980 407988 10016 S 0,0 0,2 0:02.74 Monitor-SolrSer 26641 root 20 0 3085980 407988 10016 S 0,0 0,2 0:24.39 GM KAFKA_BROKER 26647 root 20 0 3085980 407988 10016 S 0,0 0,2 0:02.14 GM REGIONSERVER 26649 root 20 0 3085980 407988 10016 S 0,0 0,2 0:08.91 GM NODEMANAGER 26651 root 20 0 3085980 407988 10016 S 0,0 0,2 0:00.31 GM OZONE_DATANO 26653 root 20 0 3085980 407988 10016 S 0,0 0,2 0:08.82 GM DATANODE 26656 root 20 0 3085980 407988 10016 S 0,0 0,2 0:06.44 GM IMPALAD 26657 root 20 0 3085980 407988 10016 R 0,0 0,2 21:01.25 ImpalaDaemonQue What are the next steps identifying the root cause of this issue ? (CDP 7.1.6) Thanks in advance for your help.   ... View more

Hi my backup is failing with following errors : 23/11/30 10:08:58 ERROR backup.KuduBackup$: Failed to back up table impala::PPR_CURATED.ACTIVITE org.apache.kudu.client.NonRecoverableException: Unauthorized action at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:470) at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:288) at org.apache.kudu.backup.KuduBackup$.org$apache$kudu$backup$KuduBackup$$doBackup(KuduBackup.scala:50) at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply$mcV$sp(KuduBackup.scala:144) at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply(KuduBackup.scala:144) at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply(KuduBackup.scala:144) at scala.util.Try$.apply(Try.scala:192) at org.apache.kudu.backup.KuduBackup$$anonfun$6.apply(KuduBackup.scala:144) at org.apache.kudu.backup.KuduBackup$$anonfun$6.apply(KuduBackup.scala:143) at scala.collection.parallel.AugmentedIterableIterator$class.map2combiner(RemainsIterator.scala:115) at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:62) at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1054) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48) at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51) at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1051) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443) at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341) at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673) at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:378) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:443) at scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:426) at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56) at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:958) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48) at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48) at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51) at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:953) at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152) at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443) at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace at org.apache.kudu.client.RpcProxy.dispatchMasterError(RpcProxy.java:411) at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:288) at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) at org.apache.kudu.client.Connection.channelRead0(Connection.java:357) at org.apache.kudu.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.kudu.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526) at org.apache.kudu.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275) at org.apache.kudu.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.kudu.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.kudu.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.kudu.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at org.apache.kudu.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at org.apache.kudu.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 23/11/30 10:08:58 ERROR backup.KuduBackup$: Failed to back up table impala::PPR_CURATED.ACTIVITE: Look back in the logs for the full exception. Error: Failure(org.apache.kudu.client.NonRecoverableException: Unauthorized action) Exception in thread "main" java.lang.RuntimeException: Kudu Table Backup application failed at org.apache.kudu.backup.KuduBackup$.main(KuduBackup.scala:178) at org.apache.kudu.backup.KuduBackup.main(KuduBackup.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:847) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Thanks in advance for your help, Best Regards ... View more