KUDU backup not working on kerberized cluster

Question

Himy backup is failing with following errors :23/11/30 10:08:58 ERROR backup.KuduBackup$: Failed to back up table impala::PPR_CURATED.ACTIVITEorg.apache.kudu.client.NonRecoverableException: Unauthorized actionat org.apache.kudu.client.KuduException.transformException(KuduException.java:110)at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:470)at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:288)at org.apache.kudu.backup.KuduBackup$.org$apache$kudu$backup$KuduBackup$$doBackup(KuduBackup.scala:50)at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply$mcV$sp(KuduBackup.scala:144)at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply(KuduBackup.scala:144)at org.apache.kudu.backup.KuduBackup$$anonfun$6$$anonfun$1.apply(KuduBackup.scala:144)at scala.util.Try$.apply(Try.scala:192)at org.apache.kudu.backup.KuduBackup$$anonfun$6.apply(KuduBackup.scala:144)at org.apache.kudu.backup.KuduBackup$$anonfun$6.apply(KuduBackup.scala:143)at scala.collection.parallel.AugmentedIterableIterator$class.map2combiner(RemainsIterator.scala:115)at scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:62)at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1054)at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1051)at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152)at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341)at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673)at scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:378)at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:443)at scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:426)at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:56)at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:958)at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:953)at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:152)at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack traceat org.apache.kudu.client.RpcProxy.dispatchMasterError(RpcProxy.java:411)at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:288)at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)at org.apache.kudu.client.Connection.channelRead0(Connection.java:357)at org.apache.kudu.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)at org.apache.kudu.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)at org.apache.kudu.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)at org.apache.kudu.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)at org.apache.kudu.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)at org.apache.kudu.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at org.apache.kudu.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at org.apache.kudu.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)at org.apache.kudu.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)at org.apache.kudu.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)at org.apache.kudu.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)at org.apache.kudu.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)23/11/30 10:08:58 ERROR backup.KuduBackup$: Failed to back up table impala::PPR_CURATED.ACTIVITE: Look back in the logs for the full exception. Error: Failure(org.apache.kudu.client.NonRecoverableException: Unauthorized action)Exception in thread "main" java.lang.RuntimeException: Kudu Table Backup application failedat org.apache.kudu.backup.KuduBackup$.main(KuduBackup.scala:178)at org.apache.kudu.backup.KuduBackup.main(KuduBackup.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:847)at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Thanks in advance for your help,Best Regards

jromero · Answer

Hello Olivier,The error you got "org.apache.kudu.client.NonRecoverableException: Unauthorized action" happens when the account you use to take the backup don't have the proper permissions to access the kudu tables.Do you happen to use ranger to authorize access on kudu?  If so, then maybe the account you are using is not authorized on ranger.  It could be as well that you are running it with the wrong kerberos principal?Regards,Jason R

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.