Hi, We are getting the error while executing Hive query on spark as execution engine. Hive version: 1.2.1, Spark version : 1.6   set hive.execution.engine=spark; set spark.home=/usr/hdp/current/spark-client; set hive.execution.engine=spark; set spark.master=yarn-client; set spark.eventLog.enabled=true; set spark.executor.memory=512m; set spark.executor.cores=2; set spark.driver.extraClassPath=/usr/hdp/current/hive-client/lib/hive-exec.jar;   Query ID = svchdpir2d_20191106105445_a9ebc8a2-9c28-4a3d-ac5e-0a8609e56fd5 Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Spark Job = c6cc1641-20ad-4073-ab62-4f621ae595c8 Status: SENT Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.   Could you please help on this. Thank you ... View more

We have installed Spark2 in HDP 3.1 but when we are trying to access spark2 history server UI,we are getting below issue.   HTTP ERROR 403 Problem accessing /. Reason: java.lang.IllegalArgumentException   log: spark-spark-org.apache.spark.deploy.history.HistoryServer-1-xxxxxx.visa.com.out   19/08/28 13:01:21 DEBUG AuthenticationFilter: Request [http://xxxxxxxx:18081/favicon.ico] triggering authentication. handler: class org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler 19/08/28 13:01:21 DEBUG AuthenticationFilter: Authentication exception: java.lang.IllegalArgumentException org.apache.hadoop.security.authentication.client.AuthenticationException: java.lang.IllegalArgumentException at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:306) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536) at org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:448) at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.spark_project.jetty.server.Server.handle(Server.java:539) at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:275) at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:365) at org.apache.hadoop.security.authentication.util.KerberosUtil$DER.<init>(KerberosUtil.java:358) at org.apache.hadoop.security.authentication.util.KerberosUtil.getTokenServerName(KerberosUtil.java:291) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:285) ... 22 more HDP - HDP 3.1 Spark 2.3 Kerberized cluster   Could you please help on this. Thank you ... View more

Hi, In the kerberized cluster, after enabling Ranger Hive plug-in in HDP 2.6.5, not able to run dfs commands in beeline. jdbc:hive2://test.xyz.com:10000> dfs -ls /; Error: Error while processing statement: Permission denied: user [user1] does not have privilege for [DFS] command (state=,code=1) Please help on this. Thank you. ... View more

Hi, We are getting below warnings while starting hive shell from CLI. $hive 19/06/20 07:53:14 WARN conf.HiveConf: HiveConf of name hive.mapred.strict does not exist 19/06/20 07:53:14 WARN conf.HiveConf: HiveConf of name hive.mapred.supports.subdirectories does not exist log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender hive> Hive version - 1.2 As per I know, we can safely ignore these warnings but, we need to get rid off these. Could you please help on this. Thanks ... View more