Created 08-02-2017 08:14 AM
I wanted to access hive table from servlet. when I run the url I get
javax.servlet.ServletException:Servlet execution threw an exception
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)RootCause
java.lang.NoClassDefFoundError:Couldnot initialize class org.apache.hadoop.security.UserGroupInformation
org.apache.spark.util.Utils$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
org.apache.spark.util.Utils$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
scala.Option.getOrElse(Option.scala:121)
org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)
org.apache.spark.SparkContext.<init>(SparkContext.scala:295)
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
org.apache.spark.sql.SparkSession$Builder$anonfun$6.apply(SparkSession.scala:868)
org.apache.spark.sql.SparkSession$Builder$anonfun$6.apply(SparkSession.scala:860)
scala.Option.getOrElse(Option.scala:121)
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)HiveRead.doGet(HiveRead.java:30)
javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
I am running this code
SparkSession spark =SparkSession.builder().appName("Java Spark SQL basic example").enableHiveSupport().config("spark.sql.warehouse.dir","hdfs://saurab:9000/user/hive/warehouse").config("mapred.input.dir.recursive",true).config("hive.mapred.supports.subdirectories",true).config("hive.vectorized.execution.enabled",true).master("local").getOrCreate();
response.getWriter().println("olo");
I looked at this question and just to test added
export SPARK_CLASSPATH=$CLASS_PATH:/home/saurab/hadoopec/spark/jars/hadoop-auth-2.7.3
since I was not using shark Though my aim is to access hive table and run sql using Spark, I am running this code to test. This code has nothing to do with hive table but If I run only this much code I get above exception. I guess it has something to do with authentication, but I can't figure out what. this is my pom.xml IS there a better way to access hive tables and run queries using servlets?
Created 08-02-2017 08:33 AM
The missing class "org.apache.hadoop.security.UserGroupInformation" can be found insode the "hadoop-commons" so please try adding that jar inside your servlet classpath. (if its a web application then add it inside the "WEB-INF/lib" of your project.
Example JAR:
/usr/hdp/2.6.1.0-129/hadoop/hadoop-common.jar /usr/hdp/2.6.1.0-129/hadoop/hadoop-common-2.7.3.2.6.1.0-129.jar
.
Created on 08-02-2017 08:44 AM - edited 08-17-2019 10:12 PM
@Jay SenSharma The hadoop-common jar is present in "WEB-INF/lib" . Still I am getting this error