Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SparkSql and servlet : org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName

avatar
Contributor

I wanted to access hive table from servlet. when I run the url I get

javax.servlet.ServletException:Servlet execution threw an exception
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)RootCause
java.lang.NoClassDefFoundError:Couldnot initialize class org.apache.hadoop.security.UserGroupInformation
    org.apache.spark.util.Utils$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
    org.apache.spark.util.Utils$anonfun$getCurrentUserName$1.apply(Utils.scala:2391)
    scala.Option.getOrElse(Option.scala:121)
    org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391)
    org.apache.spark.SparkContext.<init>(SparkContext.scala:295)
    org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320)
    org.apache.spark.sql.SparkSession$Builder$anonfun$6.apply(SparkSession.scala:868)
    org.apache.spark.sql.SparkSession$Builder$anonfun$6.apply(SparkSession.scala:860)
    scala.Option.getOrElse(Option.scala:121)
    org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)HiveRead.doGet(HiveRead.java:30)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

I am running this code

SparkSession spark =SparkSession.builder().appName("Java Spark SQL basic example").enableHiveSupport().config("spark.sql.warehouse.dir","hdfs://saurab:9000/user/hive/warehouse").config("mapred.input.dir.recursive",true).config("hive.mapred.supports.subdirectories",true).config("hive.vectorized.execution.enabled",true).master("local").getOrCreate();
response.getWriter().println("olo");

I looked at this question and just to test added

export SPARK_CLASSPATH=$CLASS_PATH:/home/saurab/hadoopec/spark/jars/hadoop-auth-2.7.3

since I was not using shark Though my aim is to access hive table and run sql using Spark, I am running this code to test. This code has nothing to do with hive table but If I run only this much code I get above exception. I guess it has something to do with authentication, but I can't figure out what. this is my pom.xml IS there a better way to access hive tables and run queries using servlets?

2 REPLIES 2

avatar
Master Mentor

@Saurab Dahal

The missing class "org.apache.hadoop.security.UserGroupInformation" can be found insode the "hadoop-commons" so please try adding that jar inside your servlet classpath. (if its a web application then add it inside the "WEB-INF/lib" of your project.

Example JAR:

/usr/hdp/2.6.1.0-129/hadoop/hadoop-common.jar
/usr/hdp/2.6.1.0-129/hadoop/hadoop-common-2.7.3.2.6.1.0-129.jar

.

avatar
Contributor

@Jay SenSharma The hadoop-common jar is present in "WEB-INF/lib" . Still I am getting this error

23391-screenshot-from-2017-08-02-14-26-46.png