Support Questions
Find answers, ask questions, and share your expertise

Kerberos Error- Accessing HIVE JDBC from Spark Application

Kerberos Error- Accessing HIVE JDBC from Spark Application

We need run Create table and alter table statements to hive from within a spark streaming application. The spark version is 2.1.X and it is HDP 2.6.2.

spark.sqlContext.sql("CREATE ...")spark.sqlContext.sql("ALTER ...")

The create statement works but alter fails with Spark error "Operation not allowed". As a work around we thought why cant we use hive jdbc to issue alter statements if spark is not allowing it. But the main problem is with hive jdbc fails with kerberos authenticaiton when tried from within a spark application. The same program ( hive -jdbc ) works with kerberos when tried as a standalone java applicaiton. What is the way to supply kerberos credentials to hive jdbc when invoked inside a spark applicaiton.

Class.forName("org.apache.hive.jdbc.HiveDriver")
  val conf: Configuration = new org.apache.hadoop.conf.Configuration();
  conf.set("hadoop.security.authentication", "Kerberos");
  UserGroupInformation.setConfiguration(conf);
  UserGroupInformation.loginUserFromKeytab("user1@Domain, "hdfs://user/user1/user1.keytab");
  println(s"********************** before connection")
  val conn=DriverManager.getConnection("jdbc:hive2://domain:port/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_HOST@Domain")
  println(s"********************** ${conn}")

This code works when run as a standalone java program but fails with kerberos error when called within a spark applicaiton. It gives error javax.security.auth.login.LoginException: Unable to obtain password from user.

Can you please help in this working from inside a spark application. Why are going this approach because spark 2.1 is not supporting alter tables.