Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Connection to the Hive metastore from a Spark Job (on a Kerberos environment)

Highlighted

Connection to the Hive metastore from a Spark Job (on a Kerberos environment)

New Contributor

Hello,

I have to create and write into hive tables executed from a spark job. I instantiate an HiveContext and its configuration with the following code:

val sparkConf = new SparkConf(true) 
implicit val sc = new SparkContext(sparkConf) 
implicit val sqlContext = new HiveContext(sc) 
sqlContext.setConf("hive.exec.dynamic.partition", "true")
sqlContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
sqlContext.setConf("hive.metastore.uris", "thrift://xxxxxxx:9083")

The program stops at the creation of the table which is the line:

sqlContext.sql("CREATE EXTERNAL TABLE IF NOT EXISTS...")

On a non Kerberised environment the code works but since we have now a new environment which is kerberised it doesn't work anymore.

Here is the error stack trace:

17/12/06 16:29:21 ERROR Driver: FAILED: IllegalStateException Unxpected Exception thrown: Unable to fetch table xxxx. null
java.lang.IllegalStateException: Unxpected Exception thrown: Unable to fetch table xxxx. null
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:10896)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:10047)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10128)
	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:209)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)

We have also noticed this warn message:

17/12/06 16:29:20 WARN metastore: set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3688)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3674)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:436)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1115)

Anyone can help me please ?

2 REPLIES 2

Re: Connection to the Hive metastore from a Spark Job (on a Kerberos environment)

Expert Contributor
@Joffrey C

17/12/0616:29:20 WARN metastore: set_ugi()not successful,Likely cause:new client talking to old server.Continuing without it.

This mostly happens when spark is using the wrong hive-site.xml file, if you notice /etc/spark/conf will have a separate hive-site.xml file which is not same as /etc/hive/conf/hive-site.xml , if you done the upgrade and replaced /etc/spark/conf/hive-site.xml with /etc/hive/conf/hive-site.xml then these kind of issues occur.

Highlighted

Re: Connection to the Hive metastore from a Spark Job (on a Kerberos environment)

New Contributor

Thanks you.

But it's not working :(, do you have any idea why or another solution ?

Don't have an account?
Coming from Hortonworks? Activate your account here