Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Connect hivemetastore in MapReduce job

Connect hivemetastore in MapReduce job

New Contributor

I am trying to run a Map Reduce job on CDH 5.3 which internally uses HiveMetaStoreCleint. The HiveMetaStore is kerberos security enabled. The HiveConf has the required properties set:

hive.metastore.sasl.enabled
hive.metastore.kerberos.principal

I am not able to create Instance of HiveMetaStoreClient. I am getting Exception "GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)"

Any guidance in this direction?

 

Regards,

 

Mahesh

4 REPLIES 4

Re: Connect hivemetastore in MapReduce job

Master Guru
The HMS has its own set of authentication requirements. Your solution should aim towards obtaining a token at the client (Driver code) and then making use of the token in the remote parts (Map/Reduce code). This is how Oozie also does it, via its HCatCredentials support: https://github.com/cloudera/oozie/blob/cdh5.3.1-release/core/src/main/java/org/apache/oozie/action/h...

Take a look at the above code, its very close to what you will need to implement for your Driver (injecting the token into the Job's configuration object before submission), and the second step is to merely ensure your HiveMetaStoreClient object uses the context.getConfiguration() during its construction, so it receives the passed token and does not require a kerberos auth.

Re: Connect hivemetastore in MapReduce job

New Contributor

Thanks Harsh J.

 

I will try this out.

 

 

Re: Connect hivemetastore in MapReduce job

New Contributor

I have one more question regarding hive metastore connection. Same as map reduce can we connect metastore in Hive UDF? What things needed to connect to metastore in hive UDF?

Highlighted

Re: Connect hivemetastore in MapReduce job

Master Guru
You can, as long as you use the shared job configuration object from
the Map/Reduce context object which carries the token required to
perform the authentication. The same instructions as before applies.