Reply
New Contributor
Posts: 3
Registered: ‎04-14-2015

Connect hivemetastore in MapReduce job

I am trying to run a Map Reduce job on CDH 5.3 which internally uses HiveMetaStoreCleint. The HiveMetaStore is kerberos security enabled. The HiveConf has the required properties set:

hive.metastore.sasl.enabled
hive.metastore.kerberos.principal

I am not able to create Instance of HiveMetaStoreClient. I am getting Exception "GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)"

Any guidance in this direction?

 

Regards,

 

Mahesh

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Connect hivemetastore in MapReduce job

The HMS has its own set of authentication requirements. Your solution should aim towards obtaining a token at the client (Driver code) and then making use of the token in the remote parts (Map/Reduce code). This is how Oozie also does it, via its HCatCredentials support: https://github.com/cloudera/oozie/blob/cdh5.3.1-release/core/src/main/java/org/apache/oozie/action/h...

Take a look at the above code, its very close to what you will need to implement for your Driver (injecting the token into the Job's configuration object before submission), and the second step is to merely ensure your HiveMetaStoreClient object uses the context.getConfiguration() during its construction, so it receives the passed token and does not require a kerberos auth.
New Contributor
Posts: 3
Registered: ‎04-14-2015

Re: Connect hivemetastore in MapReduce job

Thanks Harsh J.

 

I will try this out.

 

 

New Contributor
Posts: 3
Registered: ‎04-14-2015

Re: Connect hivemetastore in MapReduce job

I have one more question regarding hive metastore connection. Same as map reduce can we connect metastore in Hive UDF? What things needed to connect to metastore in hive UDF?

Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Connect hivemetastore in MapReduce job

You can, as long as you use the shared job configuration object from
the Map/Reduce context object which carries the token required to
perform the authentication. The same instructions as before applies.

Announcements