question Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled in Archives of Support Questions (Read Only)

Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

terance — Fri, 04 Oct 2019 21:34:15 GMT

I'm using CDH5. I have set up a Hive Metastore to use SASL. i.e. the hive-site.xml has the following properties

<property>
  <name>hive.metastore.sasl.enabled</name>
  <value>true</value>
</property>

<property>
  <name>hive.metastore.kerberos.keytab.file</name>
  <value>/etc/hive/conf/hive.keytab</value>
</property>

<property>
  <name>hive.metastore.kerberos.principal</name>
  <value>hive/hive-metastore.xxxxxxx.com@XXXXXXX.COM</value>
</property>

The logs show no errors on starting the hive-metastore service.

I'm trying to run a hive action in an oozie workflow. The oozie-site.xml file has the following property

    <property>
        <name>oozie.credentials.credentialclasses</name>
        <value>hcat=org.apache.oozie.action.hadoop.HCatCredentials</value>
    </property>

And the workflow xml file has the credentials tag

    <credentials>
        <credential name='hive_credentials' type='hcat'>
               <property>
                    <name>hcat.metastore.uri</name>
                    <value>thrift://hive-metastore.xxxxxxx.com:9083</value>
               </property>
               <property>
                    <name>hcat.metastore.principal</name>
                    <value>hive/hadoop-metastore.xxxxxxx.com@XXXXXXX.COM</value>
               </property>
         </credential>
    </credentials>

The hive action refers to the credentials using the 'cred' attribute.

    <action name="hive" cred="hive_credentials">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <job-xml>${appPath}/hive-site.xml</job-xml>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <script>${appPath}/queries.hql</script>
        </hive>
        <ok to="pass"/>
        <error to="fail"/>
    </action>

When I try to run this workflow, I get the following error.

Exception in addtoJobConf
MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: No common protection layer between client and server
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:288)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:109)
        at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:87)
        at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:52)
        at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:58)
        at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:990)
        at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:851)
        at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1071)
        at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:217)
        at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:62)
        at org.apache.oozie.command.XCommand.call(XCommand.java:280)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:334)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:169)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:109)
        at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:87)
        at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:52)
        at org.apache.oozie.action.hadoop.HCatCredentials.addtoJobConf(HCatCredentials.java:58)

It looks like oozie is not able to connect to the hive metastore using its hive client.

When I start the hive-server2 and try to connect using beeline, I get the same exception message. (i.e. javax.security.sasl.SaslException: No common protection layer between client and server)

I'm able to connect to the metastore using hive cli.

Any idea what could be causing this issue?

Thanks,

Terance.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Sun, 20 Jul 2014 07:45:26 GMT

Where exactly is the hive.metastore.sasl.enabled property applied? Are you certain it is applied to the running HiveMetaStore server?

Does a regular Hive CLI configured with hive.metastore.uris instead of DB properties run properly (i.e. show tables, etc. work fine)?

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

terance — Thu, 31 Jul 2014 07:38:16 GMT

The hive.metastore.sasl.enabled is applied to hive-site.xml in /etc/hive/conf/hive-site.xml. The same hive-site.xml is also present in the oozie app directory and is referred to in workflow.xml as

<job-xml>${appPath}/hive-site.xml</job-xml>

If the hive-site.xml has hive.metastore.uris property, i think the Hive CLI will use the metastore uri and not connect to DB directly. When this property is present in hive-site.xml along with hive.metastore.sasl.enabled, hive.metastore.kerberos.keytab.file and hive.metastore.kerberos.principal, I'm able to use Hive CLI and run show tables, etc. So I guess Hive CLI is able to talk to metastore, but oozie is not.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Thu, 31 Jul 2014 11:47:11 GMT

To have Oozie talk to a secured HiveMetaStore, you need to follow the
credentials procedure detailed at
http://archive.cloudera.com/cdh5/cdh/5/oozie/DG_UnifiedCredentialsModule.html

Basically:
1. Enable Credentials of HCat type at the Oozie Server (via service
configuration, requires restart).
2. Add a credentials section to your workflow. This section also
configures the HMS location and the SPN.
3. Add the credential name to the action that requires a token for
authentication (your hive action).

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

terance — Thu, 31 Jul 2014 12:07:22 GMT

Thanks Harsh, I've done these these. Please see my first post.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Thu, 31 Jul 2014 14:30:55 GMT

Ugh, very sorry. I replied the above post via email, thinking it was a wholly new question.

OK, so looking further in, the Oozie server is already using a TSaslTransport for the HMS client connection, so the property is likely not the problem.

Do you perhaps have "hadoop.rpc.protection", on your cluster, set to a non-default value of "privacy" (for traffic encryption in HDFS and the like)?

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

terance — Fri, 01 Aug 2014 06:18:26 GMT

Yes Harsh, the 'hadoop.rpc.protection' property is set to 'privacy'. Does this property affect communication between hive metastore and its client?

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Fri, 01 Aug 2014 16:48:43 GMT

It does lead to this error from another issue we've seen internally
(with CM canaries over HMS). Does adding the hadoop.rpc.protection
property pair (with "privacy" option set), to your passed
config file in the WF help with getting the HMS connection through?

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

terance — Mon, 04 Aug 2014 05:46:46 GMT

You mean add the 'hadoop.rpc.protection' property with value 'privacy' in the hive-site.xml file in that is used in the workflow xml? I tried this and I'm still getting the same exception.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Mon, 04 Aug 2014 06:05:43 GMT

Thank you for trying that out! I investigated further into the code
and it turns out you are hitting
https://issues.apache.org/jira/browse/OOZIE-1593, which we have
backported in our CDH 5.1.0 release. With that fix added, the
Credentials code will properly load the hadoop.rpc.protection when
making HMS connections. But it does not appear do so in prior
releases, even if you were to specify it as part of your action
configuration.

This is the fixed code line, if you are interested in taking a look:
https://github.com/cloudera/oozie/blob/cdh5.1.0-release/core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java#L92
(Note the missing line in 5.0.0 sources, at
https://github.com/cloudera/oozie/blob/cdh5.0.0-release/core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java#L86)

If you are unable to upgrade immediately, you can perhaps try
something like the below as one way of workaround for this:

# SSH to Oozie machine
# Log on as root
# Do below:
cat > /tmp/hive-site.xml
hadoop.rpc.protectionprivacy
^D
cd /tmp/
jar cf hive-rpc-protection.jar hive-site.xml
mv hive-rpc-protection.jar /var/lib/oozie/
# Restart Oozie server, and retry your WF.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Mon, 04 Aug 2014 06:43:36 GMT

Sorry, the XML appears to have got eaten up. Here's clearer instructions for releases prior to CDH 5.1.0:

# SSH to Oozie machine
# Log on as root
# Do below:
cat > /tmp/hive-site.xml
<configuration><property><name>hadoop.rpc.protection</name><value>privacy</value></property></configuration>
^D
cd /tmp/
jar cf hive-rpc-protection.jar hive-site.xml
mv hive-rpc-protection.jar /var/lib/oozie/
# Restart Oozie server, and retry your WF.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

terance — Tue, 05 Aug 2014 10:29:40 GMT

Thanks Harsh, that worked.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

suryajayanthi — Thu, 09 Aug 2018 14:06:39 GMT

Hi,

Are there any instructions for CDH 5.13.1 ?

I have a similar error message for the same config.

Hadoop RPC Protection: Authenticate

Error Message:

WARN org.apache.oozie.action.hadoop.HCatCredentials: SERVER[myhost] USER[ABC] Group[ABC] APP[ABCDEFGHIJKL] JOB[JOBID] Action[MyAction] Exception in addtoJobConf
org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : TException while getting delegation token.. Cause : org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Abdul — Fri, 04 Oct 2019 21:03:56 GMT

@Harsh J

I'm getting the same kind of issue/error while running the spark Job.

"JA009 JA009: org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : TException while getting delegation token.. Cause : org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out "

We are currently using CDH Version and CManager Version (5.13.3). So can you Please suggest me whether the procedure you provided in Version 5.1.1* above can be applied for 5.13 .

Appreciate your response.

Thanks

Mannan.

Re: Oozie/hive-server2 not able to connect to hive-metastore when SASL enabled

Harsh J — Sat, 05 Oct 2019 07:43:15 GMT

The original issue described here is not applicable to your version. In
your case it could simply be a misconfiguration that's causing oozie to not
load the right hive configuration required to talk to the hive service. Try
enabling debug logging on the oozie server if you are unable to find an
error in it. Also try to locate files or jars in your workflow that may be
supplying an invalid hive client XML.