question Access to Hive from Oozie java action with Kerberos in Support Questions

Access to Hive from Oozie java action with Kerberos

pavel_benes — Thu, 07 Jan 2016 05:12:43 GMT

Hi,

I am trying to execute Hive query from java action that is part of Oozie workflow. My preferred way is to use the Beeline or JBDC rather than old Hive CLI. However I am struggling with this a little bit, since the connection is failing due to authentication errors. When the java action code is executed on some data node in a cluster, Oozie does not do kinit for the user and thus the connection fails. Both Beeline and JDBC connection string seem to support delegation tokens, but those token can only be obtained when user is logged in by Kerberos (kinited).

We are currently using hive-0.14.0 and oozie-4.1.

I have found out that new hive2 action introduced in oozie-4.2 seems to first create jdbc connection under Oozie Kerberos login, obtain delegation token from this connection and finally pass this token to the Beeline. Maybe the same approach could be used here as well. It would require a new custom oozie action (e.g. java-jdbc). . Seems possible but it is quite complicated; is there some easier way?

Thanks for any comments,

Pavel

Re: Access to Hive from Oozie java action with Kerberos

aervits — Thu, 07 Jan 2016 11:35:03 GMT

@Pavel Benes more of a hack but maybe you can try using a shell action to call your java command, then capture output of the action?

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Thu, 07 Jan 2016 16:34:06 GMT

@Artem Ervits Thanks for reply. In my opinion it would not help. The shell action is the same as java one with respect to kerberos login, so the delegation token is still required to connect from JDBC. The only way I see is to do the initial JDBC connection within Oozie action handler/executor that is executed under kinit and pass the delegation token to the actual java action code running on a datanode. But maybe I miss something.

Re: Access to Hive from Oozie java action with Kerberos

aervits — Thu, 07 Jan 2016 20:20:43 GMT

This calls for an expert, @bsaini

Re: Access to Hive from Oozie java action with Kerberos

bleonhardi — Thu, 07 Jan 2016 22:05:26 GMT

You could use a shell action, add the token to the oozie files ( file tag ) and do the kinit yourself before running the java command. Obviously not that elegant and you have a token somewhere in HDFS but it should work. I did something similar with a shell action running a scala program and running a kinit before. ( Not against hive but running kinit then connecting to HDFS ). Ceterum censeo I would always suggest using a hive server with LDAP/PAM authentication. beeline and hive2 action has a password file option now and it makes life so much easier. As a database guy kerberos for a jdbc connection just always makes problems.

Here is the oozie shell command by the way.

<shell xmlns="uri:oozie:shell-action:0.1">
   <job-tracker>${jobTracker}</job-tracker>
   <name-node>${nameNode}</name-node>
   <exec>runJavaCommand.sh</exec>
   <file>${nameNode}/scripts/runJavaCommand.sh#runJavaCommand.sh</file>
   <file>${nameNode}/securelocation/user.keytab#user.keytab</file>
</shell>

then just add a kinit into the script before running java
kinit -kt user.keytab user@EXAMPLE.COM
java org.apache.myprogram

Re: Access to Hive from Oozie java action with Kerberos

bsaini — Thu, 07 Jan 2016 22:50:20 GMT

@Pavel Benes - Can you modify the source of the program that gets executed Java Action? If so, you can include the kinit as part of the java code.

See below link -

https://community.hortonworks.com/questions/1807/c...

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Fri, 08 Jan 2016 03:47:56 GMT

@bsaini I can modify the action code, but I cannot do the kinit there since I do not have access to user's keytab at all. My scenario is like this:

the user is logged in company network (with Kerberos)
the user access the REST API of some application server (authenticated using Kerberos)
the application server runs Oozie workflow, that includes the java task that needs to access some tables in Hive using the original user credentials.

The only way I see is the delegation token. Even if Oozie would support kinit on data, it still is no help, since the keytab/password is not available.

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Fri, 08 Jan 2016 04:00:44 GMT

@Benjamin Leonhardi Thanks for reply. However as I tried to describe above, I cannot do kinit for the user since I do not have access the his keytab at all. I am not sure, maybe in theory I could do kinit as some service user with ability to impersonate users on Haddop (e,g. like oozie), and using doAs() to get access to hive or obtain delegation token. I am not a Kerberos expert, but it still feels like a security hole to allow access to this keytab for normal users.

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Fri, 08 Jan 2016 05:40:23 GMT

@Benjamin Leonhardi Regarding the LDAP/PAM scenario you mention, I am not familiar with details, but I am afraid that our users are expecting a single sing on, so they wont be willing to enter their intranet password again to some "custom" system.

Re: Access to Hive from Oozie java action with Kerberos

konstt20000 — Wed, 17 Feb 2016 00:40:40 GMT

Did you get a solution?

Re: Access to Hive from Oozie java action with Kerberos

konstt20000 — Wed, 17 Feb 2016 00:42:44 GMT

Did you get a solution to your problem? What I don't understand it's.. is it necessary to do the kinit inside of the java code with UserGroupInformation and so on if you want to connect to Hive with Kerberos?

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Wed, 17 Feb 2016 04:10:39 GMT

@Guillermo Ortiz Not really, I have split the original java action into two Oozie actions; the first one is hive action where I get what I need from from hive (using temporary external table) and the second java actions where the data are further processed. Currently I use hive action, but it should be trivial to replace it with hive2 action in future when needed.

And yes, according to my knowledge it necessary to have valid kerberos token (kinit does not have to happen in java though) or use delegation token to connect to Kerberized hive from java.

Re: Access to Hive from Oozie java action with Kerberos

konstt20000 — Wed, 17 Feb 2016 23:14:35 GMT

is it just a problem with JDBC/Hive/Java Action and Oozie? If you want to make some query to HBase or whatever with Kerberos. Would you have the same problem?

Re: Access to Hive from Oozie java action with Kerberos

bleonhardi — Fri, 19 Feb 2016 21:10:55 GMT

@Pavel Benes

I know it has been forever. But its possible to provide a password file to beeline. So they might not have to log on. Similar to a keytab the file would be stored in their account.

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Mon, 22 Feb 2016 15:07:04 GMT

The problem is that our system does not have access to users password or keytab. It uses kerberos authentication and than Haddop proxy user to access various Hadoop services. So it is not possible for us to do kinit again on a data node or use password (in file or directly).

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Mon, 22 Feb 2016 15:12:02 GMT

@Guillermo Ortiz I would say it is Oozie/Kerberos problem. If I would like to call HBase from Oozie (there is probably not a task for it), I would end up with the same problem.

Re: Access to Hive from Oozie java action with Kerberos

ashish_gupta1 — Mon, 21 Mar 2016 17:11:35 GMT

I am also trying to execute hive query through oozie java action on kerberized environment (https://community.hortonworks.com/questions/23857/executing-hive-queries-through-oozie-java-action-o.html). I tried above solution, but still I am facing issue.

Re: Access to Hive from Oozie java action with Kerberos

padmanabhan5789 — Mon, 17 Oct 2016 14:27:46 GMT

@Pavel Benes Did you get the solution. I am facing the similar issue. My java application writes the result to HDFS and it needs kerberos authentication. When schedule my application using oozie. I am facing this issue.

Re: Access to Hive from Oozie java action with Kerberos

pavel_benes — Mon, 17 Oct 2016 15:09:47 GMT

@Padmanabhan Vijendran

Actually I did not, since the need passed. However my question was more about access to Hive. In case of HDFS it should be more simple.

In your java code you need to have something like this:

if (System.getenv("HADOOP_TOKEN_FILE_LOCATION") != null) {
            jobConf.set("mapreduce.job.credentials.binary", System.getenv("HADOOP_TOKEN_FILE_LOCATION"));
}

to tell your java app where to find delegation token needed for HDFS access.

Hope this helps,

Pavel

Re: Access to Hive from Oozie java action with Kerberos

padmanabhan5789 — Thu, 01 Dec 2016 14:31:37 GMT

@Pavel Benes

Did you fixed the hive issues in Java Action from oozie ?