Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

HDP 2.6. Kerberos enabled.

Try run HiveServer2 job using oozie like in this article -

https://community.hortonworks.com/articles/75107/oozie-hive-2-action-in-a-kerberized-cluster.html

job.properties

nameNode=hdfs://NN2:8020
jobTracker=NN1:8050
exampleDir=${nameNode}/user/${user.name}/ooziedemo/hivedem
ooozie.wf.application.path=${exampleDir}/app
oozie.use.system.libpath=true
# Hive2 action
hivescript=${oozie.wf.application.path}/add_row.sql
outputHiveDatabase=dm
jdbcURL=jdbc:hive2://NN2:10000/dm
jdbcPrincipal=hive/_HOST@REALM

workflow.xml

<workflow-app name="hivedemo" xmlns="uri:oozie:workflow:0.4">
  <global>
    <job-tracker>${jobTracker}</job-tracker>
    <name-node>${nameNode}</name-node>
  </global>
  <credentials>
    <credential name="hs2-creds" type="hive2">
      <property>
        <name>hive2.jdbc.url</name>
        <value>${jdbcURL}</value>
      </property>
      <property>
       <name>hive2.server.principal</name>
       <value>${jdbcPrincipal}</value>
      </property>
    </credential>
  </credentials>
  <start to="hive2"/>
    <action name="hive2" cred="hs2-creds">
      <hive2 xmlns="uri:oozie:hive2-action:0.1">
        <jdbc-url>${jdbcURL}</jdbc-url>
        <script>${hivescript}</script>
      </hive2>
      <ok to="End"/>
      <error to="Kill"/>
    </action>
    <kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  </kill>
  <end name="End"/>
</workflow-app>

Run oozie job under oozie user with kinit under oozie

In oozie console I got error : HiveSQLException: Error retrieving delegation token for user oozie

What I do wrong?

14 REPLIES 14

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

This is unbelievable! it seems like no-one uses oozie hive2 action against the kerberized cluster.

Or the issue is so trivial that no-one bothers to offer any clue how to solve it.

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

Mentor

@John Doe

I have just tried out the article you mentioned above successfully.

My cluster is a kerberized

1 node cluster 16GB RAM
HDP.2.6.1.0
Ambari 2.5.1.0

Here is the raw input, To authenticate with Kerberos I had to use the correct principal

login as: root
root@192.168.0.10's password:
Last login: Sat Oct  7 22:31:17 2017 from 192.168.0.178
[root@gulu ~]# su - oozie
Last login: Tue Oct 10 17:14:17 CEST 2017
[oozie@gulu ~]$ klist -kt /etc/security/keytabs/o
oozie.service.keytab  opera1.keytab
[oozie@gulu ~]$ klist -kt /etc/security/keytabs/oozie.service.keytab
Keytab name: FILE:/etc/security/keytabs/oozie.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM
   1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM
   1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM
   1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM
   1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM 

Initialize the kerberos ticket

[oozie@gulu ~]$ kinit -kt /etc/security/keytabs/oozie.service.keytab oozie/gulu.uganda.com@UGANDA.COM
[oozie@gulu ~]$ mkdir -p /tmp/crime 
[oozie@gulu ~]$ cd /tmp/crime 
[oozie@gulu crime]$ curl -o crime -L https://data.cityofchicago.org/api/views/ijzp-q8t2/rows.csv?accessType=DOWNLOAD 
% Total   % Received  % Xferd  Average     Speed Time     Time     Time    Current 
                                Dload     Upload Total    Spent    Left    Speed 
100 1447M 0 1447M     0       0 1788k     0 --:--:--      0:13:48 --:--:-- 1953k 

[oozie@gulu crime]$ ls -al 
total 1483276 
drwxr-xr-x 2 oozie hadoop 4096 Oct 10 17:16 . 
drwxrwxrwt. 43 root root 12288 Oct 10 17:35 .. 
-rw-r--r-- 1 oozie hadoop 1517368515 Oct 10 17:30 crime 

Directory /user/oozie already exists so no need to create one

[hdfs@gulu ~]$ klist -kt /etc/security/keytabs/hdfs.headless.keytab 
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab 
KVNO Timestamp Principal 
---- ------------------- ------------------------------------------------------ 
1 08/24/2017 15:42:23 hdfs-pomit@UGANDA.COM (aes256-cts-hmac-sha1-96)
.......
1 08/24/2017 15:42:23 hdfs-pomit@UGANDA.COM (des3-cbc-sha1)

Grabbed a valid Kerberos ticket

[hdfs@gulu ~] $ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-pomit@ UGANDA.COM 

Check oozie directories

 [hdfs@gulu ~]$ hdfs dfs -ls /user 
Found 9 items 
drwx------ - accumulo hdfs 0 2017-07-31 00:28 /user/accumulo 
drwxr-xr-x - admin hdfs 0 2017-08-26 23:56 /user/admin 
.....
drwxr-xr-x - hive hdfs 0 2017-08-27 00:20 /user/hive 
drwxrwxr-x - oozie hdfs 0 2017-07-31 00:35 /user/oozie 
drwxr-xr-x - opera1 hdfs 0 2017-10-05 21:37 /user/opera1 
drwxr-xr-x - zeppelin hdfs 0 2017-07-31 00:27 /user/zeppelin 
Switch to oozie user locate the correct principal and get a Kerberos ticket
[oozie@gulu crime]$ klist -kt /etc/security/keytabs/oozie.service.keytab 
Keytab name: FILE:/etc/security/keytabs/oozie.service.keytab 
KVNO Timestamp Principal 
---- ------------------- ------------------------------------------------------ 
1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM 
......
1 08/24/2017 15:42:24 oozie/gulu.uganda.com@UGANDA.COM 
[oozie@gulu crime]$ kinit -kt /etc/security/keytabs/oozie.service.keytab oozie/gulu.uganda.com@UGANDA.COM 

Copy the crime csv into HDFS

[oozie@gulu crime]$ hdfs dfs -copyFromLocal crime /user/oozie/ 
[oozie@gulu crime]$ hdfs dfs -ls /user/oozie 
Found 2 items 
-rw-r--r-- 3 oozie hdfs 1517368515 2017-10-10 17:39 /user/oozie/crime 
drwxr-xr-x - oozie hdfs 0 2017-07-31 00:35 /user/oozie/share 

Change permissions

[oozie@gulu crime]$ hdfs dfs -chmod -R 777 /user/oozie/crime 
[oozie@gulu crime]$ hdfs dfs -ls /user/oozie 
Found 2 items 
-rwxrwxrwx 3 oozie hdfs 1517368515 2017-10-10 17:39 /user/oozie/crime 
drwxr-xr-x - oozie hdfs 0 2017-07-31 00:35 /user/oozie/share

Switch to hive user to create the table in beeline,but first get a Kerberos ticket for hive

[root@gulu ~]# su - hive 
Last login: Mon Oct 2 16:28:51 CEST 2017 on pts/1 
[hive@gulu ~]$ klist -kt /etc/security/keytabs/hive.service.keytab 
Keytab name: FILE:/etc/security/keytabs/hive.service.keytab 
KVNO                Timestamp           Principal 
---- ------------------- ------------------------------------------------------ 
1 08/24/2017       15:42:24              hive/gulu.uganda.com@UGANDA.COM 
.............
1 08/24/2017       15:42:24             hive/gulu.uganda.com@UGANDA.COM 
 [hive@gulu ~]$ kinit -kt /etc/security/keytabs/hive.service.keytab hive/gulu.uganda.com@UGANDA.COM 

Create the HQL script I only changed the site-specific entries

[hive@gulu ~]$ vi load_crime_table.ddl 
[hive@gulu ~]$ ls -al total 56 -rw-r--r-- 1 hive hadoop 715 Oct 10 17:47 load_crime_table.ddl 

With a valid Kerbero ticket I connect to beeline

[hive@gulu ~]$ beeline -u "jdbc:hive2://gulu.uganda.com:10000/default;principal=hive/_HOST@UGANDA.COM" -f "/home/hive/load_crime_table.ddl" 
Connecting to jdbc:hive2://gulu.uganda.com:10000/default;principal=hive/_HOST@UGANDA.COM 
Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129) 
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129) 
Transaction isolation: TRANSACTION_REPEATABLE_READ 
0: jdbc:hive2://gulu.uganda.com:10000/default> CREATE EXTERNAL TABLE IF NOT EXISTS crime( 
0: jdbc:hive2://gulu.uganda.com:10000/default> ID STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Case_Number STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Case_Date STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Block STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> IUCR INT, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Primary_Type STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Description STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Location_Description STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Arrest BOOLEAN, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Domestic BOOLEAN, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Beat STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> District STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Ward STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Community_Area STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> FBI_Code STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> X_Coordinate INT, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Y_Coordinate INT, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Case_Year INT, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Updated_On STRING, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Latitude DOUBLE, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Longitude DOUBLE, 
0: jdbc:hive2://gulu.uganda.com:10000/default> Location STRING) 
0: jdbc:hive2://gulu.uganda.com:10000/default> COMMENT 'This is crime data for the city of Chicago.' 
0: jdbc:hive2://gulu.uganda.com:10000/default> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
0: jdbc:hive2://gulu.uganda.com:10000/default> STORED AS TEXTFILE 
0: jdbc:hive2://gulu.uganda.com:10000/default> LOCATION '/tmp/crime' 
0: jdbc:hive2://gulu.uganda.com:10000/default> TBLPROPERTIES("skip.header.line.count"="1"); 
No rows affected (32.509 seconds) 
0: jdbc:hive2://gulu.uganda.com:10000/default> 
Closing: 0: jdbc:hive2://gulu.uganda.com:10000/default;principal=hive/_HOST@UGANDA.COM 

Create the demo directory in hdfs and copy all the files created previously locally workflow.xml,crime_per_year.hql and downloaded files

[oozie@gulu crime]$ hdfs dfs -mkdir -p /user/oozie/ooziedemo 

Validation

[oozie@gulu crime]$ hdfs dfs -ls /user/oozie 
Found 3 items 
-rwxrwxrwx 3 oozie hdfs 1517368515 2017-10-10 17:39 /user/oozie/crime 
drwxr-xr-x - oozie hdfs 0 2017-10-10 17:57 /user/oozie/ooziedemo 
drwxr-xr-x - oozie hdfs 0 2017-07-31 00:35 /user/oozie/share 
[oozie@gulu app]$ vi workflow.xml [
oozie@gulu app]$ vi crime_per_year.hql 
[oozie@gulu app]$ cd ~/ooziedemo  
[oozie@gulu ooziedemo]$ hdfs dfs -copyFromLocal hivedemo /user/oozie/ooziedemo/ 
[oozie@gulu ooziedemo]$ hdfs dfs -ls -R /user/oozie/ooziedemo/ 
drwxr-xr-x - oozie hdfs 0 2017-10-10 18:10 /user/oozie/ooziedemo/hivedemo 
drwxr-xr-x - oozie hdfs 0 2017-10-10 18:10 /user/oozie/ooziedemo/hivedemo/app 
-rw-r--r-- 3 oozie hdfs 206 2017-10-10 18:10 /user/oozie/ooziedemo/hivedemo/app/crime_per_year.hql 
drwxr-xr-x - oozie hdfs 0 2017-10-10 18:10 /user/oozie/ooziedemo/hivedemo/app/lib 
-rw-r--r-- 3 oozie hdfs 968 2017-10-10 18:10 /user/oozie/ooziedemo/hivedemo/app/workflow.xml 
-rw-r--r-- 3 oozie hdfs 406 2017-10-10 18:10 /user/oozie/ooziedemo/hivedemo/job.properties 

export the OOZIE_URL environment variable

[oozie@gulu ooziedemo]$ export OOZIE_URL=http://gulu.uganda.com:11000/oozie 

Run the oozie job

[oozie@gulu hivedemo]$ oozie job -run -config job.properties -verbose -debug -auth kerberos 
Auth type : kerberos POST http://gulu.uganda.com:11000/oozie/v2/jobs?action=start 
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration> 
<property><name>nameNode</name><value>hdfs://gulu.uganda.com:8020</value></property> 
<property><name>outputHiveDatabase</name><value>default</value></property> 
<property><name>oozie.wf.application.path</name><value>${exampleDir}/app</value></property> 
<property><name>oozie.use.system.libpath</name><value>true</value></property> 
<property><name>exampleDir</name><value>${nameNode}/user/${user.name}/ooziedemo/hivedemo</value></property> 
<property><name>jdbcPrincipal</name><value>hive/_HOST@UGANDA.COM</value></property> 
<property><name>queryYear</name><value>2008</value></property> 
<property><name>jobTracker</name><value>gulu.uganda.com:8050</value></property> 
<property><name>jdbcURL</name><value>jdbc:hive2://gulu.uganda.com:10000/default</value></property> 
<property><name>hivescript</name><value>${oozie.wf.application.path}/crime_per_year.hql</value></property> 
<property><name>user.name</name><value>oozie</value></property> </configuration> 
job: 0000000-170927191637157-oozie-oozi-W

Once launched I switched to the RM to monitor the progress, See the attached screenshots for the Resource Manager. The job completed successfully, so compare and see where you went wrong or revert I could be of some help

Hope that helps


oozie-4-jobhiistory.jpgoozie-3-finishedjob.jpgoozie-2-runningjob.jpg

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

sure it works. you submit the job using oozie account.

instead, try with a regular user account which gets its token through kinit.

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

Mentor

@John Doe

I used the same environment but copied the ooziedemo and required files and changed the ownership and permissions for Opera1. The ONLY exception I didn't recreate the table in hive

login as: root
root@192.168.0.80's password:
Last login: Tue Oct 10 17:19:25 2017 from 192.168.0.178 

Locates a non HDP user for the second test opera1

[root@gulu ~]# cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
.......
......
activity_analyzer:x:1023:1002::/home/activity_analyzer:/bin/bash
slider:x:1024:970:SLIDER:/var/lib/slider:/bin/bash
spark:x:986:968:Spark:/var/lib/spark:/bin/bash
ranger:x:1025:1002::/home/ranger:/bin/bash
kms:x:1026:1002::/home/kms:/bin/bash
opera1:x:1027:1027::/home/opera1:/bin/bash 

Switched user to OPERA1

[root@gulu ~]# su - opera1
Last login: Sun Sep 10 20:34:24 CEST 2017 on pts/1 

Checked the correct principal and grabbed a Kerberos ticket

[opera1@gulu ~]$ klist -kt /etc/security/keytabs/opera1.keytab
Keytab name: FILE:/etc/security/keytabs/opera1.keytab
KVNO      Timestamp                Principal
---- ------------------- ------------------------------------------------------
   1       08/24/2017 18:28:26     opera1@UGANDA.COM 

Grabbed a valid ticket

[opera1@gulu ~]$ kinit -kt /etc/security/keytabs/opera1.keytab opera1@UGANDA.COM 

Checked ticket validity

[opera1@gulu hivedemo]$ klist
Ticket cache: FILE:/tmp/krb5cc_1027
Default principal: opera1@UGANDA.COM
Valid starting       Expires              Service principal
10/10/2017 20:28:46  10/11/2017 20:28:46  krbtgt/UGANDA.COM@UGANDA.COM 

As root copied the ooziedemo to /tmp for user Opera1

[root@gulu ~]# cd /home/oozie/
[root@gulu oozie]# ls
ooziedemo
[root@gulu oozie]# cp -R ooziedemo /tmp
[root@gulu oozie]#cd /tmp 

Gave user Opera1 correct permissions on the ooziedemo

[root@gulu tmp]# chown -R opera1:opera1 ooziedemo 

As the hdfs user copied the file to HDFS

[hdfs@gulu ~]$ hdfs dfs -cp /user/oozie/*  /user/opera1
[hdfs@gulu ~]$ hdfs dfs -ls  /user/opera1
Found 7 items
drwxr-xr-x   - opera1 hdfs          0 2017-09-07 19:57 /user/opera1/.hiveJars
drwx------   - opera1 hdfs          0 2017-09-10 20:45 /user/opera1/.staging
-rw-r--r--   3 hdfs   hdfs 1517368515 2017-10-10 20:40 /user/opera1/crime
-rw-r--r--   3 hdfs   hdfs     134241 2017-10-05 21:37 /user/opera1/hs_err_pid18536.log
drwxr-xr-x   - hdfs   hdfs          0 2017-10-10 20:40 /user/opera1/oozie-oozi
drwxr-xr-x   - hdfs   hdfs          0 2017-10-10 20:40 /user/opera1/ooziedemo
drwxr-xr-x   - hdfs   hdfs          0 2017-10-10 20:40 /user/opera1/share 

As HDFS user changed the ownership to opera1

[hdfs@gulu ~]$ hdfs dfs -chown opera1:hdfs  /user/opera1 

Copied the ooziedemo to /tmp and changed permissions for Opera1

[root@gulu hive]# cp load_crime_table.ddl /tmp/ooziedemo/
[root@gulu hive]# chown opera1:opera1 /tmp/ooziedemo/*
[root@gulu hive]# ls -al /tmp/ooziedemo/
total 24
drwxr-xr-x   3 opera1 opera1  4096 Oct 10 20:49 .
drwxrwxrwt. 46 root   root   12288 Oct 10 20:49 ..
drwxr-xr-x   3 opera1 opera1  4096 Oct 10 20:46 hivedemo
-rw-r--r--   1 opera1 opera1   715 Oct 10 20:49 load_crime_table.ddl 

export the OOZIE_URL environment variable

[opera1@gulu ~]$ export OOZIE_URL=http://gulu.uganda.com:11000/oozie 

Executed the job as opera1

[opera1@gulu hivedemo]$ oozie job -run -config job.properties -verbose -debug -auth kerberos 
Auth type : kerberos 
POST http://gulu.uganda.com:11000/oozie/v2/jobs?action=start 
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration> 
<property><name>nameNode</name><value>hdfs://gulu.uganda.com:8020</value></property> 
<property><name>outputHiveDatabase</name><value>default</value></property> 
<property><name>oozie.wf.application.path</name><value>${exampleDir}/app</value></property> 
<property><name>oozie.use.system.libpath</name><value>true</value></property> 
<property><name>exampleDir</name><value>${nameNode}/user/${user.name}/ooziedemo/hivedemo</value></property> 
<property><name>jdbcPrincipal</name><value>hive/_HOST@UGANDA.COM</value></property> 
<property><name>queryYear</name><value>2008</value></property> 
<property><name>jobTracker</name><value>gulu.uganda.com:8050</value></property> 
<property><name>jdbcURL</name><value>jdbc:hive2://gulu.uganda.com:10000/default</value></property> 
<property><name>hivescript</name><value>${oozie.wf.application.path}/crime_per_year.hql</value></property> 
<property><name>user.name</name><value>opera1</value></property> 
</configuration> 
job: 0000001-170927191637157-oozie-oozi-W 
[opera1@gulu hivedemo]$ 

Followed the execution success in RM UI see screenshots,

Hope that answers your doubts and demonstrates, please let me know whether I answered your question


oozie-opera1-runningjob.jpgoozie-opera1-finishedjob.jpgoozie-opera1-jobhiistory.jpgoozie-opera1-mapreducejob.jpg

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

great it works for you.

Could you modify your hive script and include: select current_user();?

I wonder if the result set shows opera1 user name as well.

thanks!

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

Maybe you have oozie server and hiveserver2 on different hosts. Try setting the following settings in hdfs (you can also specify the exact hosts for the oozie server):

hadoop.proxyuser.hcat.hosts=*
hadoop.proxyuser.hive.hosts=*

Remember to not only restart the suggested services/components but also hiveserver2 (it will not be included in the restart by ambari).

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

Yes, they were running on different hosts.

I made all the necessary changes and I could submit oozie job using regular user account (e.g. joe).

However, what I noticed is that the hive queries were executed using hive service account.

Although, the Oozie UI displays joe in the User column.

Can this behaviour be changed?

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

When running hiveserver2, it is recommended that all queries are run by the hive user for security reasons (you can turn this off by the hive.server2.enable.doAs=true, but this is not recommended).This does not however mean that it's not your user is not completely running the job. Hiveserver2 sets up another layer of security (the Ranger hive plugin) on top of the ones that non-hiveserver2 jobs runs that is more granular. You can for example setup masking and per column permission.

The only things you need to make sure is that the hive user can access all the hive data files in hdfs either through a hdfs policy in Ranger (I recommend this option) or via hadoop file permissions. Then you have to set up a hive policy in Ranger and give e.g. "joe" access to certain databases and/or tables.

Re: HiveServer2 action in Oozie on kerberized cluster failas with - Error retrieving delegation token for user oozie

New Contributor

it seems that adding Ranger fixed that.

when I grant "joe" user the SELECT access to a table "t" and submit the oozie job as "joe", the current_user() function invoked from hive2 action reports... "joe". While the access to other tables is denied.

I need to run more tests but so far it seems to be what we expect.

Thanks a lot for you help!