HAWQ - Using custom Kerberos Principal

[#1]

Goals

• Use custom kerberos Principal for configuring HAWQ login
• Verifying the usage by creating, loading and query an object with non-default principal

NOTE HAWQ does not allow these parameters to be changed during the installation as configuration files are only generated after initialization of the HAWQ init process, so this is applicable only when you have a HAWQ cluster up and running with kerberos enabled.

Steps

Part 1: Kerberos

• Ensure that the HAWQ is up and running
• Login to kerberos server and create a custom headless principal

kadmin.local:  addprinc -randkey custom-postgres@HORTONWORKS.COM
WARNING: no policy specified for custom-postgres@HORTONWORKS.COM; defaulting to no policy
Principal "custom-postgres@HORTONWORKS.COM" created.

• Export the key into a keytab file, ensure that the name remains as "hawq.service.keytab" or else this will be required to change

kadmin.local:  xst -k hawq.service.keytab custom-postgres@HORTONWORKS.COM
....
kadmin.local:  quit

• Copy over the keytab file over to all the hosts on which you either have HAWQ master, standby or Segments. For fail safe, keep a backup of the existing "hawq.service.keytab".
• Verify the difference between the old and the new keytabs

[gpadmin@master master]$ klist -kt /etc/security/keytabs/hawq.service.keytab.orig 
Keytab name: FILE:/etc/security/keytabs/hawq.service.keytab.orig
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 10/10/2016 22:02:01 postgres@HORTONWORKS.COM
   2 10/10/2016 22:02:01 postgres@HORTONWORKS.COM
   2 10/10/2016 22:02:01 postgres@HORTONWORKS.COM
   2 10/10/2016 22:02:01 postgres@HORTONWORKS.COM
   2 10/10/2016 22:02:01 postgres@HORTONWORKS.COM
[gpadmin@master master]$ klist -kt /etc/security/keytabs/hawq.service.keytab 
Keytab name: FILE:/etc/security/keytabs/hawq.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 10/13/2016 15:42:36 custom-postgres@HORTONWORKS.COM
   2 10/13/2016 15:42:36 custom-postgres@HORTONWORKS.COM
   2 10/13/2016 15:42:36 custom-postgres@HORTONWORKS.COM
   2 10/13/2016 15:42:36 custom-postgres@HORTONWORKS.COM
   2 10/13/2016 15:42:36 custom-postgres@HORTONWORKS.COM
   2 10/13/2016 15:42:37 custom-postgres@HORTONWORKS.COM

Part 2: HAWQ

Now that we have successfully created the new custom principal, we need to modify HAWQs configuration file. This is simplified.

• Login to HAWQ master as gpadmin user
• Verify the service name for HAWQ's existing service name

[gpadmin@master master]$ hawq config -s krb_srvname
GUC		: krb_srvname
Value		: postgres
[gpadmin@master master]$   

• Verify the Kerberos cache file name

[gpadmin@master master]$ hawq config -s krb5_ccname
GUC		: krb5_ccname
Value		: /tmp/postgres.ccname

• Retrieve the filename for Kerberos cache file, this is unique for gpadmin user and can be retrieved once you login using the new kerberos principal. Login as gpadmin user and run kdestroy before running the following

[gpadmin@master master]$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_1002)

• Execute the following commands to modify HAWQ configuration throughout master and segment nodes

[gpadmin@master master]$ hawq config -c krb_srvname -v custom_postgres --skipvalidation
GUC krb_srvname already exist in hawq-site.xml
Update it with value: custom_postgres
GUC		: krb_srvname
Value		: custom_postgres

• Modify the cache file name and location according to our setup & finding

[gpadmin@master master]$ hawq config -c krb5_ccname -v /tmp/krb5cc_1002 --skipvalidation
GUC krb5_ccname already exist in hawq-site.xml
Update it with value: /tmp/krb5cc_1002
GUC		: krb5_ccname
Value		: /tmp/krb5cc_1002

NOTE New values are not updated until the service have been restarted once. Proceed with restarting HAWQ services via Ambari or using hawq stop and hawq start.

• After the restart, verify if the values for the properties were modified

[gpadmin@master master]$ hawq config -s krb_srvname && hawq config -s krb5_ccname
GUC		: krb_srvname
Value		: custom-postgres
GUC		: krb5_ccname
Value		: /tmp/krb5cc_1002

• Login to the HAWQ database using psql prompt and verify if you are able to create objects

[gpadmin@master ~]$ psql -p 10432 hdb
psql (8.4.20, server 8.2.15)
WARNING: psql version 8.4, server version 8.2.
         Some psql features might not work.
Type "help" for help.

hdb=# drop table test;
DROP TABLE
hdb=# create table test (col1 int);
CREATE TABLE
hdb=# 

• Validate if you are able to insert and read the data

hdb=# insert into test select * from generate_series(1,100);
INSERT 0 100
hdb=# select * from test limit 5;
 col1 
------
    1
    2
    3
    4
    5
(5 rows)

hdb=# 

If the changes are not made to HAWQ configuration and custom principals are used for hawq.service.keytab generation, you will get an error similar to the following

hdb=# create table test (col1 int);
WARNING:  failed to login to Kerberos, command line: kinit -k -t /etc/security/keytabs/hawq.service.keytab -c /tmp/postgres.ccname postgres
WARNING:  failed to login to Kerberos, command line: kinit -k -t /etc/security/keytabs/hawq.service.keytab -c /tmp/postgres.ccname postgres
CONTEXT:  Dropping file-system object -- Relation Directory: '16385/16388/16514'
WARNING:  could not remove relation directory 16385/16388/16514: Permission denied
CONTEXT:  Dropping file-system object -- Relation Directory: '16385/16388/16514'
ERROR:  could not create relation directory hdfs://master.hortonworks.com.hortonworks.com:8020/hawq_default/16385/16388/16514: Permission denied
hdb=# \q