@Brandon Wilson @Khera @slachterman
Disclaimer: I work with Khera.
We are using the MS SQL JDBC connector found here: https://www.microsoft.com/en-us/download/details.aspx?id=11774
Specifically the sqljdbc42.jar file which supports Java 1.8 (We use Oracle Java 1.8 on all our Centos Nodes).
Our entire cluster is "kerberized", so all access is setup with Kerberos. All nodes are setup within Active Directory to allow Kerberos Delegation. Additionally all user accounts used to access the cluster are AD accounts, and will have valid kerberos tickets at login using one of two methods:
1. If they use a SSH client that supports GSSAPI, and requests delegation of tokens, they will have a delegated Kerberos token at login. For example:
[xxxxxx@xxxxxxx ~]$ klist -f
Ticket cache: FILE:/tmp/krb5cc_478262072
Default principal: xxxxxx@xxxx
Valid starting Expires Service principal
08/29/2016 12:54:12 08/29/2016 22:54:12 krbtgt/xxxx@xxx
renew until 09/05/2016 12:54:12, Flags: FfRA
2. If they use an SSH client and login using a password (no GSSAPI), they will automatically be granted a Kerberos ticket. For example:
[xxxxx@xxxxx ~]$ klist -f
Ticket cache: FILE:/tmp/krb5cc_478262072
Default principal: xxxxx@CORP.FOOTLOCKER.NET
Valid starting Expires Service principal
09/07/2016 10:28:36 09/07/2016 20:28:36 krbtgt/xxxxx@xxxx
renew until 09/14/2016 10:28:36, Flags: FRIAA manual kinit is always available and works, but by default will not generate forwardable tickets (the F flag seen above)
So when he calls scoop like so:
sqoop
list-databases --connect
"jdbc:sqlserver://hostname.domain.com;integratedSecurity=true;authenticationScheme=JavaKerberos"
--username 'DOMAIN\USER'
We know that it is using his Kerberos ticket to connect to the SQL server, as there is no password prompt and authentication is still required to get a list of databases (so I've been told). I do not have access to the SQL server in question, so I can't verify using the Windows event logs that Kerberos is in use, but it certainly looks like it works 🙂
However when we then try to import the data, the mapreduce job either does not have any kerberos tickets available or just not one for khera. I'm not familiar with the chain of services that is involved in the import (Scoop, then Yarn, then MapR?). Additionally our cluster is using Knox/Ranger which seems to further complicate things, as well as the fact that I don't have access to the SQL server nor the Hadoop processes involved so I can't do any troubleshooting myself.
I have a few questions, hopefully someone can answer them.
1. What is the chain of services involved? (Scoop submits job to YARN, Yarn runs Mapreduce etc?)
2. Once we know #1, do all of those services involved
a) Support Kerberos
B) Support Kerberos DELEGATION as the user
C). Forwarding the Kerberos delegated ticket to the NEXT process in the chain (or conversely requesting delegation).
Any help is appreciated.
