Hi, I'm trying to run a spark job for which all executors have to call a secured (HTTPS) web service on a dedicated server. During SSL handshake, this server returns a certificate that has been signed by a private (company specific) CA. The certificate of this CA has been added to a custom truststore (cacert) that I would like to point to in spark configuration in order for the executors to validate server's certificates without any extra configuration. I know that I can pass following option to my spark-submit command line : "--conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=<MyCaCert> -Djavax.net.ssl.trustStorePassword=<MyPassword>" ...but I would like to avoid asking this to all our users (because they are not supposed to know where this trustore is located and its password). I tried to use the "ssl.client.truststore.location" property as described in https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/ch_wire-webhdfs-mr-yarn.html but it didn't change anything. Obviously spark does not use this configuration ? Do you guys know how is configured the default truststore used by spark executors ? Any help will be highly appreciated 🙂 Thanks ... View more

Hi, I'm trying to setup web services that interact with my hadoop/hbase kerberized cluster. My application is deployed in a tomcat server and I would like to avoid recreating a new HBase connection each and every time I have to access HBase. Similarly, I want my application to be self sufficient, i.e I dont want to proceed with 'kinit' commands before starting up my tomcat server. Thus, I would like to implement a utility class in charge of managing login operation on the cluster and connection to hbase, but I'm struggling with kind of "ticket expiration" issues. First time my GetHbaseConnection() method is invoked, it properly connects to the cluster using provided keytab & principal (using UserGroupInformation.loginUserFromKeytab(user, keyTabPath) method), and create a brand new hbase connection (ConnectionFactory.createConnection(conf)) => perfect. By default, obtained ticket has a 10h lifetime (default value from /etc/krb5.conf file), so everything seems to work fine during first 10 hours period. Unfortunately, after this ticket has expired, my code fails with following exception : 17/08/01 07:40:52 http-nio-8443-exec-4 WARN AbstractRpcClient:699 - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 17/08/01 07:40:52 http-nio-8443-exec-4 ERROR AbstractRpcClient:709 - SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] => I had to setup a dedicated thread that invoke UserGroupInformation.checkTGTAndReloginFromKeytab() method on a regular basis in order to refresh the ticket. Anyway, after a long time of inactivity (typically a whole night), when I try to invoke my web service, I can see following warnings in my tomcat logs : 17/08/03 08:25:28 hconnection-0x51b0ea6-shared--pool1-t51 WARN UserGroupInformation:1113 - Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/03 08:25:29 hconnection-0x51b0ea6-shared--pool1-t51 WARN UserGroupInformation:1113 - Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/03 08:25:30 hconnection-0x51b0ea6-shared--pool1-t51 WARN UserGroupInformation:1113 - Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/03 08:25:31 hconnection-0x51b0ea6-shared--pool1-t51 WARN UserGroupInformation:1113 - Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/03 08:25:35 hconnection-0x51b0ea6-shared--pool1-t51 WARN AbstractRpcClient:695 - Couldn't setup connection for myuser@mydomain.com to hbase/myserver.mydomain.com@mydomain.com ...And my call to the web service finally fails with SocketTimeoutException... To reproduce the issue quickly, I wrote a simple java application (outside of tomcat), removed the code that logs the user in the cluster to delegate this part to an external/manual kinit operation : Proceed with a 'kinit' operation outside of my java application. This way I am able to get a "short-life" (1 minute) ticket using a custom krb5.conf file : env KRB5_CONFIG=/local/home/myuser/mykrb5.conf kinit -kt /local/home/myuser/myuser.keytab myuser@mydomain.com Then I execute my java standalone application that displays the name of one table in HBase on a regular basis (every 10 seconds). Note that I create a new HBase connection for every iteration, I dont try to reuse connection at the moment : public static void main(String[] args) throws IOException, InterruptedException { System.setProperty( "sun.security.krb5.debug", "true"); Configuration configuration = HBaseConfiguration.create(); while (true) { Connection conn = ConnectionFactory.createConnection(configuration); Admin admin = conn.getAdmin(); TableName[] tableNames = admin.listTableNames(); System.out.println(tableNames[0].getNameWithNamespaceInclAsString()); Thread.currentThread().sleep(10000); } } During 1 minute, it works perfectly, but then I face endless warnings and my code does not execute properly : 17/08/01 16:01:55 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/01 16:01:57 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/01 16:01:59 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/01 16:02:00 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 17/08/01 16:02:01 WARN ipc.AbstractRpcClient: Couldn't setup connection for myuser@mydomain.com to hbase/myserver.mydomain.com@mydomain.com ... I dont understand the how kerberos ticket expiration and hbase connection work together, does anyone could help on this topic ? In other words, I would like that my application connects to the cluster when it starts up, and create an hbase connection that I can keep "forever". Is it possible ? What did I miss ? Thanks for your help ... View more