Is it possible run Spark in local mode against a kerberized cluster

[#1]

We've recently kerberized our HDFS development cluster. Before we did this we could run Spark jobs using spark.master=local from an IDE to test new code to allow debugging before deploying the code to the cluster and running in yarn mode.

Since kerberizing the cluster I've not been able to find a way run spark jobs in local mode. We get the following error:

org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

Everything works fine if we deploy the code and run in yarn mode, but this slows down development cycles.

I've tried passing through the hdfs config files and setting "hadoop.security.authentication"="kerberos" and looked on the internet but have not found definitive answer as to whether I can run a Spark job in local mode against a kerberized cluster.

[#2]

@Marcus Aidley

You can run Spark in local mode against a kerberized cluster.

Here are some configuration values to check:

• In spark-defaults.conf (in an HDP cluster: /etc/spark/conf/spark-defaults.conf);
  • spark.history.kerberos.enabled true
  • spark.history.kerberos.keytab /your/path/to/spark.headless.keytab
  • spark.history.kerberos.principal your-principal@YOUR.DOMAIN
• In spark-env.sh make sure you have
  • export HADOOP_CONF_DIR=/your/path/to/hadoop/conf
• In core-site.xml
  • hadoop.security.authentication: kerberos

[#3]

Hi @gnovak, @Marcus Aidley,

Could you please explain if you were able to solve this issue?

I am facing the same issue even after performing the recommended steps provided by @gnovak

[#4]

We have this working now. When running in local mode we invoke the following method loginKerb, before creating the SparkSession:

import org.apache.hadoop.security.UserGroupInformation;

import javax.security.auth.callback.*;
import javax.security.auth.login.LoginContext;
import javax.security.auth.login.LoginException;
import java.io.IOException;


public class LoginKerb {

    public static void loginKerb() throws LoginException, IOException {
        LoginContext lc = kinit();
        UserGroupInformation.loginUserFromSubject(lc.getSubject());
    }
    private static LoginContext kinit() throws LoginException {
        LoginContext lc = new LoginContext(LoginKerb.class.getSimpleName(), callbacks -> {
            for(Callback c : callbacks){
                if(c instanceof NameCallback)
                    ((NameCallback) c).setName(System.getProperty("kerberos.user"));
                if(c instanceof PasswordCallback)
                    ((PasswordCallback) c).setPassword(System.getProperty("kerberos.password").toCharArray());
            }
        });
        lc.login();
        return lc;
    }
}

[#5]

@Marcus Aidley Thanks for your quick response. Could you please provide me full implementation of LoginKerb class if possible? Is kerberos.user referring to principal name or just the user?

Stacktrace with loginKerb so far :-

javax.security.auth.login.LoginException: No LoginModules configured for KerberosUtil at javax.security.auth.login.LoginContext.init(LoginContext.java:264) at javax.security.auth.login.LoginContext.<init>(LoginContext.java:417)

[#6]

I've updated the code snippet. And yes kerberos.user is the principal.

[#7]

Hi @Marcus Aidley,

I am getting the exception added below. Do you use any JAAS conf in VM arguments?

javax.security.auth.login.LoginException: No LoginModules configured for LoginKerb at javax.security.auth.login.LoginContext.init(LoginContext.java:264) at javax.security.auth.login.LoginContext.<init>(LoginContext.java:417)

[#8]

Yes you'll need a jaas.conf on your path looking something like the following and we also have a krb5.conf on the path.

com.sun.security.jgss.krb5.initiate {
    com.sun.security.auth.module.Krb5LoginModule required
    doNotPrompt=true
    principal=""
    useKeyTab=true
    keyTab=""
    storeKey=true;
};

LoginKerb {
  com.sun.security.auth.module.Krb5LoginModule required client=TRUE;
};