How to set up authentication for spark history server?

[#1]

Hi Guys,

I would like to know how to set up authentication for spark history server, so that unauthorized users cannot view the spark history server web ui, any help would be appreciated, thanks!

1. I do notice there are below statements in the official spark document:“Enabling authentication for the Web UIs is done using javax servlet filters. You will need a filter that implements the authentication method you want to deploy. Spark does not provide any built-in authentication filters.”

 2. There is also a thread on stackoverflow regardig this : “You re-use Hadoop's jetty authentication filter for Kerberos/SPNEGO: spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter and spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.params=type=kerberos,kerberos.principal=${spnego_principal_name},kerberos.keytab=${spnego_keytab_path}”.

with kerberos authentication enabled in CDH6.3, I followed the instructions in the above stackoverflow thread, but is unable to acheive the expected results, any user can view the spark history server web ui.

Thanks,

Michael

[#2 · Accepted solution]

Just an update, this is resolved.

1. With kerberos authentication enabled, you can go to the spark service's configuration tab and turn on "history_server_spnego_enabled", which will "Enables user authentication using SPNEGO (requires Kerberos), and enables access control to application history data.", and after the restart, the shs webui will be authenticated, as below screenshot shows:

Underneath, the shs is restarted with below configuration:

spark.history.kerberos.enabled=true
spark.history.kerberos.principal=xx
spark.history.kerberos.keytab=xxx
spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.type=kerberos
spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.kerberos.principal=xx
spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.kerberos.keytab=xx
spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.kerberos.name.rules=xxx
spark.history.ui.acls.enable=true
spark.ui.filters=org.apache.spark.deploy.yarn.YarnProxyRedirectFilter,org.apache.hadoop.security.authentication.server.AuthenticationFilter

2. If kerberos is not enabled, you have to implement your own authentication filter and configure below parameters：

spark.ui.filters=org.apache.spark.deploy.yarn.YarnProxyRedirectFilter,your-authentication-filter-name

spark.your-authentication-filter-name.param.parm-name=parm-value

spark.history.ui.acls.enable

spark.history.ui.admin.acls

spark.history.ui.admin.acls.groups