Skip to main content
yoonli
Silver ⭐️⭐️⭐️
October 26, 2025
Question

Submit job spark standalone emit error kerberos when enabling event log to s3.

Hi everyone,

I installed Spark cluster standalone. I want to config send log job to minio. I configured spark-default.conf file as bellow.

 
    spark.eventLog.enabled=false
spark.eventLog.dir=s3a://spark-logs/
spark.history.fs.logDirectory=s3a://spark-logs/
spark.hadoop.fs.s3a.endpoint=http://192.168.182.131:9000
spark.hadoop.fs.s3a.access.key=admin
spark.hadoop.fs.s3a.secret.key=admin12345
spark.hadoop.fs.s3a.path.style.access=true"
 
When I submit job spark-submit --master spark://spark-master-svc:7077 --conf spark.jars.ivy=/tmp/.ivy2 pi.py 1. I got the error:
5/10/26 12:13:29 INFO SparkContext: Successfully stopped SparkContext
Traceback (most recent call last):
File "/opt/bitnami/spark/examples/src/main/python/pi.py", line 32, in <module>
.getOrCreate()
^^^^^^^^^^^^^
File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 497, in getOrCreate
File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/context.py", line 515, in getOrCreate
File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/context.py", line 203, in __init__
File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/context.py", line 296, in _do_init
File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/context.py", line 421, in _initialize_context
File "/opt/bitnami/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1587, in __call__
File "/opt/bitnami/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at jdk.security.auth/com.sun.security.auth.UnixPrincipal.<init>(Unknown Source)
at jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(Unknown Source)
at java.base/javax.security.auth.login.LoginContext.invoke(Unknown Source)
at java.base/javax.security.auth.login.LoginContext$4.run(Unknown Source)
at java.base/javax.security.auth.login.LoginContext$4.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
at java.base/javax.security.auth.login.LoginContext.login(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2065)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1975)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3746)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3736)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3520)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831)
at org.apache.spark.deploy.history.EventLogFileWriter.<init>(EventLogFileWriters.scala:60)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.<init>(EventLogFileWriters.scala:213)
at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:64)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:631)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Unknown Source)

2 replies

vafs
Gold ⭐️⭐️⭐️
Gold ⭐️⭐️⭐️
November 3, 2025

Hello @yoonli, 

Thanks for contacting our Cloudera Community and sharing your question. 

Something I have to mention is that Cloudera does not support standalone Spark cluster, we only work with YARN clusters. 

Anyway, taking a quick look on this issue, I see that you're not mentioning any principal or keytab, when you use Kerberos, it will always try to run kinit, but if there are no principal and keytab, it will fail. 

Something you can try is using the Simple Auth method, you can add these two settings on your spark-defaults.conf: 

spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
spark.hadoop.security.authentication=simple
hadoop.security.authentication=simple

https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html 

Translate
Regards, Andrés Fallas -- Was your question answered? Please take some time to click on "Accept as Solution" below this post. If you find a reply useful, say thanks by clicking on the thumbs-up button.
RangaReddy1988
Silver ⭐️
December 2, 2025

Hi @yoonli As per my knowledge Cloudera will not support Standalone setup and supported types are YARN and Kubernetes. 

For your error, you need to check kerberos setup for standalone cluster and try to run simple examples like SparkPi and see whether it is working or not.

Translate