Error: a secret key must be specified via the spark.authenticate.secret config

[#1]

Hi,

I have just installed spark2 in CDH 5.13.0 

after enabling spark.authenticate i get an error when running a spark job

PYSPARK_PYTHON=/sso/sfw/cloudera/parcels/Anaconda/bin/python spark2-submit pyspark_script.py

17/11/06 16:35:53 INFO spark.SparkContext: Running Spark version 2.1.0.cloudera2
17/11/06 16:35:53 WARN spark.SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
17/11/06 16:35:54 INFO spark.SecurityManager: Changing view acls to: desind
17/11/06 16:35:54 INFO spark.SecurityManager: Changing modify acls to: desind
17/11/06 16:35:54 INFO spark.SecurityManager: Changing view acls groups to:
17/11/06 16:35:54 INFO spark.SecurityManager: Changing modify acls groups to:
17/11/06 16:35:54 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:457)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:229)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:236)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:258)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:435)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:236)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
17/11/06 16:35:54 INFO spark.SparkContext: Successfully stopped SparkContext
Traceback (most recent call last):
File "/home/desind/pyspark_script.py", line 3, in <module>
sc = SparkContext(master='local[*]', appName='MyPySparkScript')
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 182, in _do_init
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 249, in _initialize_context
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1401, in __call__
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config

[#2]

Hi,

Could you tell us what value you had set for the parameter "spark.authenticate".  if the value was set to "true"  then it will fail to start in local mode.

[#3]

> Key: SPARK-23476
> URL: https://issues.apache.org/jira/browse/SPARK-23476
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Affects Versions: 2.3.0
> Reporter: Gabor Somogyi
> Priority: Minor
>
> If spark is run with "spark.authenticate=true", then it will fail to start in local mode.
> {noformat}
> 17/02/03 12:09:39 ERROR spark.SparkContext: Error initializing SparkContext.
> java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret
config
> at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:401)
> at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:258)
> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:199)
> at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:290)
> ...
> {noformat}
> It can be confusing when authentication is turned on by default in a cluster, and one
tries to start spark in local mode for a simple test.
> *Workaround*: If {{spark.authenticate=true}} is specified as a cluster wide config, then
the following has to be added
> {{--conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf
"spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf
"spark.authenticate.enableSaslEncryption=false"}}
> in the spark-submit command.