Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error: a secret key must be specified via the spark.authenticate.secret config

Highlighted

Error: a secret key must be specified via the spark.authenticate.secret config

Expert Contributor

Hi,

 

I have just installed spark2 in CDH 5.13.0 

after enabling spark.authenticate i get an error when running a spark job

 

PYSPARK_PYTHON=/sso/sfw/cloudera/parcels/Anaconda/bin/python spark2-submit pyspark_script.py

 

17/11/06 16:35:53 INFO spark.SparkContext: Running Spark version 2.1.0.cloudera2
17/11/06 16:35:53 WARN spark.SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
17/11/06 16:35:54 INFO spark.SecurityManager: Changing view acls to: desind
17/11/06 16:35:54 INFO spark.SecurityManager: Changing modify acls to: desind
17/11/06 16:35:54 INFO spark.SecurityManager: Changing view acls groups to:
17/11/06 16:35:54 INFO spark.SecurityManager: Changing modify acls groups to:
17/11/06 16:35:54 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:457)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:229)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:236)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:258)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:435)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:236)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
17/11/06 16:35:54 INFO spark.SparkContext: Successfully stopped SparkContext
Traceback (most recent call last):
File "/home/desind/pyspark_script.py", line 3, in <module>
sc = SparkContext(master='local[*]', appName='MyPySparkScript')
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 182, in _do_init
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 249, in _initialize_context
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1401, in __call__
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config

 

2 REPLIES 2
Highlighted

Re: Error: a secret key must be specified via the spark.authenticate.secret config

Cloudera Employee

Hi,

 

Could you tell us what value you had set for the parameter "spark.authenticate".  if the value was set to "true"  then it will fail to start in local mode.

 

 

Highlighted

Re: Error: a secret key must be specified via the spark.authenticate.secret config

New Contributor
> Key: SPARK-23476
> URL: https://issues.apache.org/jira/browse/SPARK-23476
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell
> Affects Versions: 2.3.0
> Reporter: Gabor Somogyi
> Priority: Minor
>
> If spark is run with "spark.authenticate=true", then it will fail to start in local mode.
> {noformat}
> 17/02/03 12:09:39 ERROR spark.SparkContext: Error initializing SparkContext.
> java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret
config
> at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:401)
> at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:258)
> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:199)
> at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:290)
> ...
> {noformat}
> It can be confusing when authentication is turned on by default in a cluster, and one
tries to start spark in local mode for a simple test.
> *Workaround*: If {{spark.authenticate=true}} is specified as a cluster wide config, then
the following has to be added
> {{--conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf
"spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf
"spark.authenticate.enableSaslEncryption=false"}}
> in the spark-submit command.
Don't have an account?
Coming from Hortonworks? Activate your account here