Hi,
I have just installed spark2 in CDH 5.13.0
after enabling spark.authenticate i get an error when running a spark job
PYSPARK_PYTHON=/sso/sfw/cloudera/parcels/Anaconda/bin/python spark2-submit pyspark_script.py
17/11/06 16:35:53 INFO spark.SparkContext: Running Spark version 2.1.0.cloudera2
17/11/06 16:35:53 WARN spark.SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0
17/11/06 16:35:54 INFO spark.SecurityManager: Changing view acls to: desind
17/11/06 16:35:54 INFO spark.SecurityManager: Changing modify acls to: desind
17/11/06 16:35:54 INFO spark.SecurityManager: Changing view acls groups to:
17/11/06 16:35:54 INFO spark.SecurityManager: Changing modify acls groups to:
17/11/06 16:35:54 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:457)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:229)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:236)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:258)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:435)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:236)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
17/11/06 16:35:54 INFO spark.SparkContext: Successfully stopped SparkContext
Traceback (most recent call last):
File "/home/desind/pyspark_script.py", line 3, in <module>
sc = SparkContext(master='local[*]', appName='MyPySparkScript')
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 182, in _do_init
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 249, in _initialize_context
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1401, in __call__
File "/sso/sfw/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
