Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Hello -
I have installed a 4 node Spark-3.3.1 running on YARN. The nodes are: node-master, node1, node2, node3 and node4. I am using JAVA 8.
Spark is working.
I have configured Spark to use 3rd party PKI certificates and successfully created the keystore and truststore. I copied the keystore and truststore to the Spark/conf directory.
I added the following to the spark-defaults.conf file:
spark.ssl.enabled true
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_GCM_SHA384
spark.ssl.protocol TLSv1.2
spark.ssl.keyPassword key_password (i entered the correct keypassword)
spark.ssl.keyStore /path/to/my_key_store ( i entered the correct path to include the file name)
spark.ssl.keyStore
spark.ssl.keyStorePassword
spark.ssl.truststore
spark.ssl.trustStorePassword
I then started the spark cluster,
$./sbin/start-master.sh
I looked in the log specified when i ran start-master.sh, and this is what the log contained:
INFO Master: Started daemon with process name: 6096@IP...
INFO SignalUtils: Registering signal handler for TERM
INFO SignalUtils: Registering signal handler for HUP
INFO SignalUtils: Registering signal for INT
INFO SecurityManager: Changing view acls to: hadoop
INFO SecurityManager: Changing modify acls to: hadoop
INFO SecurityManager: Changing view acls groups to:
INFO SecurityManager: Changing modify acls groups to:
INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
INFO Utils: Successfully started service 'sparkMaster' on port 7077
INFO Master: Starting Spark master at spark://ip...:7077
INFO Master: Running Spark version 3.3.1
WARN SslContextFactory: No supported Cipher Suite from [...it appears to be a complete list of the ciphers supported by Java 8...] e.g., TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA2556...and this repeats w/ 45 cipher entries...
INFO Utils: Successfully started service 'MasterUI (HTTPS)' on port 8480
INFO Utils: Successfully started service 'MasterUI' on port 8080
INFO MasterWebUI: Boiund MasterWebUI to 0.0.0.0 and started https://node-master:8480
INFO Master: I have been elected leader! New state: ALIVE
Can anyone shed light on the meaning of the warning above? And, why is it spitting out all of the Java 8 ciphers?
Thank you for any guidance/recommendations anyone can provide.
Created 05-16-2023 03:56 AM
And, just to be complete, i did enter the correct values for the following items in the spark-default.conf file:
spark.ssl.keyStore
spark.ssl.keyStorePassword
spark.ssl.truststore
spark.ssl.trustStorePassword
Created 05-16-2023 04:26 AM
Additionally, i copied the keystore and truststore to the node-master. I did not yet copy the keystore and truststore to node1, node2, node3 and node4.