Member since
01-02-2023
55
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
862 | 09-18-2023 04:37 AM | |
976 | 03-17-2023 06:34 AM | |
1528 | 03-13-2023 01:43 PM |
09-18-2023
04:37 AM
Thanks Matt!
... View more
09-13-2023
04:31 AM
Is it possible, or does it make sense to use AWS Load Balancers in front of a NiFi Cluster? We are running a 3 node nifi cluster using v 1.19.1 on aws and I wanted to see if it is possible, or makes sense to use an AWS Load Balancer(s) in front of the nifi cluster. Any guidance, suggestions would be tremendously appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
05-16-2023
04:26 AM
Additionally, i copied the keystore and truststore to the node-master. I did not yet copy the keystore and truststore to node1, node2, node3 and node4.
... View more
05-16-2023
03:56 AM
And, just to be complete, i did enter the correct values for the following items in the spark-default.conf file: spark.ssl.keyStore spark.ssl.keyStorePassword spark.ssl.truststore spark.ssl.trustStorePassword
... View more
05-16-2023
03:54 AM
Hello -
I have installed a 4 node Spark-3.3.1 running on YARN. The nodes are: node-master, node1, node2, node3 and node4. I am using JAVA 8.
Spark is working.
I have configured Spark to use 3rd party PKI certificates and successfully created the keystore and truststore. I copied the keystore and truststore to the Spark/conf directory.
I added the following to the spark-defaults.conf file:
spark.ssl.enabled true
spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_GCM_SHA384
spark.ssl.protocol TLSv1.2
spark.ssl.keyPassword key_password (i entered the correct keypassword)
spark.ssl.keyStore /path/to/my_key_store ( i entered the correct path to include the file name) spark.ssl.keyStore
spark.ssl.keyStorePassword
spark.ssl.truststore
spark.ssl.trustStorePassword
I then started the spark cluster,
$./sbin/start-master.sh
I looked in the log specified when i ran start-master.sh, and this is what the log contained:
INFO Master: Started daemon with process name: 6096@IP...
INFO SignalUtils: Registering signal handler for TERM
INFO SignalUtils: Registering signal handler for HUP
INFO SignalUtils: Registering signal for INT
INFO SecurityManager: Changing view acls to: hadoop
INFO SecurityManager: Changing modify acls to: hadoop
INFO SecurityManager: Changing view acls groups to:
INFO SecurityManager: Changing modify acls groups to:
INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
INFO Utils: Successfully started service 'sparkMaster' on port 7077
INFO Master: Starting Spark master at spark://ip...:7077
INFO Master: Running Spark version 3.3.1
WARN SslContextFactory: No supported Cipher Suite from [...it appears to be a complete list of the ciphers supported by Java 8...] e.g., TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA2556...and this repeats w/ 45 cipher entries...
INFO Utils: Successfully started service 'MasterUI (HTTPS)' on port 8480
INFO Utils: Successfully started service 'MasterUI' on port 8080
INFO MasterWebUI: Boiund MasterWebUI to 0.0.0.0 and started https://node-master:8480
INFO Master: I have been elected leader! New state: ALIVE
Can anyone shed light on the meaning of the warning above? And, why is it spitting out all of the Java 8 ciphers?
Thank you for any guidance/recommendations anyone can provide.
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
04-28-2023
12:13 AM
Any Apache Spark experts out there that have experience security hardening Spark on YARN?
... View more
04-23-2023
08:06 AM
Hello - I am new to Apache Spark. I have installed a 5 node Apache Spark cluster running on YARN on AWS. I am seeking any "prescriptive" vs descriptive information on how to security harden the cluster. I have reviewed the Apache Spark security website, and scoured the internet for detailed, prescriptive information/approaches for security hardening this type of cluster and have not be been able to find information provided by folks with direct, hands on experience. I have also searched for online courses that focus on implementing security with the Spark on YARN cluster and came up short. Does anyone have any suggested references that provide/describe how to's, detailed information on hardening procedures for this type of cluster. Any suggested references would be tremendously appreciated! VR, Dave
... View more
Labels:
- Labels:
-
Apache Spark
04-23-2023
02:34 AM
thanks for the reply cotopaul. I'm still very new to NiFi clusters and was thinking, from a purely cloud architecture perspective, that it typically makes sense to abstract the specific endpoints from the end user. I'll review the url you provided. Thanks again for your reply.
... View more
04-22-2023
01:21 PM
Is it possible...does it make sense to place an AWS application load balancer in front of a NiFi cluster?
... View more
Labels:
- Labels:
-
Apache NiFi
04-21-2023
03:09 AM
1 Kudo
Hello, I have a 3 node NiFi cluster. I would like to add new users to the cluster that would be administrators i.e., a new user with all of the capabilities of the initial admin identity user (in my case, the initial admin identify is user nifi). I am using PKI certs as the means of identifty/authenticating users and this is working well. So, my question is, can I create a handful of users that are administrators that have a policy set equivalent to the initial admin identity? Is this possible?
... View more
Labels:
- Labels:
-
Apache NiFi