@MPHSpeed  Working with the actual data instead of sample data I build, i would recommend making these two changes: 1. In extractText processor change "Enable Unix Lines Mode" to true. 2. In each dynamic property in RouteOnAttribute, change "equals" function to "contains" function. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

As I was already using the Hadoop Credential Provider, I found a solution that does not require decrypting the password as follows: PySpark code: # Spark session spark = SparkSession.builder \ .config("spark.yarn.keytab=/etc/security/keytabs/<APPLICATION_USER>.keytab") \ .appName('SPARK_TEST') \ .master("yarn") \ .getOrCreate() credential_provider_path = 'jceks://hdfs/<PATH>/<CREDENTIAL_FILE>.jceks' credential_name = 'PASSWORD.ALIAS' # Hadoop credential conf = spark.sparkContext._jsc.hadoopConfiguration() conf.set('hadoop.security.credential.provider.path',credential_provider_path) credential_raw = conf.getPassword(credential_name) for i in range(credential_raw.__len__()): password = password + str(credential_raw.__getitem__(i)) The important point above is the .config() line in SparkSession. You must enter the keytab to access the password. Otherwise you will get the encrypted value. I can't say that I'm very happy with being able to directly manipulate the password value in the code. I would like to delegate this to some component in a way that the programmer does not have direct access to the password value. Maybe what I'm looking for is some kind of authentication provider, but for now the solution above works for me. ... View more

@benimaru  It is important to understand that NiFi does not replicate active FlowFiles (objects queued in connection between NiFi processor components) across multiple nodes.  So in a five node NiFi cluster where you are load balancing FlowFiles across all nodes, each node has a unique subset of the full data received. This if node 1 goes down, the FlowFiles on node 1 will not be processed until node 1 is back up. 100% agree with @joseomjr that placing an external load balancer in front of the ListenUDP endpoint is the correct solution to ensure high availability of that endpoint across all your NiFi nodes. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thanks @Chandler641  Your issue is resolved after building the spark code properly.  Note: We will not support Upstream Spark installation in our cloudera cluster because we are done lot of customisation in cloudera spark to support multiple integration components. Please let me know if you have further concerns on this issue. ... View more