@Sertac Kaya A few questions come to mind... 1. What kind of processor is feeding the connection with the large queue inside ExampleA? 2. How large is that queue? The reason I ask is because NiFi uses swapping to help to limit JVM heap usage by queued FlowFiles. How swapping is handled is configured in the nifi.properties file: nifi.queue.swap.threshold=20000 nifi.swap.in.period=5 sec nifi.swap.in.threads=1 nifi.swap.out.period=5 sec nifi.swap.out.threads=4 The above shows NiFi defaults. A few options you may do to improve performance: 1. Set backpressure thresholds on you connections to limit the number of FlowFiles that will queue at any time. Setting the value lower then they swapping threshold will prevent swapping from occurring on the connection. Newer version of NiFi by default set FlowFile object thresholds on newly created connections to 10,000. swapping is per connection and not per NiFi instance. 2. Adjust the swap.threshold value to a large value to prevent swapping. Keep in mind that any FlowFiles not being swapped are held in JVM heap memory. Setting this value to high may result in Out Of Memory (OOM) errors. Make sure you adjust your heap setting fro your NiFi in the bootstrap.conf file. 3. Adjust the swap in and swap out number of threads. Thanks, Matt ... View more

@Sanaz Janbakhsh HDF and HDP stacks may each use different version of Ranger. NiFi would not have been tested against a newer version of Ranger then what is included in HDF stack. That being said, there is no reason a single Ranger install could not be used to manage multiple services. The Ranger that is included with HDP will not include the service definition for NiFi, so it would need to be installed manually. The following link discusses how to set this up: http://bryanbende.com/development/2016/04/25/building-a-plugin-for-apache-ranger I do not know what impact manually adding service definitions HDP will have on HDP upgrades. Will those added service definitions be lost following upgrade? I would hope not, but personally have no knowledge in that area. Thanks, Matt ... View more

@kannan chandrashekaran You are correct that a function does not exist at this time for padding left or right of a string. That being said, You can easily accomplish string padding using a simple flow consisting of a RouteOnAttribute" and "UpdateAttribute" processor. The RouteOnAttribute would contain one routing rule that checks the length of an attributes value and if it is not long enough routes it to the update attribute where you add one character of padding. My rule looks like this: -- "10" is the length I want my attribute to be. -- "test" is the attribute that I am calculating the length of. My UpdateAttribute processor then simply pads the value assigned to"test" with a single character. FlowFiles will continue in this loop until the value of "test" has reached 10 charatcters in length. Any FlowFile where the value associated to "test" has a length longer then 10 is just passed on without any change. Thanks, Matt ... View more

@Sanaz Janbakhsh Once you have successfully validated your HDF upgrade, you can get rid of your backup. The backup of the Ambari resources was not always part of the procedure. It was added mainly because we found users were running a HDF upgrade on their HDP Ambari installs. By doing so they ended up messing up their HDP installs. HDF and HDP are two different product lines and the HDF mpack does not add to existing services, it makes available only those services in the HDF stack. So if you update the mpack on a HDP Ambari, you can really mess up your HDP cluster. So we added the steps to make a backup first. So after upgrade and verifying that all services are accessible and running without error there is probably no reason to keep the backup much longer then that. Thanks, Matt ... View more

@Sebastian Carroll The intent of the automated SSL setup is to help users who do not have an external CA quickly build and deploy (in the case of Ambari) a valid keystore and truststore to their NiFi instance(s). Ambari simply uses the tls-toolkit as well but with some pre-defined parameters to automated the creation of the keystores and truststore for your Ambari deployed NiFi cluster. It is really not recommended to use the NiFi CA in Ambari in a production environment. Users encouraged to use a legitimate CA to sign certificates in a production. The reason for this is because their is no inherent trust of any certs signed by the NiFi CA and every install of a HDF NiFi cluster will have its own CA. So using being able to use things like NiFi's S2S protocol between systems deployed using different NiFi CA adds a lot of additional work/overhead since you must constantly update the CAs in every systems truststore. If I am understanding you correctly, you are asking for a way to tell Ambari to generate certificates and pass them to an external company CA to get them signed? Since Ambari has no control over an external CA and the credentials needed to sign certificate requests should be highly protected, I don't see a way to securely automate this entire process. The best that could be done is to Automate the creation of the self-signed certificate and certificate signing request. The user would still need to send that request to their company CA to be signed and the import the signed response once received back in to your keystore. Users would also still need to manually obtain the public key for their company CA in order to create or add it to a truststore. The problem with having Ambari auto-generate a certificate is that many companies have specific requirements for what specifically must be defined in servers certificate. Having Ambari provide all possible options sounds like overkill. I don't see why you could not use the NiFi tls-toolkit to generate a certificate that you could then get signed by your own CA. Again, I don't really see how NiFi could automate beyond creating the cert and signing request. If I am missing something here, please let me know. In Ambari based installs you do not need to use the NiFi CA to create certificates. Simply make sure the NiFi certificate authority is not installed. Then in the NiFi configs within NiFi, configure the SSL settings to point at the PKCS12 or JKS keystore and truststore you manually obtained via your company. The configs by default in Ambari expect that every node is using a keystore named the same (content of keystore shoudl be unique on each node)and that the keystores all use the same password. Thank you, Matt ... View more