@Eric Lloyd The NiFi expression language was written specifically for working with NiFi attributes. You would first need to use ExtractText processor to get the bits from your content moved into NiFi FlowFile Attributes: Add a new property to your ExtractText procesor configured as follows: Note that my Regex above has a white space at the end. This regex will result in multiple new FlowFile Attributes being created for you: So no need to followup with any substring NiFi expression language manipulation commands. Thanks, Matt ... View more

@Prabir Guha You would certainly use the UpdateAttribute processor to do this and a NiFi expression language statement as follows : ${filename:substringAfterLast('/')} Thanks, Matt ... View more

@Ninad Patkhedkar You are correct that the File size as it exists on the SFTP server is not written to an attribute on the listed FlowFile. On every FlowFile NiFi creates a FlowFile property ( fileSize ) which records the size of the Content associated to that FlowFile. This FlowFile property is not editable and is updated with the actual size of the fetched content post FetchSFTP. I understand you don't want to actual fetch the data but only want to get some metadata (including size) about what currently exist on the SFTP server. Interesting idea. I suggest creating an Apache Jira to add an addition FLowFile Attribute on list (for example maybe "sftp.server.file.size"). We have to make sure the attribute name is very descriptive so users do not confuse it with "fileSize" that already exists on the FlowFile. We can never assume that both "fileSize" and "sftp.server.file.size" will ever be exactly the same. And "fileSize" will change depending on how the content is manipulated as it progress through a NiFi dataflow. I see a valid use case here: But adding this property would allow users to make routing decisions on listed files. Perhaps you don't want to Fetch any Files form the SFTp server if they are larger then XXX in size. Thanks, Matt ... View more

@Sertac Kaya A few questions come to mind... 1. What kind of processor is feeding the connection with the large queue inside ExampleA? 2. How large is that queue? The reason I ask is because NiFi uses swapping to help to limit JVM heap usage by queued FlowFiles. How swapping is handled is configured in the nifi.properties file: nifi.queue.swap.threshold=20000 nifi.swap.in.period=5 sec nifi.swap.in.threads=1 nifi.swap.out.period=5 sec nifi.swap.out.threads=4 The above shows NiFi defaults. A few options you may do to improve performance: 1. Set backpressure thresholds on you connections to limit the number of FlowFiles that will queue at any time. Setting the value lower then they swapping threshold will prevent swapping from occurring on the connection. Newer version of NiFi by default set FlowFile object thresholds on newly created connections to 10,000. swapping is per connection and not per NiFi instance. 2. Adjust the swap.threshold value to a large value to prevent swapping. Keep in mind that any FlowFiles not being swapped are held in JVM heap memory. Setting this value to high may result in Out Of Memory (OOM) errors. Make sure you adjust your heap setting fro your NiFi in the bootstrap.conf file. 3. Adjust the swap in and swap out number of threads. Thanks, Matt ... View more

@Sanaz Janbakhsh HDF and HDP stacks may each use different version of Ranger. NiFi would not have been tested against a newer version of Ranger then what is included in HDF stack. That being said, there is no reason a single Ranger install could not be used to manage multiple services. The Ranger that is included with HDP will not include the service definition for NiFi, so it would need to be installed manually. The following link discusses how to set this up: http://bryanbende.com/development/2016/04/25/building-a-plugin-for-apache-ranger I do not know what impact manually adding service definitions HDP will have on HDP upgrades. Will those added service definitions be lost following upgrade? I would hope not, but personally have no knowledge in that area. Thanks, Matt ... View more

@kannan chandrashekaran You are correct that a function does not exist at this time for padding left or right of a string. That being said, You can easily accomplish string padding using a simple flow consisting of a RouteOnAttribute" and "UpdateAttribute" processor. The RouteOnAttribute would contain one routing rule that checks the length of an attributes value and if it is not long enough routes it to the update attribute where you add one character of padding. My rule looks like this: -- "10" is the length I want my attribute to be. -- "test" is the attribute that I am calculating the length of. My UpdateAttribute processor then simply pads the value assigned to"test" with a single character. FlowFiles will continue in this loop until the value of "test" has reached 10 charatcters in length. Any FlowFile where the value associated to "test" has a length longer then 10 is just passed on without any change. Thanks, Matt ... View more