@forest lin Backpressure is not used to control data rate in your dataflow. The intent of the backpressure setting on connections is to control the amount of allowed queued data. Both Back pressure settings are "soft" limits. Once backpressure kicks in on a connection, the processor feeding that connection will no longer be allowed to run. So in you case above, you have backpressure set to 5 Objects (FlowFiles) or 5 KB of content. Since your queue is empty, no backpressure was being applied when the 37.05 MB FlowFile arrived at your ConvertCSVToAvro processor, so that processor was allowed to run. That 1 FlowFile was processed through and placed on the outbound connection. It is at that time back pressure kicked in because you exceeded one of your backpressure settings. The ConvertCSVToAvro processor will now be prevented from running until that backpressure drops below 5 FlowFiles or 5 KB of queued data again. If all your processor are processing FlowFiles rapidly, back pressure will be very sparsely applied. Also keep in mind for efficiency some processors work on batches of FlowFiles. You may see for example with a backpressure object threshold of 5 a queue with more then 5 FlowFiles. The batch of FlowFiles are placed on an outbound queue. That processor who did the batch processing will then not be allowed to run again until that outbound connection drops again below 5 FlowFiles. The ControlRate processor allows you to actually control the throughput of a dataflow. It does not slow the processing. The ControlRate processor will allow data to queue in its input side and based on its configured setting only allow x number of FlowFiles through over y amount of time. lets say it is configured to let 5 KB of data through every 1 minute. If you feed it a 37 MB file, it does not transfer just pieces of that FlowFile. It will feed through the entire 37 MB FlowFile and then not allow another FlowFile through until the average data per 1 minute is 5 KB. Because of how the above works, data could continue to queue in front of ControlRate. This is where backpressure settings become important to stop upstream processor from running. You can set backpressure all the way upstream to your data ingest processors so they stop accepting new FlowFiles. Thanks, Matt ... View more

@Eric Lloyd Input and Output ports are designed to send or receive data from one level up. When an input or output port is added at the root canvas level the one level up is another out of the system. You will also notice that ports added to the root canvas are rendered a little differently. There is an open Apache Jira on this subject, feel free to add your comments and use case to it: https://issues.apache.org/jira/browse/NIFI-2933 The current feeling is that adding Remote input and output ports should be left to the system administrator. This is because in a secured connection the admin must add the connecting systems as new users and authorize them to access these ports. Users are not typically granted this level of access. Thanks, Matt ... View more

@Daniel Frank What format is your data in? (text?) Is all the information you need in the content of these files? The getFile processor already writes attributes for the following on every FlowFile it creates: You could use the ExtractText processor to read the FlowFile content and extract bits to FlowFile Attributes. Thanks, Matt ... View more

@Anishkumar Valsalam There are two parts that need to be successful to access NiFi: User authentication: In your case, you are using LDAP to authenticate your users. The NiFi login-identity-providers.xml is used to configure the ldap-provider. NiFI offers two supported configurable "Identity Strategy" options (USE_DN or USE_USERNAME). USE_DN is the default. With "USE_DN" the full DN returned by LDAP after successfully authenticating a used. With "USE_USERNAME" the username entered at login will be used. Which ever strategy is used, the value used will be passed through any configured "Identity Mapping Properties" in NiFi before the resulting mapped value is passed to part two. (Review LDAP settings and Identity mapping Properties in NiFi Admin guide for more details on setup) User Authorization: In you case, you are using Ranger for user authorization. (default is NiFi's file-based authorizer). The final value derived form step one above is passed to the configured authorizer to determine what NiFi resources that authenticated user has been granted access. Based on your output above, you appear to have two options possibly to match your authenticated value with your ldap sync'd user in Ranger: Configure an "Identity Mapping Property" in NiFi that will extract on the value from CN= from the entire returned DN. Based on the DN pattern you shared, your pattern mapping would look like this: nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), OU=(.*?), OU=(.*?), DC=(.*?), DC=(.*?), DC=(.*?)$nifi.security.identity.mapping.value.dn=$1 This will return just "anish" from the DN and that is what will be passed to the authorizer. Change your "Identity Strategy" configuration in your login-identity-providers.xml file to use "USE_USERNAME". This assumes the username supplied at login matches exactly with the LDAP sync username. Add/Modify the following line in your ldap-provider: <property name="Identity Strategy">USE_USERNAME</property> Thanks, Matt ... View more