Thank you for the input, this was helpful advice! ... View more

@saivenkatg55    A very common reason for UI slowness is JVM Garbage Collection (GC).  All GC events are stop-the-world events whether it is a partial or full GC event.  Partial/young GC is normal and healthy, but if it is being triggered back to back non stop or is running against a very large configured JVM heap it can take time to complete. You can enable some GC logging in your NiFi bootstrap.conf file so you can see how often GC is running to attempt to free space in your NiFi JVM.  To do this you need to add some additional java,arg.<unique num>= entries in your NiFi bootstrap.conf as follows: java.arg.20=-XX:+PrintGCDetails java.arg.21=-XX:+PrintGCTimeStamps java.arg.22=-XX:+PrintGCDateStamps java.arg.23=-Xloggc:<file> The last entry allows you to specific a separate log file for this output to be written in to rather than stdout. NiFi does store information about component status in heap memory.  This is the info you can see on any component (processor, connection, process group, etc.) when you right click on it and select "view status history" from the displayed context menu.  You'll notice that these component report status for a number of data points.  When your restart your NiFi, everything in the JVM heap memory is gone.  So over the next 24 hours (default data point retention) the JVM heap will be holding a full set of status points again.   You can adjust the component status history buffer size and datapoint frequency to reduce heap usage here if this status history is not that important to you via the following properties in the nifi.properties file: nifi.components.status.repository.buffer.size=1440 nifi.components.status.snapshot.frequency=1 min above represents defaults.  For every status history point for every single component, NiFi will retain 1440 status points (recording 1 point every 1 min).   This totals 24 hours worth of status history for every status point.  Changing the buffer to 288 and frequency to 5 minutes will reduce number of points retained by 80% while still giving your 24 hours worth of points. The dataflows you build may result in high heap usage triggering a lot of heap pressure.  Those NiFi components that can result in high heap usage are documented.  From the NiFi Global menu in the upper right corner of the NIFi UI, select "Help".  You will see the complete list of components on the left had side.  When you select a component, details about that component will be displayed.  One of the details is "System Resource Considerations". For example, here is the system resource considerations for the MergeContent processor: You may need to make adjustments to your dataflow designs to reduce heap usage.   NiFi also holds FlowFile metadata for queued FlowFiles in heap memory.  NiFi Does have a configurable swap threshold (which is applied per connection) to help with heap usage here.  When a queue grows too large, FlowFile metatdata in excess of the configured swap threshold will be written to disk.  The swapping in and swapping out of FlowFiles from disk can affect dataflow performance.  NiFi's default backpressure object thresholds settings for connections is set low enough that swapping would typically never occur.  However, if you have lots and lots of connections with queued FlowFiles, that heap usage can add up.  This article I wrote may help you here: https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166 ----- Other than heap usage, component validation can affect NiFi UI responsiveness.  Here is an article i wrote about that: https://community.cloudera.com/t5/Community-Articles/HDF-NiFi-Improving-the-performance-of-your-UI/ta-p/248211   Here is another useful article you may want to read: https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999   Hope this helps you with some direction to help improve your NiFi UI responsiveness/performance, Matt     ... View more

@vikram_shinde the Escape Character is a setting in the CSVReader : CSVReader 1.9.0.3.4.1.1-4 my nifi version is: 1.9.   Escape Character is a basic of CSV, as it is required to ignore quotes inside of the "enclosed by quotes".  So if you feed the escaped string to the CSV reader it will output the correct values w/o the escape character.   ... View more

@DennisJaheruddi ....Thanks much for making the Christmas more merrier 🙂 I agree to your statement and have configured the flow accordingly. I am marking your reply as accepted solution. Great advise and kudos to you again.  ... View more

Hi Dennis,   Thank you for the reply. I could use the ListFile and FetchFile processors to get the files of the target host (such as: MySQL log files).   ... View more

I'm happy to see you resolved your issue. Please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.            ... View more

From the Ranger email list, this is another bit of information that I found helpful:   ----------   I’ve configured ranger using the following approach to control who must be synced with AD. Only users belonging to groups inside a specific OU will be synced. I’ve created the OU  OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC=domain,DC=com Create a group called R2Users inside that OU. I put all desired sync users as its members. Also, you can put other groups as its member.  And, you can create other groups like R2TEAM as well. Remember to update this property ranger.usersync.ldap.user.searchfilter to include more than one. I’ve configured ranger to sync groups before users.   Here is the configuration. in COMMON CONFIGS Label Property Value LDAP/AD URL ranger.usersync.ldap.url ldap://myacticedirectoryserver.domain.com:389 Bind User ranger.usersync.ldap.binddn CN=LDAP_AD_ACCOUNT,OU=Service Accounts,OU=LCB,OU=Brazil,DC=domain,DC=com Bind User Password ranger.usersync.ldap.ldapbindpassword LDAP_AD_ACCOUNT user’s password Inclemetal Sync ranger.usersync.ldap.deltasync Yes Enable LDAP STARTTLS ranger.usersync.ldap.starttls No GROUP CONFIGS Label Property Value Enable Group Sync ranger.usersync.group.searchenable Yes Group Member Attribute ranger.usersync.group.memberattributename member Group Name Attribute ranger.usersync.group.nameattribute Cn Group Object Class ranger.usersync.group.objectclass Group Group Search Base ranger.usersync.group.searchbase OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC=domain,DC=com Group Search Filter ranger.usersync.group.searchfilter   Enable Group Search First ranger.usersync.group.search.first.enabled Yes Sync Nested Groups is_nested_groupsync_enabled Yes Group Hierarchy Levels ranger.usersync.ldap.grouphierarchylevels 5 USER CONFIGS Label Property Value Username Attribute ranger.usersync.ldap.user.nameatributte sAMAccountName User Object Class ranger.usersync.ldap.objectclass User User Search Base ranger.usersync.ldap.searchbase DC=domain,DC=com User Search Filter ranger.usersync.ldap.user.searchfilter (memberOf=CN=R2Users,OU=ArthurAmericasGroups,OU=Security Groups,OU=Groups,OU=SHARED,OU=Brazil,DC=domain,DC=com) User Search Scope ranger.usersync.ldap.user.searchscope Sub User Group Name Attribute ranger.usersync.ldap.groupnameattribute sAMAccountName Group User Map Sync ranger.usersync.group.usermapsyncenabled Yes Enable User Search ranger.usersync.user.searchenabled Yes ADVANCED Ranger Settings Label Property Value Authentication method   ACTIVE_DIRECTORY AD Settings Label Property Value AD Bind Password ranger.ldap.ad.bind.password LDAP_AD_ACCOUNT user’s password Domain Name (Only for AD) anger.ldap.ad.domain DC=domain,DC=com AD Base DN ranger.ldap.ad.base.dn DC=domain,DC=com AD Referreal ranger.ldap.ad.referreal Follow AD User Serach Filter ranger.ldap.ad.user.search (sAMAccountName={0}) Advanced ranger-ugsync-site Label Property Value ranger.usersync.ldap.referral ranger.usersync.ldap.referral Follow ... View more