Member since
07-30-2019
3470
Posts
1641
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 212 | 05-06-2026 09:16 AM | |
| 400 | 05-04-2026 05:20 AM | |
| 285 | 05-01-2026 10:15 AM | |
| 493 | 03-23-2026 05:44 AM | |
| 373 | 02-18-2026 09:59 AM |
12-13-2024
08:48 AM
1 Kudo
@tono425 Apache NiFi 2.0.0 has only been tested and verified against Java 21. I suggest switching to latest Java 21 JDK update and seeing if same messages persist in your Linux /var/log/messages log. NiFi 2.0.0 System requirements Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-13-2024
08:41 AM
1 Kudo
@Zifo1 When using Site-to-SIte via Remote Process Groups (RPG) and Remote Input or Output ports between NiFi clusters, it is most efficient to push rather then pull data (FlowFiles). The NiFi RPG always acts as the client side of the connection. It will either send FlowFiles to a Remote Input Port or fetch FlowFiles from a Remote Output port. I would avoid fetching from Remote Output ports. You get better FlowFiles distribution across teh destination cluster when you send FlowFiles from the RPG. If the FlowFiles traverse both directions, you would simply setup a RPG on both NiFi clusters to push FlowFiles to the Remote Input Ports on opposite clusters. Details about Site-To-Site can be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site As far as the RPG goes, I recommend using the "RAW" transport protocol over HTTP. RAW requires that the dedicated RAW port is configured in the server side NiFi's nifi.properties file. RAW establishes a raw socket connection on the dedicated configured port. HTTP utilizes the same HTTPS port that all other NiFi interactions use. You'll need to make sure the network connectivity exists between both your NiFi Clusters on both the HTTP(s) and RAW ports. HTTP is always used to fetch Site-to-Site Details. Setting up the client side (Remote Process Group) Documentation is here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-client-nifi-instance Setting up the sever side (NiFi with Remote Input or Remote Output ports) documentation can be found here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#configure-site-to-site-server-nifi-instance Even with Site-To-Site the communications between the two NiFi clusters requires both authentication and authorization. Authentication is established via a mutual TLS handshake initiated by the RPG. For Site-to-Site, the keystore and truststore setup en each NiFi's nifi.properties file are used in the MutualTLS exchange. NOTE: The NiFi out-of-box auto generated keystores and truststores are not suitable for negotiating a successful Mutual TLS handshake. There are numerous authorization policies that must be setup on the server side (remote ports NiFi) so that the client side (NiFi with RPG) is able to successfully send FlowFiles over Site-to-Site: 1. Retrieve Site-to-Site Details - This policy authorizes the client NiFi nodes (so all nodes in the client side NiFi cluster must be authorized) to retrieve site-to-site details from the server side NiFi. This includes details like number of nodes, load on those nodes, authorized remote ports, site-to-site raw port, https port, etc. 2. Receive data via Site-To-Site - This policy is setup on Remote Input ports to authorize the client side NiFi nodes to send FlowFiles to this specific port. 3. Send data via Site-to-Site - This policy is setup on the Remote Output Ports and allows authorized client nodes to fetch FlowFiles from the Remote output port. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-10-2024
11:16 AM
@Dahlgren - Make sure you are using a Java 21 JDK and not a Java 21 JRE. - Try excluding the NiFi directory to include all the repositories from your Virus scanning software and see if that helps resolve the startup issue. Here is an Apache NiFi slack thread discussing this same issue: https://apachenifi.slack.com/archives/C0L9UPWJZ/p1731963549046199 Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-10-2024
09:43 AM
1 Kudo
@Dahlgren I conversation with a committer to the Apache NiFi product and he stated that this is a known issue they are aware of and investigating. There is no Apache NiFi jira that I can find specifically for this issue. . The best suggestion I can make at this point is maybe raising an Apache NiFi jira and if possible share thread dumps (3 dumps spaced 5 minutes apart). Looking at what threads are not progressing from one thread dump to the next may provide clues as to where specifically it is getting hung in the startup code. That should narrow down the scope of investigation and confirm that all experiencing this issue are getting hung in the same code area. Of course including your NIFi 2.0.0-Mx specific version, OS version, and Java version is also important in that Apache Jira. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-10-2024
06:09 AM
@sha257 Did you restart your NiFi after modifying the logback.xml? NiFi executes the ldap-user-group-provider during startup, so you should see DEBUG output in the nifi-app.log at that time and each time the sync interval happens. As far as ldap sample, I would need to see a ldap search group output that contains "member" entries for users you expect to see. Thanks, Matt
... View more
12-10-2024
06:04 AM
@Dahlgren @mohdriyaz Before launching did you check to make sure all the previous running NiFi java process were gone? Can you share more detail about your environments (Windows version and Java version)? Any details on the Java process you have to terminate after quitting with ctrl-c in the CMD window? Thanks, Matt
... View more
12-09-2024
12:25 PM
@mohdriyaz These are just INFO level log lines. What logging did you see in the NiFi bootstrap.log? Does the process die or does just appear hung and not progressing to point where UI is becomes available? On the times where NiFi did start, what log lines are seen immediately after above are output? Thanks, Matt
... View more
12-09-2024
09:41 AM
@sha257 You do NOT need to conduct both a user search and group search in order to get your users and groups. I suggested that you try only executing a group sync from which member users will be looked up. I see that you still have the a user search being executed. Unset the following properties: <property name="User Object Class">person</property>
<property name="User Search Filter">(objectClass=*)</property> leave only the following user sync properties set: <property name="User Search Scope">SUBTREE</property>
<property name="User Identity Attribute">sAMAccountName</property> You can also add the following logger to the NiFi logback.xml file to see what user and group strings are being returned by this provider: <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/> Without a group ldap entry example, I can only make suggestions. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-06-2024
01:11 PM
1 Kudo
@sha257 NiFi is failing to startup because the execution of the ldap-user-group-provider during startup failed to complete successfully. Did you set the page size to 500 in the ldap-user-group-provider? There really is no harm in syncing groups that have no members. They can't be used for authorization since a group can't authenticate into NiFi. All that matters is that the groups with members are getting synced and those members are listed for the group. Then group based authorizations can be established to control access for those member identities. Keep in mind that the user and group identity strings being synced are loaded into NiFi heap memory. So you want to make sure your group search filter is syncing only the few groups containing user who will need to access your NiFi. So instead of using wildcards, declare the specific CNs for only the groups containing users that need to access your NiFi. Typically NiFi specific groups are created in LDAP/AD for managing authorized access to NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-05-2024
07:11 AM
@sha257 NiFi's ldap-user-group-provider does not support nested groups (NIFI-8035) Based on what you have shared, your user "W0YZ1" is not a direct memberOf group: AG-X-SAMPLE-ADMIN The user is a member of a bunch of other groups and I am guessing that one of more of these groups are a member of the above group. And since your user search filter is only going to return users that are a direct memberOf: CN=AG-X-SAMPLE-ADMIN,OU=Groups,DC=corp1,DC=ad1,DC=xyz,DC=net NiFi's ldap-user-group-provider is not going to return the sample user you shared above. Something you may want to try here: It is not required that both the user and group search properties are configured in order to get users and groups returned. In your case I would suggest only performing the group sync to see what you get. The following properties should be adjusted: <property name="Page Size">500</property>
<property name="Sync Interval">30 mins</property>
<property name="User Search Base"></property>
<property name="User Object Class"></property>
<property name="User Search Scope">SUBTREE</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">sAMAccountName</property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">OU=Groups,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">SUBTREE</property>
<property name="Group Search Filter">(|(cn=AG*)(cn=UG*))</property>
<property name="Group Name Attribute">sAMAccountName</property>
<property name="Group Member Attribute">member</property>
<property name="Group Member Attribute - Referenced User Attribute"></property> The above changes assume that your groups have "member" attribute. I recommend always setting a page size of either 500 to avoid missing returns for large queries. I do not recommend re-syncing users and groups every 2 mins as you had originally set. This adds unnecessary load on CPU. Keep in mind that all user and group identities synced are loaded into NiFi's heap memory. The above setup will return all groups starting from the group search base that start with yoru configured search filter. From each returned group all the member lines will be returned which should contain the full DN for user members. Those returned DNS are then looked up to return the "sAMAccountName" string for each user member. I saw you were using full DNs for yoru groups previously (which is ok). The sAMAccountName string for your groups will also be used as the group identity. You can also out the ldap-user-group-provider class in debug in the NiFi logback.xml to get output of the user and group identity strings synced with ease sync execution: org.apache.nifi.ldap.tenants.LdapUserGroupProvider Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more