Member since
07-30-2019
3114
Posts
1552
Kudos Received
905
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
190 | 12-05-2024 06:38 AM | |
188 | 11-22-2024 05:50 AM | |
162 | 11-14-2024 01:03 PM | |
168 | 11-12-2024 10:46 AM | |
162 | 11-12-2024 06:17 AM |
12-10-2024
11:16 AM
@Dahlgren - Make sure you are using a Java 21 JDK and not a Java 21 JRE. - Try excluding the NiFi directory to include all the repositories from your Virus scanning software and see if that helps resolve the startup issue. Here is an Apache NiFi slack thread discussing this same issue: https://apachenifi.slack.com/archives/C0L9UPWJZ/p1731963549046199 Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-10-2024
09:43 AM
1 Kudo
@Dahlgren I conversation with a committer to the Apache NiFi product and he stated that this is a known issue they are aware of and investigating. There is no Apache NiFi jira that I can find specifically for this issue. . The best suggestion I can make at this point is maybe raising an Apache NiFi jira and if possible share thread dumps (3 dumps spaced 5 minutes apart). Looking at what threads are not progressing from one thread dump to the next may provide clues as to where specifically it is getting hung in the startup code. That should narrow down the scope of investigation and confirm that all experiencing this issue are getting hung in the same code area. Of course including your NIFi 2.0.0-Mx specific version, OS version, and Java version is also important in that Apache Jira. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-10-2024
06:09 AM
@sha257 Did you restart your NiFi after modifying the logback.xml? NiFi executes the ldap-user-group-provider during startup, so you should see DEBUG output in the nifi-app.log at that time and each time the sync interval happens. As far as ldap sample, I would need to see a ldap search group output that contains "member" entries for users you expect to see. Thanks, Matt
... View more
12-10-2024
06:04 AM
@Dahlgren @mohdriyaz Before launching did you check to make sure all the previous running NiFi java process were gone? Can you share more detail about your environments (Windows version and Java version)? Any details on the Java process you have to terminate after quitting with ctrl-c in the CMD window? Thanks, Matt
... View more
12-09-2024
12:25 PM
@mohdriyaz These are just INFO level log lines. What logging did you see in the NiFi bootstrap.log? Does the process die or does just appear hung and not progressing to point where UI is becomes available? On the times where NiFi did start, what log lines are seen immediately after above are output? Thanks, Matt
... View more
12-09-2024
11:53 AM
@laurie_mcintosh The Wait/Notify processors have existed a long time in Apache NiFi and are complicated to use correctly. I strongly encourage you to read the following about the Wait/Notify: https://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/ https://pierrevillard.com/2018/2018-06-27-nifi-workflow-monitoring-wait-notify-pattern-with-split-and-merge/ With advancements in Apache NIFi and depending on your use case, there may be easier ways to handle your use case. Process Group FlowFile Concurrency in conjunction with Process Group Outbound Policy is a newer option that has often been used in place of wait/notify setups. Unrelated... If you are running a NiFi cluster and the target directory is a network mount directory, you should be using the ListFile/FetchFile processors in place of the GetFile. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-09-2024
09:41 AM
@sha257 You do NOT need to conduct both a user search and group search in order to get your users and groups. I suggested that you try only executing a group sync from which member users will be looked up. I see that you still have the a user search being executed. Unset the following properties: <property name="User Object Class">person</property>
<property name="User Search Filter">(objectClass=*)</property> leave only the following user sync properties set: <property name="User Search Scope">SUBTREE</property>
<property name="User Identity Attribute">sAMAccountName</property> You can also add the following logger to the NiFi logback.xml file to see what user and group strings are being returned by this provider: <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/> Without a group ldap entry example, I can only make suggestions. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-06-2024
01:11 PM
1 Kudo
@sha257 NiFi is failing to startup because the execution of the ldap-user-group-provider during startup failed to complete successfully. Did you set the page size to 500 in the ldap-user-group-provider? There really is no harm in syncing groups that have no members. They can't be used for authorization since a group can't authenticate into NiFi. All that matters is that the groups with members are getting synced and those members are listed for the group. Then group based authorizations can be established to control access for those member identities. Keep in mind that the user and group identity strings being synced are loaded into NiFi heap memory. So you want to make sure your group search filter is syncing only the few groups containing user who will need to access your NiFi. So instead of using wildcards, declare the specific CNs for only the groups containing users that need to access your NiFi. Typically NiFi specific groups are created in LDAP/AD for managing authorized access to NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-05-2024
07:11 AM
@sha257 NiFi's ldap-user-group-provider does not support nested groups (NIFI-8035) Based on what you have shared, your user "W0YZ1" is not a direct memberOf group: AG-X-SAMPLE-ADMIN The user is a member of a bunch of other groups and I am guessing that one of more of these groups are a member of the above group. And since your user search filter is only going to return users that are a direct memberOf: CN=AG-X-SAMPLE-ADMIN,OU=Groups,DC=corp1,DC=ad1,DC=xyz,DC=net NiFi's ldap-user-group-provider is not going to return the sample user you shared above. Something you may want to try here: It is not required that both the user and group search properties are configured in order to get users and groups returned. In your case I would suggest only performing the group sync to see what you get. The following properties should be adjusted: <property name="Page Size">500</property>
<property name="Sync Interval">30 mins</property>
<property name="User Search Base"></property>
<property name="User Object Class"></property>
<property name="User Search Scope">SUBTREE</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">sAMAccountName</property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">OU=Groups,DC=corp1,DC=ad1,DC=xyz,DC=net</property>
<property name="Group Object Class">group</property>
<property name="Group Search Scope">SUBTREE</property>
<property name="Group Search Filter">(|(cn=AG*)(cn=UG*))</property>
<property name="Group Name Attribute">sAMAccountName</property>
<property name="Group Member Attribute">member</property>
<property name="Group Member Attribute - Referenced User Attribute"></property> The above changes assume that your groups have "member" attribute. I recommend always setting a page size of either 500 to avoid missing returns for large queries. I do not recommend re-syncing users and groups every 2 mins as you had originally set. This adds unnecessary load on CPU. Keep in mind that all user and group identities synced are loaded into NiFi's heap memory. The above setup will return all groups starting from the group search base that start with yoru configured search filter. From each returned group all the member lines will be returned which should contain the full DN for user members. Those returned DNS are then looked up to return the "sAMAccountName" string for each user member. I saw you were using full DNs for yoru groups previously (which is ok). The sAMAccountName string for your groups will also be used as the group identity. You can also out the ldap-user-group-provider class in debug in the NiFi logback.xml to get output of the user and group identity strings synced with ease sync execution: org.apache.nifi.ldap.tenants.LdapUserGroupProvider Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-05-2024
06:38 AM
@bgumis One way you can accomplish what you want to do is using an UpdateAttribute processor and RouteOnAttribute processor between your PutEmail and InvokeHTTP processors. The UpdateAttribute would be used to add an attribute to the FlowFile with the current time in milliseconds using NiFi Expression Language (NEL) statement: ${now():toNumber()} The success relationship of the update Attribute processor would be routed to the RouteOnAttribute processor. In the RouteOnAttribute processor you setup a new property that compares the records timestamp added to FlowFile by UpdateAttribute against the current time in milliseconds to see if 24 hours has passed yet. If true, the FlowFile will route to the expression property name relationship. If false, the FlowFile will route to the "unmatched" relationship which can be looped back on RouteOnAttribute. The "unmatched" relationship should be configured to use "retry" to trigger penalization of FlowFiles. This helps limit cpu usage by slowing how often the FlowFile is re-processed by penalizing the FlowFile on each retry. The "Retry Max Back Off Period" would translate into the max time beyond 24 hours a FlowFile may get routed to the invokeHTTP. I assume 24 hours is a min wait period and not a hard limit. The NEL statement used in RouteOnAttribute would look like this: ${now():toNumber():minus(${emailTime}):ge('86400000')} The other option would require a custom script to set penalization of 24 hours on each FlowFile after the putEmail processor. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more