Member since
07-30-2019
3471
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 175 | 06-03-2026 06:06 PM | |
| 471 | 05-06-2026 09:16 AM | |
| 902 | 05-04-2026 05:20 AM | |
| 515 | 05-01-2026 10:15 AM | |
| 640 | 03-23-2026 05:44 AM |
09-08-2021
11:16 AM
@wbivp When NiFi is configured to use the ldap-user-group-provider, it must be able to successfully execute that provider during startup to generate a list of users and groups within NiFi. The exception points that that provider being unable to execute successfully. The exception in the logs shows: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C090439, comment: AcceptSecurityContext error, data 52e, v4563] This points at an authentication issue when trying to communicate with your ldap server. (misconfiguration int the provider, bad Manager or Manager password provided) From the NiFi host can you make run a ldapsearch query against your ldap server using all the same configured values from your provider? Without your authorizers.xml file, it would be difficult for me to point out any other misconfigurations if present. If you found the provided response(s) assisted with your query, please take a moment to login and click on "Accept as Solution" below each solution that helped you. Thank you, Matt
... View more
09-02-2021
10:21 AM
@wbivp Within Ranger you can authorizer users and/or groups to the policies you define. The Ranger plugin with the NiFi service runs in the background within NiFi that connects with Ranger to download the latest set of policies. What is provided by Ranger is simply user(s) A, B, C strings and/or group(s) X, Y, Z strings are authorized read and/or write to NiFi Resource Identifier(s). There is nothin in what is downloaded from Ranger that will tell NiFi as the client what users belong to group(s) X, Y, or Z. This means that NiFi itself needs to be aware of these associations. This is why in the nifi-user.log you see the following: o.a.n.w.a.c.AccessDeniedExceptionMapper identity[test.username], groups[] does ... This log line tells us that NiFi is unaware of any groups the the authenticated user string "test.username" is a member. If NiFi was aware the "groups[]" in this log line would show a comma separated list of all these group strings. NiFi offers numerous user-group-providers that can be added to the authorizers.xml that allow these associations between user and groups to be set. Your authorizers.xml file shared contains the "cm-user-group-provider" (only used to assign NiFi node hostnames to a group string "nifi") and the "file-user-group-provider" [1] which gives users a way of manually adding group strings and associating users to that group directly from the NiFi UI. So with your current setup, you would login as your authorized user, go to the NiFi Global Menu, and then select "users". This will open the NiFi Users UI where you should see your initial admin user which you defined in your file-user-group-provider. You would need to click on the icon to add additional users and groups manually. Adding users and groups here has nothing to do with authentication. You are using this Ui to establish user to group associations. So I would start by creating a new group. The Identity string used must match case sensitive the exact group string as seen in Ranger. Then you can start adding your user strings (must match user strings case sensitive as seen in Ranger) As you add users you will be able to select the group(s) you added as those that user should be associated with. Using above as an example, NiFi would then associate user string "JoeSmith" with group string "admins". To see what other user-group-providers exist within your NiFi version, you should look at the "Admin Guide" found under help within your NiFi's embedded documentation access via the UI. A very commonly used user-group-provider is the "ldap-user-group-provider" [2] which can be used to sync user and groups strings from LDAP/AD and establish the associations between them based on what is in LDAP/AD. [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider If you found these responses assisted with your query, please take a moment to login and click on "Accept as Solution" below each post. Thank you, Matt
... View more
09-02-2021
09:18 AM
@Justee First thing I would do is add a new Attribute on my FlowFile that specifies the year I'd be searching for in the lines contained within the content of that FlowFile. (optional) For example adding an attribute "year" with a value of "1995". In the routeText processor, I'd then be able to use NiFi Expression Language (NEL) in my java regular expression as supported by this processor component: ^\|(.*?)\|(.*?)\|${year}\|(.*?)$ The above java regular expression will match on lines that begin with a pipe "|" followed by a non greedy wildcard match of one or more character until the very next pipe "|", then again for field 2, then for field three I used NEL which resolves to "1995", and then finally i match via wildcard the remainder of the line. Of course you could simply put "1995" in place of "${year}" in the above regex. The routeText processor component configuration would look like this: The result would be two FlowFiles. One FlowFile would be routed to the relationship "1995" (based on property name used) which would have content only containing lines with "1995". The second FlowFile would route to the "unmatched" relationship and would contain all the non-matching lines ( you may to choose to just auto-terminate this relationship if you don't care about these lines). If you found these responses addressed your query, please take a moment to login and click on "Accept as Solution" below each response that helped you. Thank you, Matt
... View more
09-01-2021
12:07 PM
@wbivp Is the exact user string you see in the nifi-user.log the same (case sensitive) as what is set in Ranger? is this user string authorized for the /flow policy? Thank you, Matt
... View more
09-01-2021
12:04 PM
@wbivp The nifi-user.log output you shared indicates that the user string "cdpadmindev@DOMAIN" has not been authorized to against the NiFi /flow resource identifier (View the user interface). The authorizers.xml configuration file controls how user and/or group based authorizations are setup and managed. So first things is what is configured in that file? Is it using a file based authorizer or external Ranger based authorizer? Is it using any user group providers? Did you configure and initial admin identity? if so, does that initial admin identity string exactly match your user string from the nifi-user.log you shared? If using the file based authorizer, you should have in the authorizers.xml both the "file-user-group-provider" and the "file-access-policy-provider". These providers are used to create the users.xml and authorizations.xml file on startup if they do NOT already exist. so even if you do have the initial admin set correctly, if NiFi was started previously before the user string was set, you would have existing users.xml and authorizations.xml files without this user and the required admin policies set. (remove these files and restart NiFi so they are generated again). If using Ranger as your authorizer, you need to make sure that the user string exactly as you see in the nifi-user.log exists as a user in Ranger and the the NiFi service plugin in ranger is setup and has the correct NiFi resource identifier policies authorized for that user. Here is a reference article on those Ranger based NiFi policies: https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-01-2021
11:47 AM
1 Kudo
@Justee ListSFTP only generate a FlowFile with attributes/metadata about the file on the SFTP processor. It does not look at the content itself. So your filtering options are limited to what is in those generated attributes. The FetchSFTP processor uses these attributes/metadata to retrieve the actual content and add it to the existing FlowFile produced by the ListSFTP processor. So unfortunately you would need to fetch the all files and then keep on those that contain the desired value in the third field. You may want to look at the RouteText [1] processor for handling these Files after they are the content is fetched. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.RouteText/index.html If you found this response addressed your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
08-24-2021
10:59 AM
@smartraman This can also be accomplished through a different and more complex configuration of the ReplaceText processor: Using below input content example: {
"TOT_NET_AMT" : "[\"55.00\"]",
"H_OBJECT" : "File",
"H_GROSS_AMNT" : "[\"55.00,58.00\"]",
"TOT_TAX_AMT" : "[9.55]"
} I would set up the replaceText processor as follows: Instead of just searching for those character patterns and replacing them with nothing, I break entire input line-by-line in to a series of capture groups. That way I can omit the capture groups matching the patterns you want removed ([ or [\" or \"] or ]) and then manipulate the capture group containing a possible comma separated list, so that only the last value in that list is returned. I used below java regular expression which results in 5 capture groups: (.*?)([\Q[\E]\\\"|[\Q[\E])(.*?)(\\\"[\Q]\E]|[\Q]\E])(.*?)$ I then used the following Replacement Value in which I used NiFi expression language against the 3rd capture group. If that capture group does not contain any commas, the entire string is returned. With example above and this configuration, you end up with the following new content: {
"TOT_NET_AMT" : "55.00",
"H_OBJECT" : "File",
"H_GROSS_AMNT" : "58.00",
"TOT_TAX_AMT" : "9.55"
} If you found this helped with your latest query, please take a moment to login and click on "Accept as Solution" below this response. Thank you, Matt
... View more
08-13-2021
12:36 PM
@mickt Correct.... This recent bug fix is fixed in the current master and will be part of the next Apache NiFi 1.15 release.
... View more
08-13-2021
12:26 PM
@dupuy_gregory Can you share the configurations for both your GetHDFS processors?
... View more
08-13-2021
12:20 PM
@SolidSnake Site-To-Site is not a proxy. Perhaps a diagram of what you are trying to accomplish may make it easier to follow your use case? Could you share such a diagram that shows flow of connections and what is happening now versus a diagram that shows what you want to happen. You have a 2 node nifi cluster using invokeHTTP processor making a request to a handleHttpRequest processor on the standalone NiFi (1 node cluster) and then the FlowFile generated by the HandleHttpRequest processor is end via Site-To-Site to the NiFi cluster that then routes to a HandleHttpResponse? Maybe i am not following very well, but I don't understand why you are doing that if I understand your flow correctly? Why isn't the same node that receives the the request via the HandleHttpRequest processor also sending the response via a HandleHttpResponse processor? Thanks, Matt
... View more