Member since
07-30-2019
3467
Posts
1641
Kudos Received
1016
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 131 | 05-04-2026 05:20 AM | |
| 437 | 03-23-2026 05:44 AM | |
| 328 | 02-18-2026 09:59 AM | |
| 576 | 01-27-2026 12:46 PM | |
| 1009 | 01-20-2026 05:42 AM |
12-28-2020
10:33 AM
@adhishankarit You dataflow screenshot does not reflect the entire dataflow you are then trying to describe making this use case hard to follow. 1. Your flow starts with a single zip file? 2. You unzip that file to produce numerous output FlowFiles? 3. You use load balanced connections to distribute all the produced FlowFiles across all nodes in your cluster? 4. Then you modify content of FlowFile using convertAttribitesToJson processor (Destination=flowfile-content)? Looks like you route the "success" relationship twice from this processor which means you have cloned your FlowFiles. Why? 5. One of these connection looks like it uses a Load Balance connection (how is it configured?) to feed a MergeContent. MergeContent can not merge across multiple nodes (can only merge FlowFiles on same node. How is MergeContent configured? Your desire output does not look like Json, but you are using AttributesToJson processor? 6. Where do the "failure" FlowFiles get introduced in to this dataflow? When you unpack your original FlowFile each produced FlowFile will have new attributes set on it to include segment.original.filename, fragment.identifier, fragment.count, and fragment.index. These attributes can be used with the "defragment" merge strategy in MergeContent. So I would avoid cloning FlowFiles post unpack. Process each FlowFile in-line. When you encounter a "failure", set an attribute on these FlowFiles only that states a failure occurred (successfully processed FlowFiles should not have this unique attribute). Then use MergeContent and set keep all Unique Attributes. This will allow the unique attribute if exists on any one FlowFile to show up on the output merged FlowFile will not work if same attribute exists on multiple FlowFiles with different values). Now after merge you can modify the content again using ReplaceText processor configured with Append to add the first line with overall status of this file from that unique attribute you preserved through the merge. Also not following statement: "also noticed that if there is a delay in processing " Hope this helps, Matt
... View more
12-24-2020
07:35 AM
@adhishankarit There is nothing you can pull form the NiFi Rest-API that is going to tell you about successful outcomes form processor execution on a FlowFile. Depending on data volumes, this also sounds like a resource expensive endeavor. That being said, NiFi does have a Site-To-SIte (S2S) Bulletin reporting task. When a processor throws an error it will produce a bulletin and this reporting task can capture those bulletins and send them via NiFi's S2S protocol to another NiFi instance directly into a dataflow were you can handle via dataflow design however you like. Only way you can get INFO level logs in to bulletins is by setting the bulletin level to INFO on all you processors. This only works if you have also configured your NiFI logback.xml so that all NiFi components log at the INFO level as well. Downsides to this: 1. Every processor would display the red bulletin square in upper right corner of processor, This makes using this to find components that are having issues difficult 2. This results in a lot of INFO level logging to the nifi-app.log. You mention edge nodes. You could setup a tailFile processor that tails the nifi-app.log and then send via a dataflow that log data via FlowFiles to some monitor NiFi cluster where another dataflows parses those records via a partitionRecord processor by log_level and then routes based on that log_level for additional handling/notification processing. Downside here: 1. Since you what to track success, you still need INFO level logging enabled for all components. This means even this log collection flow is producing log output. So large logs and logs being written to even when actual data being processed in other flows is not happening. NiFi does have a master bulletin board which you could hit via the rest-api, but this does not get you past the massive logging you may be producing to monitor success. https://nifi.apache.org/docs/nifi-docs/rest-api/index.html Hope this gives you some ideas, Matt
... View more
12-23-2020
07:46 AM
@Nyk That is correct, dynamically added properties are all of type non-sensitive. You would need to build a custom processor with static configurable properties that have a PropertyDescriptor with ".sensitive(true)". I am not a developer myself, but you may find this resource useful: https://itnext.io/deep-dive-into-a-custom-apache-nifi-processor-c2191e4f89a0 If you found my answer addresses your question, please click on "Accept as Solution" below the answer. Hope this helps you, Matt
... View more
12-23-2020
07:27 AM
1 Kudo
@murali2425 I was not able to reproduce the missing attributes in the content of a produced FlowFile from the AttributesToJson processor. What version of NiFi are you using? Did you inspect the attributes on the FlowFile in the immediate connection feeding the AttributesToJson before starting that processor? Your desired output from the AttributesToJson seems to be very specific and not include all attributes including the core attributes anyway. My suggestion would be to use an UpdateAttribute processor just before your AttributesToJson processor to build the specific attributes you want to have in your produced Json output content.
You would then add two custom dynamic properties where you would use NiFi Expression Language to populate the values from other attributes/metadata on the source FlowFile:
You could then configure your AttributesToJson processor to build the JSON content using only those two new attributes you just constructed:
Keep in mind that the AttributesToJson processor will add attributes to the Json in Lexicographical order. So if you want the uuid before the filepath, you will need to adjust the property names used in the UpdateAttribute processor. For example "auuid" instead of "myuuid" so that it comes before "filepath" in order. Hope this helps,
Matt
... View more
12-23-2020
06:34 AM
@Nyk You'll need to provide more context around your use case to get a good answer. It is not clear what you are trying to accomplish here. - Are you trying to build your own custom NiFi processor where you want to define some configurable properties as type sensitive? - Are you trying to use the "+" icon to add a new dynamic property to an existing NiFi processor and hide/encrypt the value set in that property? In this case, that is not possible since all dynamic properties are nonsensitive. The use of invokeScriptedProcessor you saw was a recommendation made to sue it rather than ExecuteScript processor. That is because you can define properties via the invokeScriptedProcessor custom script you write additional processor properties to be used within that processor only as sensitive. But looks like there may be some issues around that ability outlined here: https://issues.apache.org/jira/browse/NIFI-7012 Hope this helps, Matt
... View more
10-19-2020
08:34 AM
@sarath_rocks55 If you are looking for assistance with an issue, please post a "question" in the community rather than adding a comment to a community article. Comments on community articles should be about the article content only. Thank you
... View more
09-14-2020
09:20 AM
2 Kudos
@Umakanth From your shared log lines we can see two things: 1. "LOG 1" shows "StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902" and "LOG 2" shows "Successfully sent [StandardFlowFileRecord[uuid=f74eb941-a233-4f9e-86ff-07723940f012". This tells us these "RandomFile1154.txt" are two different FlowFiles. So does not look like RPG sent the same FlowFile twice, but rather sent two FlowFiles with each referencing the same content. I am not sure how you have your LogAttribute processor configured, but you should look for the log output produced by these two uuids to learn more about these two FlowFiles. I suspect from your comments you will only find one of these passed through your LogAttribute processor. 2. We can see from both logs that the above two FlowFiles actually point at the exact same content in the content_repository: "LOG 1" --> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237] "LOG 2" --> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1109014, length=237],offset=0,name=RandomFile1154.txt,size=237] This typically happens when a FlowFile becomes cloned somewhere in your dataflow. For example: when a relationship from a processor is defined twice. Since you saw that GetFile only ingested file once, that rules out GetFile as the source of this duplication. But had it been GetFile, you would have not seen identical claim information. LogAttribute only has a single "success" relationship, so if you had drawn two connections with "Success" relationship defined in both, you would have seen duplicates of every ingested content. So this seems unlikely as well. Next you have your PutFile processor. This processor has both "success" and "failure" relationships. I suspect the "success" relationship is assigned to the connection going to your Remote Process Group" and the "failure" relationship assigned to a connection that loops back on the PutFile itself(?). Now if you had accidentally drawn the "failure" connection twice (one may be stack on top of the other), anytime a FlowFile failed in the putFile it would have been routed to one failure connection and cloned to other failure connection. Then on time they both processed successfully by putFile and you end up with the original and clone sent to your RPG. Hope this helps, Matt
... View more
08-14-2020
06:37 AM
@DivyaKaki The exception implies that the complete trust chain does not exist to facilitate a successful mutual TLS handshake between this NiFI and the target NiFi-Registry. NiFi uses the keystore and truststore configured in its nifi.properties and NiFi-Registry uses the keystore and truststore configured in its nifi-registry.properties files. Openssl can be used to public certificates for the complete trust chain: openssl s_client -connect <nifi-registry-hostname>:<port> -showcerts openssl s_client -connect <nifi-hostname>:<port> -showcerts for each public cert you will see: -----BEGIN CERTIFICATE-----
MIIESjCCAzKgAwIBAgINAeO0mqGNiqmBJWlQuDANBgkqhkiG9w0BAQsFADBMMSAw
HgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMjETMBEGA1UEChMKR2xvYmFs
U2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0xNzA2MTUwMDAwNDJaFw0yMTEy
MTUwMDAwNDJaMEIxCzAJBgNVBAYTAlVTMR4wHAYDVQQKExVHb29nbGUgVHJ1c3Qg
U2VydmljZXMxEzARBgNVBAMTCkdUUyBDQSAxTzEwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDQGM9F1IvN05zkQO9+tN1pIRvJzzyOTHW5DzEZhD2ePCnv
UA0Qk28FgICfKqC9EksC4T2fWBYk/jCfC3R3VZMdS/dN4ZKCEPZRrAzDsiKUDzRr
mBBJ5wudgzndIMYcLe/RGGFl5yODIKgjEv/SJH/UL+dEaltN11BmsK+eQmMF++Ac
xGNhr59qM/9il71I2dN8FGfcddwuaej4bXhp0LcQBbjxMcI7JP0aM3T4I+DsaxmK
FsbjzaTNC9uzpFlgOIg7rR25xoynUxv8vNmkq7zdPGHXkxWY7oG9j+JkRyBABk7X
rJfoucBZEqFJJSPk7XA0LKW0Y3z5oz2D0c1tJKwHAgMBAAGjggEzMIIBLzAOBgNV
HQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMBIGA1Ud
EwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFJjR+G4Q68+b7GCfGJAboOt9Cf0rMB8G
A1UdIwQYMBaAFJviB1dnHB7AagbeWbSaLd/cGYYuMDUGCCsGAQUFBwEBBCkwJzAl
BggrBgEFBQcwAYYZaHR0cDovL29jc3AucGtpLmdvb2cvZ3NyMjAyBgNVHR8EKzAp
MCegJaAjhiFodHRwOi8vY3JsLnBraS5nb29nL2dzcjIvZ3NyMi5jcmwwPwYDVR0g
BDgwNjA0BgZngQwBAgIwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly9wa2kuZ29vZy9y
ZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEAGoA+Nnn78y6pRjd9XlQWNa7H
TgiZ/r3RNGkmUmYHPQq6Scti9PEajvwRT2iWTHQr02fesqOqBY2ETUwgZQ+lltoN
FvhsO9tvBCOIazpswWC9aJ9xju4tWDQH8NVU6YZZ/XteDSGU9YzJqPjY8q3MDxrz
mqepBCf5o8mw/wJ4a2G6xzUr6Fb6T8McDO22PLRL6u3M4Tzs3A2M1j6bykJYi8wW
IRdAvKLWZu/axBVbzYmqmwkm5zLSDW5nIAJbELCQCZwMH56t2Dvqofxs6BBcCFIZ
USpxu6x6td0V7SvJCCosirSmIatj/9dSSVDQibet8q/7UK4v4ZUN80atnZz1yg==
-----END CERTIFICATE----- Above is just example public cert from openssl command against google.com:443 You will need to make sure that every certificate in the chain when run agains NiFi UI is added to the truststore on NiFi-Registry and vice versa. You'll need to restart NiFi and NiFi-Registry before changes to your keystore or truststore files will be read in. Hope this helps, Matt
... View more
07-15-2020
09:23 AM
@bhara While ldap/AD queries are generally case insensitive by default, NiFi is not case insensitive. So user "bbhimava" and user "Bbhimava" would be treated as two different users. Within the nifi.properties file you can utilize identity and group mapping patterns to manipulate the case of the returned user and groups strings. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties Note that mapping patterns are evaluate in alphanumeric order. First pattern that matches is applied. so make sure less specific patterns like "^(.*)$" are lats to be evaluated. For example: nifi.security.identity.mapping.pattern.dn=<java regex> Above would be evaluated before: nifi.security.identity.mapping.pattern.username=<java regex> When/If a match is found, the corresponding value is applied and transform performed: nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=<NONE, LOWER, or UPPER> Looking at your ldap-user-group-provider configuration, I see the following needed changes: 1. I recommend user set the "Page Size" property. If the number of results exceeds the ldap max default page size, not all results may be returned to NiFi. BY setting a page size, NiFi will asks for results in multiple pages instead of one response with all results. Generally, defaults are either 500 or 1000, so setting a "Page Size" of 500 is safe. 2. The "User Search Scope" and "Group Search Scope" properties should be set to "SUBTREE" and not "sub". 3. The "User Group Name Attribute" property does not support a comma separated list of attributes. Suggest just setting it to "memberOf". Without sample output from your ldap/AD for user bbhimava and one of the groups you are trying to sync based upon, it would be impossible for me to validate any of your other settings. The log line shared below: o.a.n.w.a.c.AccessDeniedExceptionMapper identity[bbhimava], groups[none] does not have permission to access the requested resource. No applicable policies could be found. Returning Forbidden response. indicates that when NiFi looked for user "bbhimava" was looked for by the NiFi authorizer, no associated groups were returned. Which means Ranger policies could only be queried for policies assigned directly to "bbhimava". Adding the following line to your NiFi logback.xml file will give you debug output in your nifi-app.log when the ldap-user-group-provider executes to show you exactly what users and groups were returned based upon your settings and the resulting user/group associates that were discovered. <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/> Hope this helps, Matt
... View more
07-09-2020
01:32 PM
@johannes_meixne The only place you would use "{0}" would be in the ldap-provider used for user authentication. The {0} gets replaced with the username entered in the login UI. The LdapUserGroupProvider however is used by the nifi authorizer to assemble a list of users and group strings (along with the association/membership between those user and groups) which can be used to assign authorizations against. The LdapUserGroupProvider executes on NiFi startup and then on a configured schedule to refresh the synced users and groups. Since it takes no input you can not use "{0}" in the search filter.
... View more