Member since
07-30-2019
3421
Posts
1624
Kudos Received
1010
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 39 | 01-13-2026 11:14 AM | |
| 178 | 01-09-2026 06:58 AM | |
| 503 | 12-17-2025 05:55 AM | |
| 564 | 12-15-2025 01:29 PM | |
| 561 | 12-15-2025 06:50 AM |
03-15-2017
12:43 PM
1 Kudo
@nyakkanti FlowFiles consist of FlowFile attributes and FlowFile content. - FlowFile attributes are kept in heap during processing and persisted to the FlowFile repository. - FlowFile content is kept in claims within the content repository. A claim is moved is moved to archive once their no longer exists any FlowFiles still active anywhere in your dataflow pointing at it. Archiving is enabled by default but can be disabled in the nifi.properties file: nifi.content.repository.archive.enabled=true If you disable archiving, the claim is purged from NiFi's content repository rather the being archived. What is important to understand is how claims work. By default in the nifi.properties file, claims can contain up to 100 FlowFiles or a min 10 MB of data (whichever occurs first). So a claim will not be purged until every piece of content in that claim has completed processing. As long as just one piece of content in that claim is still referenced, the entire claim will still exist in the content repository. As far as FlowFile attributes are concerned, they are persisted in NiFi provenance based on the configured retention in the nifi.properties file. You can perform provenance searches within NiFi to return FlowFile history and look at the attributes of those FlowFiles at any point int their lineage. Thanks, Matt
... View more
03-15-2017
12:31 PM
@mayki wogno Are you issuing commands against the rest api or are you trying to make a change within the UI when this occurs? Sounds like multiple changes being made against the same component at the same time. Each component has a revision number so that two people can't make changes to the exact same component at the same time. So when the second change is applied using the same revision as the first request which was successful, you get these responses. Two ways this can occur... 1. Two authenticated users are making a change to the configuration of the same processor. User 1 hits apply and that change is applied. User 2 then hits apply and a conflict response occurs from the first node that receives the request. 2. Multiple rest api call are being made against the same component without updating the revision number in the subsequent rest api calls. As far as node going down.... Do you mean you lose the api and have to refresh the browser? or does the cluster completely go down forcing you to restart nodes to get nodes to rejoin cluster? Thanks, Matt
... View more
03-14-2017
06:25 PM
1 Kudo
@Raj B
Thank you... Sometimes the most important piece of information is in the fine details. Other give away that it was clustered was that both FlowFiles in that queue had same position "1". Two FlowFiles in the same queue on the same node cannot occupy the same position.
... View more
03-14-2017
06:10 PM
@Constantin Stanca @Mohammed El Moumni Queue thresholds are per node and will cause a queue to no longer accept additional FlowFiles, It will not prevent downstream processor from processing FlowFiles that are already in that queue.
Had he received two 700MB CSV files on one node, then the 1GB threshold would have been exceeded thus preventing any additional FlowFiles from entering that queue (including the corresponding 70 byte header files). In that case you would be stuck, since merge would not have the files even on a single node needed to merge a bin. Thanks, Matt
... View more
03-14-2017
05:48 PM
4 Kudos
@Mohammed El Moumni Each Node in a NiFi cluster runs its own copy of the dataflow and works on its own set of FlowFiles. Looking at the screenshot you have above of your queue list, you can see that the two FlowFiles are not on the same node. So each node is running a MergeContent processor and each node is waiting for another FlowFile to complete their bins. You will need to look back earlier in your dataflow to see how your data is being ingested by your nodes to make sure that the matching sets of files end up on the same node for merging. Thanks, Matt
... View more
03-14-2017
02:46 PM
1 Kudo
@matthew N Any time you see the following: Unable to perform the desired action due to insufficient permissions. Contact the system administrator. You are having an authorization issue and not an authentication issue. If you look tail your nifi-app.log while you try to login again, you will see two lines output. One will state successful authentication. 2017-03-14 14:36:43,402 INFO [NiFi Web Server-5418] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<cn=Matt,ou=People,dc=sme,dc=nifi><CN=nifi-11.openstacklocal, OU=SME, O=NIFI, L=Baltimore, ST=MD, C=US><CN=nifi-13.openstacklocal, OU=SME, O=HWX, L=Baltimore, ST=MD, C=US>) GET https://nifi-11.openstacklocal:9091/nifi-api/flow/config (source ip: x.x.x.x)
2017-03-14 14:36:43,402 INFO [NiFi Web Server-5418] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=Matt,ou=People,dc=sme,dc=nifi Then you will see a line that contains the above response. You need to make sure that the DN shown in this log matches exactly with what you had configured for your "Initial Admin Identity". You entry in LDAP may have uppercase CN, OU, ..., but in your nifi use log you may see lower case cn=, ou, ... It must match what is in the user.log. You can also take a look in the users.xml file that NiFi generates to make sure your user exists. It will have a UUID assigned to your user. Take that UUID and check which resources that UUID has been granted access to in the authorizations.xml file. For an ADMIN user, you will need at a minimum: /flow
/controller
/policies
/tenants Thanks, Matt
... View more
03-14-2017
02:18 PM
@Christophe Vico
Confirm the DN of the PrivateKeyEntry in the keystore of all you nodes. # keystore -v --list -keystore <keystorename> Also confirm the patterns have been set on all nodes in the nifi.properties file. All it takes is a slight difference (extra space, missing space, case sensitivity, etc...) Thanks, Matt
... View more
03-14-2017
02:14 PM
@Christophe Vico Where in the UI did you look for the controller service? Controller services that are used by processors such as HIveConnectionPool are created within process groups and not at the core level found in the hamburger menu. You can access controller services within process groups by selecting the process group and then clicking on the "gear" icon in the operate panel:
Thanks, Matt
... View more
03-13-2017
06:32 PM
1 Kudo
@matthew N Accessing the NiFi UI requires to things to be successful: 1. User Authentication: You appear to be using LDAP to handle this part. 2. User Authorization: By default NiFi uses its internal file based authorizer (Configured in authorizers.xml) If an authenticated user lacks sufficient authorization to access a NiFi resource, you will see the "Unable to perform the desired action due to insufficient permissions. Contact the system administrator." response from NiFi. In order for an authenticated user to see the NiFi UI, they must at a minimum be granted the "view the user interface" access policy. Whichever user was configured as your "Initial admin Identity" will need to access the UI and add additional users and access policies for those users. Also keep in mind that NiFi generates the users.xml and authorizations.xml files only once the first time your NiFi is started securely. If you update who your initial admin identity is later, it will not get updated if these files already exist. If this is the first time setting up a new system, simply delete the users.xml and authorizations.xml files and restart NiFi. They will then be created again based on the current configurations in the authorizers.xml. Before updating your initial admin identity in the authorizers.xml file, I suggest looking in your nifi-user.log to versify the exact string being passed to the authorizer. It must match exactly since it is case sensitive and spaces also count as valid characters. (for example: CN= is not the same as cn=) The string you see output in the nifi-user.log is what will be passed to the authorizer. Thanks, Matt
... View more
03-13-2017
03:28 PM
@matthew N Sharing the full stack trace that goes along with the above NPE will help identify where the issue is occurring.
... View more