Member since
11-22-2025
13
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1492 | 12-11-2025 03:06 PM |
05-14-2026
04:32 PM
Hi Matt Sorry its taken a few days to post back, we've been running tests and observing behaviour of the clusters. So we have nifi 2.5.0 on a 3 node cluster on ubuntu, and we run zookeeper as the cluster manager. We have a client Windows 11 machine with nifi 2.5.0 on it that sends data files to the cluster, the cluster processes them and sends acknowledgement files ( one each per data file ) back via Nifi to the windows machine. Its at this point where the acknowledgement files get stuck between the cluster and the client machine. We've done belt and braces baselining of zookeeper - looking at logs, making sure NTP had node times correct, memory correct, looked for errors in logs. That came up clean. In the nifi cluster its also 3 node, and what we did last night was delete the S2S output port and that seemed to fix it. But only temporarily - the files are getting stuck again this morning. This port connects to a down stream windows machine with nifi 2.5.0 on it. We have the port opened right up to 100,000 files and 1TB for back pressure. Penalty is 30 seconds. Traffic seems to flow reasonably evenly between all 3 nifi nodes. What we have observed and i don't know if its related, but in our non-clustered lab set up, we observed slow site to site between dockers ( 2 dockers on same ubuntu machine ), and so we had to install nifi onto linux directly which seemed to fix it. Then it slowed down again - so we deleted everything on the canvas and re-created it by importing the canvas backup JSON file we took before wiping everything .It then ran OK and seemed happy enough, with decent throughput at levels we would expect. We also observed files would "disappear" when we had 2 downstream sites connected to the same output port, we think there is a not-so-"round robin" happening, whereby we have files moving to one downstream site and others going to a second site but not in any particular sequence which makes them look like on one site they have "disappeared". We still have files being stuck again. I'm just keen to get to the bottom of it. 🙂 We
... View more
05-12-2026
03:07 PM
We have Nifi 2.5.0 and a problem whereby some files seem to be "stuck" after leaving a 3 node cluster, and get "stuck" in a site-to-site connection queue after the cluster. We ingest 60,000 small files out of a series of folders that are nested to a depth of 10 folders. Most files seem to move through the cluster OK, but we get a few that seem to enter the cluster fine , but then have a problem leaving it. Its different files each time and they sit there for 30-60 mins and eventually move on. Its not the same files each time. The network and compute resources are fine, the memory that Nifi is using is about 47% of the JVM maximum. Disk space doesn't seem to be an issue either. I had a look through the nifi-app.log file and nothing appears to show as an issue. Cluster heartbeat is average 10 milliseconds, GC runs in 50 milliseconds every 30 seconds. Would welcome some suggestions. Thank you in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
02-03-2026
06:59 PM
We were looking at storing the provenance *.gz ( lucene ) files so we could have a record of all activity through Nifi, as our IT security people are fussy about such things. It seems Nifi writes to multiple files at the same time, so this may not be possible, or perhaps there is a better way of doing this? I would welcome thoughts as to how we might do this, or maybe propose a different way to achieve the same logging outcome. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
02-02-2026
07:56 PM
1 Kudo
Hi all, I recently spent a full 2 days over a rainy weekend trying to get a stand alone instance of Nifi v2.5.0 on ubuntu 22.04 trying to integrate with AD, to use AD for Nifi authentication, but with no joy. It almost worked but couldn't get past a certain point. The problem I discovered is there doesn't seem to exist on the internet a simple step by step process on how to set up Nifi with AD/LDAP. My set up is a linux PC called nifi1 and had an old windows 2012 R2 domain controller with LDAP running on it I decided to use. I created a domain called testdomain.local and created a domain user called testuser in the AD User OU that I wanted to use as a nifi user. I also created a service account called svc-nifi also in the User OU that is used to log onto AD as a service. I ran a series of tests using LDAP command line queries from the linux PC whereby the service account could run LDAP queries from linux that proved it could access AD, read the AD info it needed and could connect OK to AD etc. The testuser logon also works fine in logging onto the domain on other PCs. Is there a documented basic step by step process of how to do this on a linux ( or windows ) PC please? I just need the minimum basic working example of nifi.properties, authorizers.xml, uses.xml and authorize.xml etc files please. There are so many variables in each file that make it difficult to work through sequentially, as I dont have expert Nifi knowledge. I wondered if I had maybe skipped a step or something, like : * Do I need to add the linux PC to AD itself first , and if so, how is this set up in the config files? * What do i need to add to the nifi.properties, authorizers.xml file etc to get it to work please? * etc. Any help appreciated - this should really be simple, but its not...... I'm happy to write up a step by process and post it back here once I have it running, to help others. Thanks in advance. 🙂
... View more
Labels:
- Labels:
-
Apache NiFi
12-17-2025
07:13 PM
Thanks Matt, thats a very useful explanation.
... View more
12-17-2025
07:10 PM
Hi I wanted to look into a need to transfer 1 TB files by using chunking in Nifi. Each file also has to have its 20 items of meta data associated with it remain intact so the metadata and the data both survive the breaking up of the file ( chunking ) into 1000 chunks and re-assembling the file at the destination ( de-chunking ). Also, is the meta data for the large file duplicated onto each of the file 1000 chunks or is it a sub-set of the meta data? Someone mentioned nifi passes the file chunks data through JVM memory on its way to the content repository. Can I confirm whether file chunks pass through JVM memory as they are written to the file/content repository for a large file ( or any file for that matter ?) - I was fairly sure they aren't, otherwise the JVM size ( limited by machine RAM ) on the machine would limit reading in of large file data, and that would limit large file transfer speed - is that correct? I'm trying to confirm my understanding of how Nifi handles these large files please. Any help appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
12-13-2025
11:00 PM
Hi, I did some further digging and found this post ( below ) which has fixed my problem. https://community.cloudera.com/t5/Support-Questions/No-show-Users-and-Policies-in-Global-Menu/td-p/339127 This prompted some questions on nifi security and how its supposed to function. My setup is as a stand alone nifi instance on windows 10. The main reason I wanted admin privs for my generated user was to be able to create new processors and links etc. So I have a couple of questions please: Once I have created processors etc, should I drop the user back to being a normal user instead of being an admin? Is Nifi normally designed to be (say) LDAP connected ? If I wanted to have multiple users as admins, what is the best way to do this on a stand alone instance, or is this something that perhaps should be via LDAP created groups? I think I may not fully understand the concept behind nifi security and how its supposed to work? I'm new-ish to nifi and am used to windows and its security. Thanks in advance.
... View more
12-11-2025
03:16 PM
Hi, So I have a standard stand alone install of nifi 2.5.0 on windows 10 ( no LDAP etc ) which created a generated username and password, and which allows me to log in fine. What I want to do is change this existing nifi generated user from being a normal user to an administrator. Its so i can create new processors via the api. Apparently on the very first login to a new nifi instance it logs you in as admin, but I tried this and couldnt see any options on the menu etc that looked like I had admin privs. But when it starts and runs, it uses port 8400, so its possible someone in the company may have modified the set up slightly? When it starts for the first time it goes through the expansion process. Can someone please suggest the steps to change the generated normal user to admin level please, or if maybe I'm going about this the wrong way? I have consulted the following doco but no joy. https://nifi.apache.org/nifi-docs/administration-guide.html#creating-users-groups I have spent 4 hours today trying to work it out and every time i try making changes to the authorizers.xml and login-authentical-provider.xml and just refuses to start due to java errors. Any help appreciated Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
12-11-2025
03:06 PM
Hi Matt, What I discovered is that when running nifi on windows 10 ( and possibly linux as well?), any passwords that have special characters like + or \ etc need to be converted to ascii I think it is. so : / becomes %2F + becomes %2B Once i did this, it worked OK.
... View more
11-26-2025
08:19 PM
Hi, Thanks, will do. I did try reading Provenance data via the nifi 2.5.0 API and it seems to fail I installed NIfi on my PC with no modifications and all default settings, and just use the default user and password that's created when you install it. I run this in my local windows CMD window and get "unknown error" : curl --insecure -X POST -H "Authorization: Bearer <my token> " https://localhost:8443/nifi-api/provenance Then I run this and get : curl --insecure -X POST -H "Authorization: Bearer <token>" https://localhost:8443/nifi-api/provenance > test.txt In text.txt file I get : "An unexpected error has occurred. Please check the logs for additional details." I just wanted to check I have the syntax right. I ran other API queries for flows etc and they work fine. What I'm trying to do is get a list of all files that have passed through my small local nifi installation so we can track what files have moved through nifi and when, as a form of monitoring.
... View more