We were looking at storing the provenance *.gz ( lucene ) files so we could have a record of all activity through Nifi, as our IT security people are fussy about such things. It seems Nifi writes to multiple files at the same time, so this may not be possible, or perhaps there is a better way of doing this?  I would welcome thoughts as to how we might do this, or maybe propose a different way to achieve the same logging outcome. Thanks in advance.  ... View more

Hi all,  I recently spent a full 2 days over a rainy weekend trying to get a stand alone instance of Nifi v2.5.0 on ubuntu 22.04 trying to integrate with AD,  to use AD for Nifi authentication, but with no joy. It almost worked but couldn't get past a certain point. The problem I discovered is there doesn't seem to exist on the internet  a simple step by step process on how to set up Nifi with AD/LDAP. My set up is a linux PC called nifi1 and had an old windows 2012 R2 domain controller with LDAP running on it I decided to use. I created a domain called  testdomain.local and created a domain user called testuser in the AD User OU that I wanted to use as a nifi user. I also created a service account called svc-nifi also in the User OU that is used to log onto AD as a service. I ran a series of tests using LDAP command line queries from the linux PC whereby the service account could run LDAP queries from linux that proved it could access AD, read the AD info it needed and could connect OK to AD etc. The testuser logon also works fine in logging onto the domain on other PCs. Is there a documented basic step by step process of how to do this on a linux ( or windows ) PC please?  I just need the minimum basic working example of nifi.properties, authorizers.xml, uses.xml and authorize.xml etc files please. There are so many variables in each file that make it difficult to work through sequentially, as I dont have expert Nifi knowledge. I wondered if I had maybe skipped a step or something, like :     * Do I need to add the linux PC to AD itself first , and if so, how is this set up in the       config files?      * What do i need to add to the nifi.properties, authorizers.xml file etc to get it to work please?       * etc. Any help appreciated - this should really be simple, but its not......  I'm happy to write up a step by process and post it back here once I have it running, to help others. Thanks in advance. 🙂     ... View more

Hi I wanted to look into a need to transfer 1 TB files by using chunking in Nifi. Each file also has to have its 20 items of meta data associated with it remain intact so the metadata and the data both survive the breaking up of the file ( chunking ) into 1000 chunks and re-assembling the file at the destination ( de-chunking ). Also, is the meta data for the large file duplicated onto each of the file 1000 chunks or is it a sub-set of the meta data? Someone mentioned nifi passes the file chunks data through JVM memory on its way to the content repository. Can I confirm whether file chunks pass through JVM memory as they are written to the file/content repository for a large file ( or any file for that matter ?) - I was fairly sure they aren't, otherwise the JVM size ( limited by machine RAM ) on the machine would limit reading in of large file data, and that would limit large file transfer speed - is that correct? I'm trying to confirm my understanding of how Nifi handles these large files please. Any help appreciated.   ... View more

Hi What is the preferred method to alter a nifi processor configuration please? We want to be able to get a user to choose from one of 3 options from a drop down menu in a GUI screen. The chosen option then is passed to either a curl command to make changes to the processor via the Nifi API, or can we create a python command to then alter a nifi processor configuration possibly by curl, but maybe there is another way? Curl looks a safer option, but python might be better? Not sure....? Thanks in advance.  ... View more