About manishg

manishg · ‎09-14-2023

I have a nifi flow, working fine. It picks file from a source directory, processes it, and file is deleted from source directory once picked up. This all works fine first time. But later on, if I copy another file in source directory, that file is not picked up. I even stop all processors, clean any queue if not empty, and restart processors, but still file is not picked from input directory. What causes this behaviour? As a result, I have to stop-restart nifi, making it a bit cumbersome. PS: Based on some google suggestions, I tried to delete flowfile_repository, but it doesn't allow so with message 'Device or resource busy'.

manishg · ‎09-12-2023

So basically all nodes perform exactly entire task. There is no divide and rule by default. Flow designer has to introduce any such parallelism by herself.

manishg · ‎09-11-2023

I have some basic questions about nifi cluster. I understand that nifi makes use of multiple nodes to speed up the processing of data. But how do we ensure durability of data processed? Suppose we distribute n flowfiles in a queue to several nodes, with each node getting a subset of n files. Does nifi assign same subset to more than one nodes, so that if one of those processing node goes down, we still have others working on same subset? Second, how is result of processing done on different nodes aggregated? Is it done on primary node OR there is some other mechanism?

manishg · ‎09-10-2023

@SAMSAL I experimented same template with nifi 1.10.0, and found that FetchXMLFile has no issues with execution node as PRIMARY. It seems this new requirement mentioned by you was introduced only after 1.10.0.

manishg · ‎09-08-2023

What does this message mean on an invalid processor: I see it for processor FetchXMLFile. On scheduling tab, execution dropdown has 'Primary Node' selected. On changing it to 'All Nodes', error goes away. I am on 1.22.0(with some nars from 1.10.0 also added to lib folder).

manishg · ‎09-07-2023

So I copied only those nars which we use, and container could launch now. Though I have to remove few nars which were causing issues, like nifi-ssl-context-service-nar-1.10.0.nar. And now existing flows dont have issues with properties which are obsolete in 1.22.0 as 1.10.0 nars are used for those components. Thanks for all the inputs.

manishg · ‎09-06-2023

Ok. So I would need to copy them one by one and check if container launches. It would be lot of work I guess.

manishg · ‎09-06-2023

For my work, I had to copy nifi 1.10.0 nars into lib folder of nifi 1.22.0. I do this as part of docker image build process. This work seems to support this. First, container launch has issues with multiple jetty nars, so I dont't copy them, and jetty issue is gone. But container launch still fails with: java.lang.IllegalStateException: Multiple framework NARs discovered. Only one framework is permitted Is this due to nar versions(1.10.0 and 1.22.0) or do I need some more configuration to make it work?

manishg · ‎09-05-2023

Also, I was wondering if it is feasible to keep multiple versions of NARs in lib folder. This way existing templates may continue working without any issue. @MattWho

manishg · ‎09-05-2023

Sorry for this late response. So I have flow templates made for NiFi 1.10. Hence they have their properties and other needed stuff set accordingly. Now, I have created a new docker image from dockerfile(which is mostly just like official nifi 1.22.0 docker file with some basic customization), and in our maven build process I copy our custom NARs to /libs folder. All works fine. I test our existing templates with this new image and I see that some processors are invalid(like CSVRecrodLookUpService no longer supports 'csv-format' property OR if FetchXMLFile's Scheduling's Execution is 'Primary' then it makes processor invalid). So I wanted to know do we rectify such invalid processors manually or we have a documented way for it? Issue is that we have large number of templates and if all of them have such invalid processors due to migration to 1.22.0 then many of our templates would stop working out of the box, and hence need quite an effort.

Online	Offline
Last Visited	‎06-10-2024 06:50 AM

Member Since	‎07-27-2023 04:53 AM
Last Visited	‎06-10-2024 06:50 AM
Posts	55
Kudos received	19

Cloudera Community

Re: Nifi: file not picked from input directory aft...

Nifi: file not picked from input directory after f...

Re: nifi cluster: durability , aggregation

nifi cluster: durability , aggregation

Re: 'Execution node' is invalid because processors...

'Execution node' is invalid because processors wit...

Re: Multiple framework NARs discovered. Only one f...

Re: Multiple framework NARs discovered. Only one f...

Multiple framework NARs discovered. Only one frame...

Re: Reusing nifi 1.10 nars with 1.22

Re: Reusing nifi 1.10 nars with 1.22