I believe a ReplaceText where you just match the entire encoded content and then inject it into an xml already written as the replacement value would be the ideal way to do this.  ... View more

I've run into similar issues and haven't reached a clear conclusion either. It seems your have very high heap usage which might potentially be relevant.  ... View more

@yashratan  Is it possible your nifi is configured to run the embedded zookeeper despite your trying to connect to your own zookeepers? Check if the nifi.state.management.embedded.zookeeper.start property in your nifi.properties file is set to true.   Check if you are able to communicate with all your zookeepers from each of your nodes. This definitely seems like an issue communicating with your ZKs.  ... View more

EDIT: @MattWho 's answer made it clear to me I slightly misunderstood the question. His suggestion of managing the entire process using a script is definitely the way to go and would perfectly fit your use-case of someone creating a new instance of an existing process group. If I may add, it sounds like using the nifi registry might benefit you. You could upload your base process group to the registry and use version control with it. Then, when creating new copies, you would instead 'pull' the same process group from the registry instead of creating a copy of process group that exists in your canvas and might unintentionally get changed or deleted. ----------------   I do not believe there is a built-in way to directly run an event when a process group is started/stopped, but a reasonable workaround would be to monitor nifi's app-logs and trigger your own event when you receive a log that mentions starting your PG.   If you only want to use tools offered by nifi, you could use the TailFile processor and configure it to run over your app-logs file, then use another processor (such as RouteOnContent) to match the log for starting the PG. From there you can try and do whatever it is that starts the administrative tasks (such as sending an http request with InvokeHTTP processor)   If this answer helped, please mark it as 'solved' and/or apply 'kudos' 🙂 ... View more

It's a bit hard to imagine your flow just from the description, but I think I understood it. What other questions do you have about it?   In my opinion it doesn't sound too great adding an attribute to every flowfile after it is written to the DB, only to then write it to a cache which control m will query (if I understood correctly).   If your only requirement is to know whether all the files were successfully written to your DB, you should simply ignore files which were successfully inserted and only apply some logic when an insert fails. Perhaps if a file fails you can write it to someplace else so you will be able to investigate why it failed (some place more persistent than a cache). If you just want to be alerted when an insert fails / want to return a response to control m, just add an invokehttp processor after the failure relationship from your putDB processor (if I correctly understood that control m expects http calls).   Because nifi is stream oriented, it's hard to tell  exactly when a batch of files has finished writing to your DB unless you know exactly how many records should be written (and then counting the flowfiles passed to success is actually reasonable). ... View more

In general, Nifi is not very well suited for event-based processing (E.G. an external scheduling tool pinging nifi to start a process group run). I do not know how Control M works, but what you're describing sounds like it could be achieved with Nifi's REST API (you can directly start/stop a specific process group by its ID).   The requirement for checking if everything got inserted to your database is also quite hard to accomplish accurately. You could use the REST API once more to check your process group has no queued files (which would mean all your flowfiles successfully passed through the flow), though you'll also have to think about what should happen if writing to the DB fails. I don't believe there is any great way to check if your scheduled run 'completed', but you could definitely use some other processor to 'notify' yourself if something failed.   If this answer helped, please mark it as 'solved' and/or apply 'kudos' 🙂. ... View more

A bit late to the party, but do you (or anyone else who might have encountered this problem) have any extra info to share about it? I am currently experiencing a similar issue. ... View more

@hegdemahendra  I have found this article by Pierre V. Where he goes into deeper detail about the logback.xml file. He mentions something that might be relevant to what you're looking for - the following two passages are what caught my eye:   ``` "We can also define new appenders in the log configuration file and change it according to our needs. In particular, we could be interested by the SMTP Appender that can send logs via emails based on quite a large set of conditions. Full documentation here."   "Obviously  you can also configure this configuration file so that NiFi log files integrate with your existing systems. An idea could be to configure a Syslog appender to also redirect the logs to an external system." ``` I myself have never done something like this, but it sounds like a step in the right direction for directly writing logs from nifi to mongo. ... View more