Hi @ushasri, There are plenty of ways to perform such a thing, but here are three ways I have already implemented the alert in my flows so far: You can have an PutEmail at the end of each failure Queue and in that why, when a message gets rejected you will get notified by email. This can be done strictly in NiFi. You can create a monitoring system which will interrogate your Bulletin Board using NiFi's REST API and send you whatever alerts you want. This can be done either directly inside NiFi or with a shell/python script on your server. You tail the nifi-app.logs on each server and extract only the errors you are interested in. This solution can be implemented both in NiFi and on your local server. ... View more

hi @ushasri, What processor are you using to save the data within your DB and how do you build up the query? If you want to add the filename to the record which gets inserted within your table, you should use an UpdateRecord Processor to generate a new column and populate it using the ${filename}. Next, the generated record should go further down your line up until it gets inserted in your database. ... View more

hi @ushasri, Can you provide some more details about your flow? Without knowing what you are doing in your flow, I can only tell you that you can use the Expression Language from NiFi and extract the filename of the flowfile and send it into your stream. The filename can be found as an attribute in each flowfile and can be called as ${filename}. More about NiFi's Expression Language: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html ... View more

hi @databoi, It would help if you could also provide your NiFi Version, as each version has it s own perks and twitches. What you have experienced so far can have plenty of root causes and it is not quite easy to debug 😞 I assume that this happens only a single node, all the time, right? Something similar happened to me as well and it was not easy to fix ... or at least it was not for me.   My problem was mostly related to how I configured the NiFi Cluster. I have been told that there are some best practices when it comes to configuring NiFi, especially on a bare metal machine: the repositories (content, flowfile and provenence) should be stored on separate drives, with a high I/O as these repositories are mostly used by NiFi to persist data. (nifi.properties) assign no more than 40% of your node's RAM memory to your heap configs. (bootstrap.conf) make sure that your open files and max user processes are set to a higher value than normal. setting the correct number of threads. (2 - 4 times the number of cores of your server)   There were three problems on my side and the solution was as follows: - I moved the repositories on a different drive (a SSD) with a high I/O, so it could read and write the content faster. - I increased the open files and the max user processes to 50000 and 10000. And I will increase it again in a couple of days. - And my third problem was related to the disk hardware, as it was dying, it started to malfunction causing this stop-the-world delays. I replaced it and everything went back to normal.   You should also pay attention to the JVM memory of that particular node. In addition, you could activate the debug mode and even generate some dumps to further analyze (./nifi.sh dump > <name of your dump file>). Another point you could check are the processes on your affected node. Maybe something is causing NiFi to become an zombie process (or your have some zombie processes) which are affecting your overall performance. I do hope that something from this message might lead you to your root cause. In any case, I strongly recommend you to take into consideration other opinions as well, from other community members, with far more experience than myself. ... View more

hi @MKothari, Are you certain that your messages get transferred successfully? Based on your screenshot, I can see that no FlowFile got IN in the past 5 mins, meaning that your flow is not running, even though everything is green and you have no errors in your GUI. I assume that the flowfile format matches whatever you have configured in your kafka. Have a look within PublishKafka and see if you do not have a strange scheduling set, causing you to see the number increasing instead of decreasing. Another thing you might try is stop both EvaluateJsonPath and PublishKafka and right click on your matched queue and open List Queue and see if there are any flowfile there and if so, on which node. Next, you can go on that node and further check the log file for possible error messages. ... View more

hi @ushasri, What do you mean when you say that you want to extract one of the columns? You want to extract it as an Attribute in your FlowFile or? If yes, you can use an ExtractText processor for example, where you add a new property (using the + sign ) and you define your extraction rule, like a Regex. As you are using a csv file, I think the following example suits your use case perfectly: https://community.cloudera.com/t5/Support-Questions/How-to-ExtractText-from-flow-file-using-Nifi-Processor/m-p/190826 ... View more