Matt, In the evaluation I'm doing NiFi versus StreamSets as a user, I prefer NiFi due to its richest pipeline management and clustering option in opensource version, and a lot of other things. The question posted by dhieru singh make enormous sense, in a perspective that a lot of critical systems (or even not critical ones) do this. I mean, if you accidentally stop all process groups at once and then, as soon you noticed that, restart all process again (imagine a DevOps guy waking up at 3:00AM 🙂 ) you may also start process groups that you don't want. This would be the second accident. It happened with me during evaluation, where I had built several process groups and some of them were intentionally suspended (ok, good practice says if you don't use, remove it, but, sure I'm not the only one when it comes to testing). Anyway, maybe is a good practice/idea to notify the user when he/she is about to stop the whole thing or, as you mention, add the ability to "lock" the current running state of a given process group. Thank you ! Julio ... View more

@Wynner you were correct the content and flowfile were refering to root, changed it and fixed it by referring content repo to different disk and flow file to different raid/disk this link also helped me https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html ... View more

@dhieru singh FlowFiles generated by ListenUDP are placed on the outbound connection. One of the easiest ways to see the sizes of those FlowFiles is to right click on that connection (while it has queued data) and select "list queue" from the context menu that is displayed. It will open a new UI that will list all FlowFiles queued on that connection along with their details. Matt ... View more

@Wynner Appreciate your help. Yes it is indicating that I have run out of the disk space. which was proved by this screenshot, however the acutal size of disk 4.5TB. Thanks contentstorage.pngflowstorage1.png ... View more

Hi @Shu Thanks for the reply, Yes you are absouletly correct about the error message being a generic one complaining about the config issue. Here my udp port was already occupied by some other app. Chnaged the post number and worked like a charm Thanks again appreciate it Dhiren ... View more

@Shu I was just reading the docs found out ${now():format('yyyy')}. Thanks for the quick response. Much appreciated. Dheeru ... View more

@Shu Thanks a lot for quick reply. Much appreciated. This solved the issue. The problem was once I set the expression language for creating directory structure. The putFile was looking for the directory structure hence the error. So I had to delete the PutFile processor and the use your EL and it worked like a charm. Thanks again ... View more

@Matt Clarke Thanks for the quick response,I would raise the JIRA request, here is the enhancement request https://issues.apache.org/jira/browse/NIFI-4458 ... View more

I am facing the same issue right now. My current approach is to have a script to read the first line of all my csv files and then transform the first-line text into create sql statement. I wonder whether this is some built-in solution from HDP. ... View more