@Raymond Honderdors - The reason SAM and Schema Registry were removed when installing a HDP + HDF cluster has to do with dependency issues. It comes down to issues in Ambari 2.6.x version that come in to play when both HDP and HDF mpack are both installed that affects those services. - Those issues have been addressed in Ambari 2.7.0. Since both HDP 3.0 and HDF 3.2 (not yet released) support Ambari 2.7.0, full support of SAM and Schema Registry will return with those versions loaded together. - HDF 3.2 is baselined off Apache NIFi 1.7 (so it will include everything in apache Nifi 1.7 plus some additional HDF specific changes and some additional fixes/enhancements) - Thank you, Matt - If you found this Answer addressed your original question, please take a moment to login and click "Accept" below the answer. ... View more

@Harish Vaibhav Kali - With the release of Apache NiFi 1.0 there were major design changes to NiFi at its core. Most obvious major changes include: 1. Gone is the NiFi Cluster Manger in favor of a new zero master clustering for example. 2. Additionally NiFi's authorization model moved to a very granular authorization setup and also added multi-tenancy capability (Multiple users can be logged in at the same time making changes concurrently to different components). - With this major re-work to user authentication/authorization the ability to request access was left out. Because of the new very granular authorization capability the notion of user requesting access became very convoluted. Suppose a user does have access to the /flow NiFi resource. This does not mean they have the ability to actually build, view, or modify any of the actual components on the canvas. And there is not capability for that user (who ha authorized access to the UI) to request additional accesses. So it felt just like an incomplete solution just to give only user who could not access UI ability to request access. - Not to say that such a capability may not come back in the future, i just believe it is a low priority type item. - As of Apache NiFi 1.5, Nifi provides the ability within the authorizers.xml to sync NiFi's list of users/groups with LDAP. This will auto populate users in the NiFi users UI to which an admin user can then apply various authorizations. - Thank you, Matt - When an "Answer" addresses/solves your question, please select "Accept" beneath that answer. This encourages user participation in this forum. ... View more

@rajat puchnanda - NiFi is generally designed for independent FlowFile operation. A NiFi processor component is designed to execute against a FlowFile with no regard to other FlowFiles in the flow or coming in to the flow. - NiFi did however introduce some processors that can help achieve the logic you are looking for with some limitations. Those processors in NiFi that will allow such logic in a dataflow design would be the Wait and Notify processors. - The wait processors is designed to allow FlowFiles to pass only when a release signal is matched. The Notify processor is responsible for setting that release signal. - Question 1: You stated that you have multiple CSV files and each of those CSV files contains "date_column". Is that a single line per each incoming CSV file? or is each file a multi-line CSV file where there is a date that needs to validated in the date_column of each line? - Question 2: Are you try to fail the just a single file if any one line in it fails to validate against the date_column or are you trying to fail every file if the date column fails to validate in any one of the files? - Question 3: Do you know how many files are going to be processed? Even with Wait and Notify processors, you need to be able to tell the NiFi dataflow in this case how many files are expected; otherwise, the flow would have no way of knowing if it is complete and can release the files downstream. - The solution here may be multipart, but it all starts with knowing exactly how many files you are dealing with. Depending on the answers to above questions, you may be able to accomplish this using the below link as a reference only: https://gist.github.com/ijokarumawak/375915c45071c7cbfddd34d5032c8e90 *** This covers incoming CSV where each CSV contains only a single date_column field that needs to be validated. You must know number of incoming FlowFiles in your batch. - or you mat find yourself needing to employ the concepts covered in below link in conjunction with above link: http://ijokarumawak.github.io/nifi/2017/02/02/nifi-notify-batch/ *** This scenario covers a two phase wait and notify to handle a scenario where each CSV has multiple lines each with a dat_column to be validated. While it talks about two phase splitting, instead phase one would be based on first link above and second wait/notify loop would be based on splitting each CSV into individual FlowFiles per line to be evaluated. - Thank you, Matt ... View more