Hi @Michael Strasser, I am not sure the way you are proposing is the best approach : if you have consecutive flow files entering your process group and if the file is split in individual rows, you may have difficulties to keep things clear since you will have flow files representing rows from different files. It is certainly doable though. However, what I would recommend, at first glance, is to use the ExecuteScript processor and code something in groovy (for example). This way you don't need to split the file, you keep the entire file and you are easily able to reject the whole file if the value is not equal to the row count. You will find a useful post regarding how to use this processor here : http://funnifi.blogspot.fr/2016/02/executescript-processor-hello-world.html Let me know if you need additional details. ... View more

@Muhammed Yetginbal Considering your last comment and information provided, I had a look at: https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore#HCatalogLoadStore-HCatStorer Can you confirm that the pre-requisites are met in your case? The table 'riskFactor' exists (with the correct schema)? Besides is Hive up and running? Are you running your script where Hive is installed? are you in a clustered env? ... View more

Hi Andrew, The recommendation is to start with an unsecure cluster and to add levels of protections one by one to allow benchmarking. Overhead will depend of the use of the cluster. The numbers I have are the following: - wire encryption inside the cluster: 2x overhead - data encryption (Ranger KMS): 15%-20% overhead (but I guess it highly depends of what you are encrypting, not sure every single file must be encrypted). - for Kerberos, Knox and Ranger: this is not significant and it depends of the installation and the use (network performance to KDC, number of Knox gateways, etc). regarding Ranger, since rules are "copied" locally for each service this not significant. Hope that helps. ... View more