@stevenmatison    For the sake of completeness and to confirm your suspicions: With NiFi 1.11.1 parameters also work in the PutEmail-Processor. ... View more

Well, I'm trying to put together a fairly generic service to ingest files.  One of the reqs is to be able to strip off any header or trailer lines.  Files can come from any number of sources, so I don't think there will be any kind of pattern for a regex.   ... View more

Thanks  @stevenmatison   I am using Parquet format, I tried with ORC not a significant difference, then I changed following setting as follows: Not knowing a lot on the following settings but based on my research. I am not using partitions yet.  set hive.cbo.enable=true; set hive.compute.query.using.stats=true; set hive.stats.fetch.column.stats=true; set hive.stats.fetch.partition.stats=true; set hive.vectorized.execution = ture set hive.vectorized.execution.enabled = true also I changed following execution engine set hive.execution.engine = spark I think changing engine to spark made a lot of difference.... Now query is running from 2.48 min to 15 sec I am quite satisfied with current performance but I would sure appreciate other advise for me and for the community.   Thanks and appreciate you response. Andy      ... View more

@wengelbrecht thank you that is exactly what i needed to see.  I am having an issue with the parquet-hadoop-1.10 and need to get a 1.12 version working in NiFi and Hive....      ... View more

Hi all,   Above solution is failing at one scenario, Scenario: if multiple flow files processed at a time and  landed in the nifi queue which is used after update query ( i.e. puthiveql which increment processed_file_cnt by one for every flow file ) processor ,then there might be chances of triggering the next flow multiple times and that is wrong.   Because we do select processed_file_cnt first and then doing the comparison for processed_file_cnt with input_file_cnt.  ... View more