Our application is currently running load through a split content, doing some enrichment on the 2 split pieces in parallel, then merging the results back together again. At a glance there were no problem with the merge content processor, but when we actually started measuring the time it was taking to to merge flowfiles we could see some results under 100ms, but also a good chunk that spread thinly from 100 to 2000ms. A little research led to the reminder that flowfile content is kept on disk until its merged, which means that we have to deal with a large number of random disk reads, and that seems to explain the slow down. Is there something, anything we can do to keep the reads towards the lower end? How about a way to pre-fetch or cache the flowfile content? An alternative solution I was knocking around in my head, is there a way to merge just attributes for two flowfiles to skip the diskread? The payloads are small enough to fit as an attribute. ... View more

We’ve experienced an error in Nifi that I can only think is a bug. While we were running a Hive query in a processor group that is read/write only for Group A, using a Hive controller service configured with Group A’s specific hive user id, the processor failed due to insufficient permissions to access the content for the query on HDFS with user=Group B’s hive controller service id, which is configured in Group B’s processor group, which is only accessible by Group B. So we are seeing a conflict of configurations between two completely separate processor groups. Here is a snippet from the error in the logs org.apache.hadoop.security.AccessControlException: Permission denied: user=group_b_id, access=EXECUTE, inode="/user/ group_a_id /tech/poc /out_table": group_a_id:group_a_group:drwxr-x--- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) The point of contention is user=group_b_id. Nowhere in our processor group do we reference that user id. It would only be used over in a completely different processor group. The act of disabling and then enabling the controller service temporarily fixed the issue and we cannot recreate it. Have there been other documented cases of controller service configurations for similar components interfering with one another? ... View more

In NiFi I am attempting to pass custom properties to a HIVE controller service. If I create a custom hive-site.xml file and add the properties I am looking to use to that file, during run time these properties are not being used. For example, I added in the execution engine parameter as below... <property> <name>hive.execution.engine</name> <value>tez</value> </property> However, on starting a processor that executes a count on a table that would result in a map reduce job being executed, the job was started as MR instead of TEZ as listed through the Job server, indicating that the property was not used. We know that the file was picked up by virtue of the controller service working at all. So does there exist a means to specify additional properties in a file to Hive for execution of a query through a nifi Hive controller service? If so, how do I achieve this? ... View more