What version of NiFi/HDF are you using? As of NiFi 1.2.0 / HDF 3.0.0, PutHiveQL can accept multiple statements in one flow file, so if you are currently dealing with one INSERT statement per flow file, try MergeContent to batch them up into a single flow file. This should increase performance, but since Hive is an auto-commit database, PutHiveQL is probably not the best choice for large/fast ingest needs. You may be better off putting the data into HDFS and creating/loading a table from it. For PutHiveStreaming, there is a known issue that can reduce performance, it also was fixed in NiFi 1.2.0 / HDF 3.0.0. ... View more

Usually (as is the case for the nifi-hadoop-bundle), the NAR depends on the nifi-hadoop-libraries-nar, as it provides the Hadoop libraries (such as the provided dependencies you have in your processor POM like hadoop-common), and its parent NAR is nifi-standard-services-api-nar (which you have in your NAR POM). Currently, NARs can only have one parent, so you wouldn't be able to depend on both the hadoop-libraries and standard-services-api NARs at the same time. Since the former depends on the latter, this works for your processor. Try replacing the NAR POM dependency on nifi-standard-services-api-nar to nifi-hadoop-libraries-nar, this should provide all the classes/JARs/dependencies you need. ... View more

The following configuration of ReplaceText should work for you: where the Search Value is the following: [\[\]](\{|\}) This matches [{ or ]} (and also [} and ]{ which shouldn't show up if your input is valid JSON) and replaces it with whichever curly brace it found. Note that this is a fairly specific solution, where the array is the last element of an object (i.e. the end pattern is not the end-of-object followed by end-of-array, rather the reverse). A more forgiving solution (for your input JSON) might be to use the following Chain spec in a JoltTransformJSON processor: [ { "operation": "shift", "spec": { "result": { "curves": { "*": { "@": "result.curves" } }, "*": "result.&" }, "*": "&" } } ] This "hoists" the object in the 1-element array up one level, achieving the same result as the ReplaceText pattern above. ... View more

Is SMPPServerSimulator.run() an asynchronous method, meaning it returns immediately? If so, then the framework may have already committed your session (after returning from the onTrigger() call in AbstractProcessor), in which case the transfer() either didn't work (but it was too late to throw a runtime exception up through to the framework) or it did work but the session now needs another call to commit(). These types of processors that use a separate thread/lifecycle to manage I/O are tricky to integrate into the NiFi architecture, it takes great care to make sure operations are performed in an order that is consistent with the behavior of both the separate entity and the NiFi framework. If the run() method is synchronous, then there is something else going on, but it still seems related to the session not being committed. Can you attach a debugger and see if sesson.commit() gets called after your session.transfer()? ... View more

What version of NiFi/HDF are you using? As of NiFi 1.2.0 or HDF 3.0.0, PutHiveQL supports multiple statements (via NIFI-3031) and there is also an EnforceOrder processor (via NIFI-3414), the latter of which could be configured to use the fragment.index attribute for the Order Attribute property and ${fragment.identifier} for the Group Identifier property. Prior to NiFi 1.2.0, you can try adding an UpdateAttribute processor between SplitContent and PutHiveQL, setting the "priority" attribute to ${fragment.index}. Then use a PriorityAttributePrioritizer on the connections between the SplitContent -> UpdateAttribute -> PutHiveQL. I'm not sure if this works as-is because the documentation suggests the priority comparator is lexicographical and not numeric. If that's the case, you'd need some Expression Language functions or an ExecuteScript processor to left-pad the fragment.index values with zeros to make them all the same length. ... View more