You could also use MergeContent before PutHDFS so that you can merge together all of the small files based on size or time before writing to HDFS. ... View more

It looks like an unexpected exception is happening and since the processor isn't catching it, the framework rolls back the session which puts the flow file being processed back into the original queue that it was taken from. Obviously this isn't ideal for this case and it would be preferable to improve the processor to handle this and route to failure. In the mean time, you can stop the convert processor and right click on the queue and perform a listing and from the listing you should be able to download the contents of the flow files if you want to inspect them or perform analysis. You can also right-click and clear the queue to get the problem flow files out of there. If you are able to upgrade to NiFi 1.3.0 then you may have better results with the ConvertRecord processor which is not based on the Kite library. ... View more

Is there an error that is shown on the Remote Process Group? If so can you provide that error and possibly the stacktrace that goes with it from nifi-app.log? ... View more

That looks close... where you have "name" : "children" it seems like that should be "name" : "fields" since "fields" is the array element in your JSON. Your example JSON also has top-level fields for "type" and "name" so I would think you want to declare those in the schema, unless you just want to skip over them. ... View more

The incoming JSON you showed does not match your schema... Your schema above shows that there is a single record with the fields "businessentityid", "accountnumber", "name", "creditrating". Your incoming JSON shows a field named "fields" with an array of JSON documents where each JSON document then has the four fields. If you want to make a schema that matches that JSON, your schema needs to declare a field called "fields" of type array, where each entry of the array is a type record with the four fields. ... View more

In SimpleKeyValueLookupService you can add as many user-defined properties as you want, lets say for example you added: field1 = value1 field2 = value2 field3 = value3 Then you would use LookupRecord processor and add a user-defined property like: key = field1 The LookupRecord processor is going to pass a Map with "key" -> "field1" to the SimpleKeyValueLookupService, the service is going to take the value of "key" which is "field1" and then lookup the value of that which is "value1". When "value1" is returned to the processor it will be placed in each record in the location specified by "Result Record Path". So if your schema had a field called "lookupValue" and you set "Result Record Path" to "/lookupValue" then after passing through this processor, every record would have the field "lookupValue" set to "value1". ... View more