Ok, how about ExecuteStreamCommand which accepts incoming flow files? ... View more

FetchFile should be used with ListFile to handle this. ... View more

All - just an update. I was able to get help resolving this on StackOverflow. See the post here: https://stackoverflow.com/questions/47399391/using-nifi-to-pull-elasticsearch-indexes?noredirect=1#comment82139433_47399391 ... View more

You could probably implement a custom processor like "JsonToAttributes" very easily, iterate through a JSON document and add each field/value as an attribute to the flow file, or do it with a Groovy script in ExecuteScript. The reason it doesn't exist is that in the general case you should be careful about adding a significant amount of attributes because those are held in memory, so if we provided this processor people would start using it to adding 100s of attributes to their flow files which could lead to poor performance and memory issues. ... View more

Here is the description of the Kafka properties from their source code... max.request.size The maximum size of a request. This is also effectively a cap on the maximum record size. Note that the server has its own cap on record size which may be different from this. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. buffer.memory The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will either block or throw an exception based on the preference specified by <code>block.on.buffer.full</code>. This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests. For your case I don't think you really need to change either of these values from the defaults since you are sending 4Kb messages. Usually you would increase max.request.size if you have a single message that is larger than 1MB. ... View more