You're probably better served with MergeRecord, that will output the merged JSON records as an array ... View more

MonitorActivity is kind of an "inverse" processor, it does work when nothing's happened. So in your case, MonitorActivity downstream from QueryES would actually generate a flow file when none has been generated from QueryES. This in a sense emulates the behavior of NIFI-3576, by emitting a flow file where there are no query results (after X time has passed, not after the query is complete). Your original question was about empty results, I don't think this would apply when you get Y results but expect Z. ... View more

ExecuteScript makes two relationships available, REL_SUCCESS and REL_FAILURE, you can use session.transfer(flowFile, relationship) to do the routing. Check my ExecuteScript Cookbook series (part 1) for examples on how to transfer a flow file. Once you have a boolean or other determining factor on where to route it, you can use the example to do the transfer. If you have multiple relationships, and you can't use RouteOnAttribute due to the conditions being in the content, consider QueryRecord as you can use a SQL statement to do routing on the content. If that doesn't satisfy your use case, you can use InvokeScriptedProcessor to define relationships on the fly, based on some condition (perhaps user-defined properties like RouteOnAttribute and QueryRecord use). Consult the code for these processors for examples on how to implement this with InvokeScriptedProcessor. ... View more

Yeah that's true, I misread the first sentence of your question and was thinking conversion to JSON only, my bad ... View more

Are your files still in the source directory? GetFile defaults to removing the files you specify, so if something went wrong you might not have those files anymore, you'd need to set Keep Source File to true in that case (as Matt Clarke recommended). That's why the recommended option is ListFile->FetchFile, as it will keep track of the files it's seen and not fetch those again. ... View more

Adding to Bryan's answer, if you have the schema available to put in the registry, you can set it to Validate Field Names to false, meaning you could have field names defined in the Avro schema that do not conform to the stricter Avro rules. We should consider adding this property to readers that generate their own schema, such as CSVReader... ... View more

The HiveConnectionPool is a special type of DBCPConnectionPool, so instances of it get listed with all the others, as it is not the connection pool that doesn't support the operations, but the driver itself. What format is the input data in? You should be able to use ConvertRecord with a FreeFormTextWriter to generate SQL from your input data (don't forget the semicolon at the end of the line), then you can send that to PutHiveQL. ... View more