Since the data coming in is JSON concatenated together with newlines the first thing you can do is use SplitText to split on newlines. Then the resulting objects are JSON documents at which point in NiFi you can use all kinds of fun processors. The suggestion above to use MergeContent *I think* would head down the opposite path from what I believe the question was but perhaps I understood it incorrectly. ... View more

You'll want to review the heap dump further but it sounds highly likely that FlowFile objects are being built up in large numbers (10s/100s of thousands+) within the ProcessSession which can quickly take up a lot of heap. The heap storage for that is not the content of the flowfiles but is the attributes and that can still add up fast. You want to ensure you're committing the session frequently to move those along and not have the session have too many flowfiles being tracked at once. Now, having said that you should also consider doing ConsumePulsarRecord instead of just ConsumePulsar, for example. Doing it via the record model will dramatically outperform doing it otherwise unless you do your own raw record framing such as newlines or something where a single flowfile represents many records at once. If it is a single record is a single flowfile then just be sure you're committing the session frequently. ... View more

Hello there Chris. While not a direct answer to your question the community made the NiFi 1.x release line available in Aug 2016. In the latest release on the 1.x line (1.5.0) the community introduced the Apache NiFi Registry. This provides a really powerful and well integrated way to have versioned flows stored in a central registry which you can use to have good SDLC behaviors from dev, to staging, to prod and which handles things like sensitive properties and processor group level variables well. It also lets you have nested versioned groups which is really useful for multi-tenant/team cases. ... View more

are any updates working or is the only one being tried? ... View more