@Mutt Burgess E.g: I want to add the geoip data (from ip field) to the json, before shipping to elasticsearch. I don't know how to use jolt to include flowfile attribute data, yet (if possible) P.S: thanks for helping. ... View more

@Wynner JoltTransformJson: I couldn't find a way to include flowfile attributes in the jolt spec (so it may be 'default' included in the output JSON). If I'm missing something, pointing me to the right direction would be highly appreciated. ... View more

It works 🙂 thanks a lot. (however, I'll still have to figure out a way to retain original data types when doing an AttributesToJson, instead of having everything as a string) ... View more

@Wynner Read, Enrich and Ship logs from remote sensors. We have lots of logs in JSON format (http, dns, conn ... etc.), so, I want to add to the json document few fields (e.g. sensor location), enrich (e.g. geoip) then send them over to elasticsearch. I couldn't find a straight forward way to deal with the log entry content (like removing/adding fields) without either: a) manipulate the json log entry as 'text', which is far from being optimal ... or b) convert json content as attributes, benefit from attributes flexibility, then back to json again (which has it's own issues still: like having everything converted to strings :/). Your follow up is appreciated 🙂 ... View more

Whoa! ... thanks, that's a lot to go through (I come from the elastic/logstash universe), I just installed NiFi. I'll try and feedback (especially on performance). Thanks a lot. ... View more

Thanks @Wynner for your reply, and for the valuable tips. It's for log files (BroIDS, to be specific https://www.bro.org/). Each log entry is one flow file, and the the whole content is around 500 bytes ... so size per flow file shouldn't be an issue (the rate, however, might be because it hits around 10,000 per second). ... I'll keep looking for a solution 🙂 ... View more