question Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes) in Support Questions

NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Sat, 08 Apr 2017 17:55:23 GMT

Is there a way to convert everything in the json message to FlowFile attributes with corresponding values?

Example:

{"ts":1491598800154,"id.orig_h":"172.17.25.52","id.orig_p":59648,"id.resp_h":"82.148.98.187","id.resp_p":80} will automatically create the "ts,id.orig_h ... etc." attributes.

I know how to 'manually' do it one-by-one using EvaluateJsonPath.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

Wynner — Sat, 08 Apr 2017 23:39:23 GMT

@Sherif Eldeeb

I am not aware of any automatic way for NiFi to convert all of the Json content into one for one attributes.

I will mention you should be cautious when doing this depending on the the size of your Json files and the volume of data in your flow. Having a large number of flow files with very large attributes will require a large size JVM for NiFi to run smoothly. NiFi keeps all of the attributes of the flow files in the graph in the JVM memory configured in the bootstrap.conf file.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Sun, 09 Apr 2017 01:05:41 GMT

Thanks @Wynner for your reply, and for the valuable tips.

It's for log files (BroIDS, to be specific https://www.bro.org/).

Each log entry is one flow file, and the the whole content is around 500 bytes ... so size per flow file shouldn't be an issue (the rate, however, might be because it hits around 10,000 per second).

... I'll keep looking for a solution 🙂

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

mburgess — Sun, 09 Apr 2017 01:43:45 GMT

I second @Wynner 's comment about being cautious. If you determine that you still want all the JSON fields as attributes, you can do it all at once with ExecuteScript. Here is a Groovy script that expects a "flat" JSON file and turns all the fields in into attributes:

import org.apache.commons.io.IOUtils
import java.nio.charset.*
def flowFile = session.get();
if (flowFile == null) {
    return;
}
def slurper = new groovy.json.JsonSlurper()
def attrs = [:] as Map<String,String>
session.read(flowFile,
    { inputStream ->
        def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
        def obj = slurper.parseText(text)
        obj.each {k,v ->
           attrs[k] = v.toString()
        }
    } as InputStreamCallback)
flowFile = session.putAllAttributes(flowFile, attrs)
session.transfer(flowFile, REL_SUCCESS)

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Sun, 09 Apr 2017 01:48:44 GMT

Whoa! ... thanks, that's a lot to go through (I come from the elastic/logstash universe), I just installed NiFi.

I'll try and feedback (especially on performance).

Thanks a lot.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

Wynner — Sun, 09 Apr 2017 05:30:21 GMT

@Sherif Eldeeb

What are you trying to do with your flow?

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Mon, 10 Apr 2017 10:31:58 GMT

@Wynner

Read, Enrich and Ship logs from remote sensors.

We have lots of logs in JSON format (http, dns, conn ... etc.), so, I want to add to the json document few fields (e.g. sensor location), enrich (e.g. geoip) then send them over to elasticsearch.

I couldn't find a straight forward way to deal with the log entry content (like removing/adding fields) without either: a) manipulate the json log entry as 'text', which is far from being optimal ... or b) convert json content as attributes, benefit from attributes flexibility, then back to json again (which has it's own issues still: like having everything converted to strings :/).

Your follow up is appreciated 🙂

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Mon, 10 Apr 2017 10:34:26 GMT

It works 🙂

thanks a lot.

(however, I'll still have to figure out a way to retain original data types when doing an AttributesToJson, instead of having everything as a string)

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

Wynner — Mon, 10 Apr 2017 23:20:53 GMT

@Sherif Eldeeb

Have you looked at using the JoltTransformJson processor? You could use it with a Jolt specification and add the additional properties at the same time. Also, the Jolt specification can keep the format of the variables.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

mburgess — Tue, 11 Apr 2017 01:59:23 GMT

A bit of a hack is to use ConvertJSONToSQL, this will put attributes such as sql.args.N.value (the value of your field) and sql.args.N.type (the JDBC SQL type of the value). Alternatively in the Groovy script you can check the type of the variable v and set another attribute corresponding to the key k such as k.type that contains a data type identifier.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

mburgess — Tue, 11 Apr 2017 02:01:08 GMT

I should've asked before, what is your use case where you'd need all the JSON fields as attributes then convert back to JSON at the end using AttributesToJSON? If you have a JSON transformation to perform, please consider JoltTransformJSON, it is very powerful and can do the transformation(s) inline rather than moving the JSON fields to attributes and back.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Tue, 11 Apr 2017 09:57:51 GMT

@Wynner

JoltTransformJson: I couldn't find a way to include flowfile attributes in the jolt spec (so it may be 'default' included in the output JSON).

If I'm missing something, pointing me to the right direction would be highly appreciated.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

deeb — Tue, 11 Apr 2017 13:13:02 GMT

@Mutt Burgess

E.g: I want to add the geoip data (from ip field) to the json, before shipping to elasticsearch.

I don't know how to use jolt to include flowfile attribute data, yet (if possible)

P.S: thanks for helping.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

jomach — Fri, 02 Mar 2018 15:53:34 GMT

This Script is gold !!! Thank you !!!

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

mburgess — Sat, 03 Mar 2018 05:44:05 GMT

The JoltTransformJSON processor accepts NiFi Expression Language in the spec, so you can do something like:

[
  {
    "operation": "default",
    "spec": {
      "newfield": "${my.attr}"
    }
  }
]

And it will add "newfield" to the top level object, with a string value of whatever is in my.attr. Note that it (currently) has to be enclosed in quotes (and thus must be a string field), otherwise the spec validator will try to parse the EL itself and the processor will be marked invalid. This is a bug that will hopefully be fixed in an upcoming release.

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

pongthep — Mon, 28 May 2018 15:28:42 GMT

Hi @Sherif Eldeeb

You mention that You know how to convert Json to attribute one-by-one using EvaluateJsonPath.

Please advice me How to do that?
I need to convert 1 Json value to be attribute for process RouteOnAttribute

Thank You

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

pongthep — Wed, 30 May 2018 10:13:10 GMT

Hi @Sherif Eldeeb

You mention that You know how to convert Json to attribute one-by-one using EvaluateJsonPath.

Please advice me How to do that?
I need to convert 1 Json value to be attribute for process RouteOnAttribute

Thank You

Re: NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

pongthep — Wed, 30 May 2018 15:06:07 GMT

Hi @Sherif Eldeeb

I got it. By example
https://community.hortonworks.com/questions/118237/how-to-use-apache-nifi-evaluatejsonpath-for-json-t.html

Thank @Matt Burgess