Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)

avatar
Explorer

Is there a way to convert everything in the json message to FlowFile attributes with corresponding values?

Example:

{"ts":1491598800154,"id.orig_h":"172.17.25.52","id.orig_p":59648,"id.resp_h":"82.148.98.187","id.resp_p":80} will automatically create the "ts,id.orig_h ... etc." attributes.

I know how to 'manually' do it one-by-one using EvaluateJsonPath.

1 ACCEPTED SOLUTION

avatar
Master Guru

I second @Wynner 's comment about being cautious. If you determine that you still want all the JSON fields as attributes, you can do it all at once with ExecuteScript. Here is a Groovy script that expects a "flat" JSON file and turns all the fields in into attributes:

import org.apache.commons.io.IOUtils
import java.nio.charset.*
def flowFile = session.get();
if (flowFile == null) {
    return;
}
def slurper = new groovy.json.JsonSlurper()
def attrs = [:] as Map<String,String>
session.read(flowFile,
    { inputStream ->
        def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
        def obj = slurper.parseText(text)
        obj.each {k,v ->
           attrs[k] = v.toString()
        }
    } as InputStreamCallback)
flowFile = session.putAllAttributes(flowFile, attrs)
session.transfer(flowFile, REL_SUCCESS)

View solution in original post

17 REPLIES 17

avatar
Explorer

It works 🙂

thanks a lot.

(however, I'll still have to figure out a way to retain original data types when doing an AttributesToJson, instead of having everything as a string)

avatar
Master Guru

A bit of a hack is to use ConvertJSONToSQL, this will put attributes such as sql.args.N.value (the value of your field) and sql.args.N.type (the JDBC SQL type of the value). Alternatively in the Groovy script you can check the type of the variable v and set another attribute corresponding to the key k such as k.type that contains a data type identifier.

avatar
Master Guru

I should've asked before, what is your use case where you'd need all the JSON fields as attributes then convert back to JSON at the end using AttributesToJSON? If you have a JSON transformation to perform, please consider JoltTransformJSON, it is very powerful and can do the transformation(s) inline rather than moving the JSON fields to attributes and back.

avatar
Explorer

@Mutt Burgess

E.g: I want to add the geoip data (from ip field) to the json, before shipping to elasticsearch.

I don't know how to use jolt to include flowfile attribute data, yet (if possible)

P.S: thanks for helping.

avatar
Explorer

This Script is gold !!! Thank you !!!

avatar
New Contributor

Hi @Sherif Eldeeb

You mention that You know how to convert Json to attribute one-by-one using EvaluateJsonPath.

Please advice me How to do that?
I need to convert 1 Json value to be attribute for process RouteOnAttribute

Thank You

avatar
New Contributor

Hi @Sherif Eldeeb


You mention that You know how to convert Json to attribute one-by-one using EvaluateJsonPath.

Please advice me How to do that?
I need to convert 1 Json value to be attribute for process RouteOnAttribute

Thank You

avatar
New Contributor