Support Questions

deeb · ‎04-08-2017

Is there a way to convert everything in the json message to FlowFile attributes with corresponding values?

Example:

{"ts":1491598800154,"id.orig_h":"172.17.25.52","id.orig_p":59648,"id.resp_h":"82.148.98.187","id.resp_p":80} will automatically create the "ts,id.orig_h ... etc." attributes.

I know how to 'manually' do it one-by-one using EvaluateJsonPath.

mburgess · ‎04-08-2017

I second @Wynner 's comment about being cautious. If you determine that you still want all the JSON fields as attributes, you can do it all at once with ExecuteScript. Here is a Groovy script that expects a "flat" JSON file and turns all the fields in into attributes:

import org.apache.commons.io.IOUtils
import java.nio.charset.*
def flowFile = session.get();
if (flowFile == null) {
    return;
}
def slurper = new groovy.json.JsonSlurper()
def attrs = [:] as Map<String,String>
session.read(flowFile,
    { inputStream ->
        def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
        def obj = slurper.parseText(text)
        obj.each {k,v ->
           attrs[k] = v.toString()
        }
    } as InputStreamCallback)
flowFile = session.putAllAttributes(flowFile, attrs)
session.transfer(flowFile, REL_SUCCESS)

View solution in original post

deeb · ‎04-10-2017

It works 🙂

thanks a lot.

(however, I'll still have to figure out a way to retain original data types when doing an AttributesToJson, instead of having everything as a string)

mburgess · ‎04-10-2017

A bit of a hack is to use ConvertJSONToSQL, this will put attributes such as sql.args.N.value (the value of your field) and sql.args.N.type (the JDBC SQL type of the value). Alternatively in the Groovy script you can check the type of the variable v and set another attribute corresponding to the key k such as k.type that contains a data type identifier.

mburgess · ‎04-10-2017

I should've asked before, what is your use case where you'd need all the JSON fields as attributes then convert back to JSON at the end using AttributesToJSON? If you have a JSON transformation to perform, please consider JoltTransformJSON, it is very powerful and can do the transformation(s) inline rather than moving the JSON fields to attributes and back.

deeb · ‎04-11-2017

@Mutt Burgess

E.g: I want to add the geoip data (from ip field) to the json, before shipping to elasticsearch.

I don't know how to use jolt to include flowfile attribute data, yet (if possible)

P.S: thanks for helping.

jomach · ‎03-02-2018

This Script is gold !!! Thank you !!!

pongthep · ‎05-28-2018

Hi @Sherif Eldeeb

You mention that You know how to convert Json to attribute one-by-one using EvaluateJsonPath.

Please advice me How to do that?
I need to convert 1 Json value to be attribute for process RouteOnAttribute

Thank You

pongthep · ‎05-30-2018

Hi @Sherif Eldeeb

You mention that You know how to convert Json to attribute one-by-one using EvaluateJsonPath.

Please advice me How to do that?
I need to convert 1 Json value to be attribute for process RouteOnAttribute

Thank You

pongthep · ‎05-30-2018

Hi @Sherif Eldeeb

I got it. By example
https://community.hortonworks.com/questions/118237/how-to-use-apache-nifi-evaluatejsonpath-for-json-...

Thank @Matt Burgess

Cloudera Community

Support Questions

NiFi - convert everything in json to attributes, not one by one (i.e JsonToAttributes)