NiFi: JSON Arrays

[#1]

Hi everyone, I am a newbie to NiFi. I have a requirement to create JSON which will be similar as below:

{ "errors":[ { "errorMessage":"missing field_1" }, { "errorMessage":"missing field_2" } ] }

I am using AttributesToJSON NiFi processor, but not sure how to set the arrays in attribute/flow file.

Any help would be highly appreciated.

[#2 · Accepted solution]

What are your attributes named, and what are their values? AttributesToJSON will not create an array for you, but you can use JoltTransformJSON after AttributesToJSON to create an array from the attributes. Given this example input JSON:

{
  "error_message_1": "missing_field_1",
  "error_message_2": "missing_field_2"
}

You can use the following Shift spec in the JoltTransformJSON processor:

{
  "operation": "shift",
  "spec": {
    "error_message_*": "errors[].errorMessage"
  }
}

This gives the following output (which matches your goal above):

{
  "errors" : [ {
    "errorMessage" : "missing_field_1"
  }, {
    "errorMessage" : "missing_field_2"
  } ]
}

[#3]

Thanks @Matt Burgess, that worked. Actual requirement is, I have to read some data(JSON format) from Kafka(using ConsumeKafka_ processor) and validate it and if any field missing, then I have to send(PublishKafka_) some details message about missing field.

Expected JSON should contain field_1, filed_2, field_3 and filed_4

But it has field_1 and field_3( 2 and 4 are missing).

I/P: {"field_1":"value_1","field_3":"value_3"}

O/P: { "errors":[ { "errorMessage":"missing data field_2" }, { "errorMessage":"missing the data field_4" } ] }

I am using:

1. EvaluateJsonPath (to read the json data)

2. RouteOnAttribute ( to check if empty i.e. ${FIELD_1:isEmpty()}

3. UpdateAttribute (errorMessage ${RouteOnAttribute.Route})

4. AttributeToJSON

Please let me know if you have any suggestion for my use case?

Thanks again for your help.

[#4]

In response to your comments/requirements on my other answer, you could do the following:

1) EvaluateJsonPath (to read the JSON data), let's say you get attributes field_1, field_2, field_3, field_4.

2) UpdateAttribute, to set attributes "Missing data field_1", "Missing data field_2", "Missing data field_3", "Missing data field_4" to true or false, based on your aforementioned "field_N:isEmpty()" expression

3) AttributesToJSON to put the "Missing data" attributes into the content

4) JoltTransformJSON, to find the missing fields and add an error message, using the following ShiftR spec:

{
  "operation": "shift",
  "spec": {
    "Missing data field_*": {
      "true": {
        "$1": "errors[].errorMessage"
      }
    }
  }
}

That gives the following output:

{
  "errors" : [ {
    "errorMessage" : "Missing data field_2"
  }, {
    "errorMessage" : "Missing data field_4"
  } ]
}

[#5]

@Matt Burgess first of all thanks for replying my question.

{  
   "name":"John",
   "requestId":"123OAO0435",
   "email":"john@gmail.com"
}
Getting more complex now, in the above example
1. "name" is missing(name:isEmpty())
2. "email" is missing(email:isEmpty())
It should create JSON like: 
{
"requestId": "123OAO0435",
"errors": [{
"errorMessage": "Name is missing"
}, {
"errorMessage": "Email cannot be empty"
}]
}

If all the fields have data(name, requestId and email), then it is calling some rest api(using InvokeHttp processor).
It should create JSON like:
{  
   "name":"John",
   "requestId":"123OAO0435",
   "apiResponse":{
         // response from API call
   }
}
Thats the reason I was RouteOnAttribute processor, any better way to achieve this?

[#6]

You can still use RouteOnAttribute to route the flow files with missing values somewhere separate from the files that have all the values populated. In UpdateAttribute, you can call the attribute whatever you want the message to be, and for the JOLT transform, you can have multiple entries in the spec, each matching a separate entry from the JSON.

[#7]

I have the below response from rest API call(InvokeHttpt) and I want to transform as below(wrap whole response in "responseData"), tried using JoltTransforJSON but not getting as expected. Any idea what I am missing? Any help would be appreciated.

I/P
{
	"query": {
		"email": "john@gmail.com",
		"count": 1,
		"created": "2017-05-17T15:40:21Z",
		"lang": "en-US",
		"responseCount": 1
	},
	"responseStatus": {
		"status": "success",
		"errorCode": "0",
		"description": ""
	}
}


Expected O/P:
{
	"responseData": {
		"query": {
			"email": "john@gmail.com",
			"count": 1,
			"created": "2017-05-17T15:40:21Z",
			"lang": "en-US",
			"responseCount": 1
		},
		"responseStatus": {
			"status": "success",
			"errorCode": "0",
			"description": ""
		}
	}
}




Jolt spec I used:
[
  {
    "operation": "shift",
    "spec": {
      "query": "responseData",
      "*": {
        "@": "&"
      }
    }
    }
]