How to capture both key and value of json data with dynamic key on Nifi?

[#1]

I have json data listen from JettyWebsocketServer like:

{
	"project_1": {
		"host": "V001",
		"data": [{
			"sensor_code": "temp",
			"value": {
				"max": 26.2,
				"avg": 26.1,
				"min": 26.1
			}
		},
		{
			"sensor_code": "humid",
			"value": {
				"max": 48.8,
				"avg": 48.7,
				"min": 48.6
			}
		}]
	}
}

Because my data from multiple project, so I have dynamic key as: project_1, project_2. And I have to route those data into corresponding destination.

I found EvaluateJsonPath to parse Json data but it need static key.

There is any way to capture both key and value of it to route data by key?

[#2]

@Kiem Nguyen

you can use RouteonContent processor and change

Match Requirement property to Content must contain Match

and add the below properties

1.project_1 as .*project_1.*
2.project_2 as .*project_2.*

This processor will checks the content of flowfile if it is having project_1 in it it transfers the ff to project_1 relationship, if the content having project_2 then it transfers to project_2 relationship.

Flow Screenshot:-

As shown in the above screenshot you can use the relationships project_1,project_2 and store or process those content as per your requirements.

[#3]

Thanks @Shu

Please let me clarify my problem. Because I don't know key before. So I need to create a FlowFile which can run with dynamic key without stopping Job when having a data with new key.

If we know certainly list of key, I think we can use EvaluateJsonPath to pull key into Attribute and use RouteOnAttribute as alternative way.

Again, my problem is I don't know when new key incoming to stop Job and add more key into RouteOnContent as your way.

[#4]

Hi @Kiem Nguyen

What will be the rest of the flow after routing? can you give more details?

[#5]

Hi @Abdelkrim Hadjidj

After routing, the rest of flow is json string like:

        {
		"host": "V001",
		"data": [{
			"sensor_code": "temp",
			"value": {
				"max": 26.2,
				"avg": 26.1,
				"min": 26.1
			}
		},
		{
			"sensor_code": "humid",
			"value": {
				"max": 48.8,
				"avg": 48.7,
				"min": 48.6
			}
		}]
	}<br>

It is value of key "project_1", "project_2",....

For my case, I will use key as database name on MongoDB. Example:

Value of data from project_1 will be insert into database project_1.

Value of data from project_2 will be insert into database project_2.

Because PutMongo processor of Nifi version 1.4.0 currently supports expression language.

So if I can capture key of data I can insert into corresponding database using ${key}

Please help me if you have any suggestion.

Thanks,

Kiem

[#6]

Hi @Kiem Nguyen

In this case you can use ExtractText to scan your flow content and extract the name of the project. With the right regular expression you can extract your key and store in an attribute projectname.

Once you have this, use it in the PutMongodb to update the required collection using ${projectname} for the Mongo Collection Name.

Can you try this and let me know if this works ?

Thanks

[#7]

@Kiem Nguyen

I am not a Regex expert but you can test this and work on it to have a better implementation.

ExtractText with following Regex (\w+)

You can see that there's an attribute projectname which contains project_1

[#8]

Thanks @Abdelkrim Hadjidj,

Although using regex with dynamic key seem difficult for me. I will try this.

However as you see, the data to insert into MongoDB is value of key, it is not whole incoming data. (not include "project_1"...)

So we need to capture this data too. And it will be very difficult to obtain value of key if also using regex.

I need to capture both key and value separately. 😞

[#9]

@Kiem Nguyen Since you have the key with this regex, you can use it to get value with EvaluateJsonPath.

[#10]

@Abdelkrim Hadjidj,

As your suggestion, I already got project_1, project_2... under the name: projectname. And it is an attribute of incoming flowfile for EvaluateJsonPath.

But when I try to get value of above key, I add property content with value: $.${projectname}

Output flowfile of EvaluateJsonPath has content attribute is empty string. I think EvaluateJsonPath uses JsonPath expression and It doesn't know variable "projectname" attribute of incoming flowfile.