How to merge many json files together using one common field?
Hello everyone!
I have many json files like this:
{
"table_name" : "train_vd",
"data" : [ {
"battery_power" : 1954,
"clock_speed" : 0.5
} ]
}
{
"table_name" : "train_vd",
"data" : [ {
"battery_power" : 842,
"clock_speed" : 2.2
} ]
}
...
I used the MergeContent and MergeRecord processors and used the table_name field as the Correlation Attribute Name (i have ${table_name} attribute). However, this does not work and the result is as follows:
[{
"table_name" : "train_vd",
"data" : [ {
"battery_power" : 509,
"clock_speed" : 0.6
} ]
}{
"table_name" : "train_vd",
"data" : [ {
"battery_power" : 842,
"clock_speed" : 2.2
} ]
}]
...
However, I want to get the following result:
[{
"table_name" : "train_vd",
"data" : [ {
"battery_power" : 509,
"clock_speed" : 0.6
},
{
"battery_power" : 842,
"clock_speed" : 2.2
}]
}]
May you tell me how to solve this problem? Need i use a complex Jolt transformation or to configure the incoming Avro schema in the MergeRecord processor, so that then everything is combined using a single field?
