Bryan's InferAvroSchema answer should work well in this case, but as an alternative, you might consider "normalizing" your schema by using JoltTransformJSON to change each flow file into the same schema. For example, using the following Chain spec: [
{
"operation": "shift",
"spec": {
"id_*": {
"@": "entry.[#2].value",
"$(0,1)": "entry.[#2].id"
}
}
}
] And the following input: { "id_4344" : 1532102971, "id_4544" : 1532102972 } You get the following output: {
"entry" : [ {
"value" : 1532102971,
"id" : "4344"
}, {
"value" : 1532102972,
"id" : "4544"
} ]
} This allows you to predefine the schema, removing the need for the schema and readers to be dynamic. If you don't want the (possibly unnecessary) "entry" array inside the single JSON object, you can produce a top-level array with the following spec: [
{
"operation": "shift",
"spec": {
"id_*": {
"@": "[#2].value",
"$(0,1)": "[#2].id"
}
}
}
] Which gives you the following output: [ {
"value" : 1532102971,
"id" : "4344"
}, {
"value" : 1532102972,
"id" : "4544"
} ]
... View more
