Hi, i need to ingest only the JSON files following a valid schema. i am trying to achieve this by validate-record processor. i am supplying the same schema for both JSONTreeReader and JSONRecsetWriter. I am not using AVRO because my input contains _ in the names. (but i came up with this schema by modifying the input file without _ and using inferAvroSchema and then changed both to use _ to match the input file) my schema and files are matching but its sending it to invalid relation. Anything wrong that i am doing..?? Schema : {
"type": "record", "name": "iHist", "fields": [ {
"name": "file_name",
"type": "string"
}, {
"name": "plant",
"type": "string"
}, {
"name": "collector",
"type": "string"
}, {
"name": "name",
"type": "string"
}, {
"name": "unique_id",
"type": "string"
}, {
"name": "description",
"type": "string"
}, {
"name": "general_1",
"type": "string"
}, {
"name": "general_2",
"type": "string"
}, {
"name": "general_3",
"type": "string"
}, {
"name": "general_4",
"type": "string"
}, {
"name": "general_5",
"type": "string"
}, {
"name": "data_points",
"type": {
"type": "array", "items": {
"type": "record", "name": "data_points", "fields": [ {
"name": "timestamp",
"type": "string"
}, {
"name": "value",
"type": "string"
},
{
"name": "quality",
"type": "string"
}
]
}
}
}
]
} Data file.. {
"file-name": "tp-tcollec.tag.json", "plant": "P11A3", "collector": "test_Collector", "name": "tag_SAFETY_MARGN.F_CV", "unique-id": "1532358720761", "description": "test", "general-1": "", "general-2": "", "general-3": "", "general-4": "", "general-5": "", "datapoints": [ {
"timestamp": "2016-07-19T10:25:43.000Z",
"value": "177",
"quality": "100"
}, {
"timestamp": "2016-07-19T10:25:42.000Z",
"value": "177",
"quality": "100"
}, {
"timestamp": "2016-07-19T10:25:41.000Z",
"value": "177",
"quality": "100"
}
]
} I just need to validate if the input file is following the schema. any better ways to do this.??
... View more
