question Re: JSONTreeReader behaving different between V1.21 and V1.18 in Support Questions

JSONTreeReader behaving different between V1.21 and V1.18

ollyhank — Fri, 19 May 2023 18:36:29 GMT

When parsing a json array with a json tree reader (in this case using convert record), I am presented with this error, this testing was done in Nifi Version 1.21.0. However this error never occurs in Version 1.18.0.

An example of json that fails in 1.21.0 that works in 1.18.0

[ {"a": [ {"b": [{}]}]}, {"a": [ {"b": []}]} ]

Any help would be greatly appreciated.

Re: JSONTreeReader behaving different between V1.21 and V1.18

cjervis — Fri, 19 May 2023 18:43:41 GMT

Welcome to the community @ollyhank . I'm not an expert but I did find notes that version 1.2 included:

New or Improved Processors, Controller Services, and Reporting Tasks

New Record oriented abstraction for reading/writing schema aware event streams from CSV, JSON, AVRO, Grok, and plaintext with easy extension for other formats/schemas
- QueryRecord processor to execute SQL queries over a stream of records powered by Apache Calcite
- ConvertRecord processor to efficiently transform records from a given schema and format into another schema and format
- SplitRecord processor to efficiently split huge record bundles into configurable batch sizes for divide and conquer or protect downstream systems
- Processors to efficiently stream Records into and out of Apache Kafka in a format and schema aware manner and which automatically handle achieving high throughput and full provenance
- Controller Services for plugging into and managing data schemas (Avro Schema Registry, Hortonworks Schema Registry) that integrate nicely into the record readers and writers

Not sure if it is the issue here, but is something to consider while awaiting someone with more experience to reply.

Re: JSONTreeReader behaving different between V1.21 and V1.18

ChuckE — Mon, 22 May 2023 16:38:00 GMT

There were some changes in 1.20 to address better schema inference. See the references below for release notes for 1.20 and specifically the issue you are experiencing.

[NIFI-10956] - Schema Inference returns incorrect datatype for records where some arrays are empty

https://issues.apache.org/jira/browse/NIFI-10956

Re: JSONTreeReader behaving different between V1.21 and V1.18

ollyhank — Wed, 24 May 2023 14:30:35 GMT

Hi @cjervis, thanks for offering a solution but that change was V1.2, our error only occurs in the difference between much later 1.18 -> 1.21.

Re: JSONTreeReader behaving different between V1.21 and V1.18

ollyhank — Wed, 24 May 2023 14:31:25 GMT

Hi @ChuckE, yes i had seen that note before and imagine that they fixed a similar issue but didnt catch this specific case, i am posting an issue in Jira to hopefully find a fix.