Can't redefine org.apache.nifi.dataType - Schema error

hjfigueira — Wed, 09 Feb 2022 19:22:37 GMT

Hi Guys,

I'm currently having an issue where a schema that I'm using is erroring out in a RecordLookup processor ( with a Rest Service ). The message I get is the one in the title of this post, "Can't redefine org.apache.nifi.dataType". Below you can find the schema that I'm using.

{ "type": "record", "name": "FullTicketData", "fields": [ { "name" : "terminal", "type": { "type": "record", "name" : "terminal_array", "fields": [ { "name": "success", "type": "string" }, { "name": "data", "type": { "type": "array", "items": { "name": "data_array", "type": "record", "fields": [ { "name": "ID", "type": "int" } ] } } }, { "name": "count", "type": "int" } ] } }, { "name" : "driver", "type": { "type": "record", "name" : "driver", "fields": [ { "name": "success", "type": "string" }, { "name": "data", "type": { "type": "array", "items": { "name": "data_record", "type": "record", "fields": [ { "name": "ID", "type": "int" } ] } } }, { "name": "count", "type": "int" } ] } }, { "name": "ticket", "type": { "name" : "ticket", "type" : "record", "fields": [ { "name": "success", "type": "string" }, { "name": "data", "type": { "name": "data", "type": "record", "fields": [ { "name": "ID", "type": "int" }, { "name": "job", "type": { "name": "job", "type": "record", "fields": [ { "name": "ID", "type": "int" } ] } } ] } } ] } }, { "name": "order", "type": { "name": "order", "type": "record", "fields": [ { "name": "externalJobNo", "type": "string" }, { "name": "jobNo", "type": "string" }, { "name": "driverName", "type": "string" }, { "name": "startTime", "type": "string" }, { "name": "endTime", "type": "string" }, { "name": "comment", "type": "string", "default": "empty" }, { "name": "readStart", "type": "string", "default": "0" }, { "name": "readEnd", "type": "string", "default": "0" }, { "name": "removed", "type": "string" }, { "name": "installed", "type": "string" }, { "name": "endTopCut", "type": "string" }, { "name": "endBottomCut", "type": "string" }, { "name": "startTemp", "type": "string" }, { "name": "endTemp", "type": "string" }, { "name": "swPer", "type": "string" }, { "name": "gravity", "type": "string" }, { "name": "volume", "type": "string" }, { "name": "grossVolume", "type": "string" }, { "name": "temp", "type": "string" }, { "name": "dist", "type": "string" }, { "name": "explanation", "type": "string" }, { "name": "status", "type": "string" }, { "name": "driverID", "type": "int" }, { "name": "billingQuestionComment", "type": "string", "default": "empty" } ] } }, { "name": "jobTimeType", "type": { "name": "jobTimeType", "type": "record", "fields": [ { "name": "startTime", "type": "string" }, { "name": "endTime", "type": "string" }, { "name": "startlatitude", "type": "string" }, { "name": "startlongitude", "type": "string" }, { "name": "endlatitude", "type": "string" }, { "name": "endlongitude", "type": "string" } ] } }, { "name": "jobTimeTypeTerminal", "type": { "name": "jobTimeTypeTerminal", "type": "record", "fields": [ { "name": "terminal", "type": "string" }, { "name": "startTime", "type": "string" }, { "name": "endTime", "type": "string" }, { "name": "startlatitude", "type": "string" }, { "name": "startlongitude", "type": "string" }, { "name": "endlatitude", "type": "string" }, { "name": "endlongitude", "type": "string" } ] } } ] }

And the data, that it maps. BTW, this is just an example data.

{ "terminal": { "success": "string", "data": [ { "ID": 0 } ], "count": 0 }, "driver": { "success": "string", "data": [ { "ID": 0 } ], "count": 0 }, "ticket": { "success": "string", "data": { "ID": 0, "job": { "ID": 0 } } }, "order": { "externalJobNo": "string", "jobNo": "string", "driverName": "string", "startTime": "string", "endTime": "string", "comment": "string", "readStart": "string", "readEnd": "string", "removed": "string", "installed": "string", "endTopCut": "string", "endBottomCut": "string", "startTemp": "string", "endTemp": "string", "swPer": "string", "gravity": "string", "volume": "string", "grossVolume": "string", "temp": "string", "dist": "string", "explanation": "string", "status": "string", "driverID": 0, "billingQuestionComment": "string" }, "jobTimeType": { "startTime": "string", "endTime": "string", "startlatitude": "string", "startlongitude": "string", "endlatitude": "string", "endlongitude": "string" }, "jobTimeTypeTerminal": { "terminal": "string", "startTime": "string", "endTime": "string", "startlatitude": "string", "startlongitude": "string", "endlatitude": "string", "endlongitude": "string" } }

So, for testing, I tried to use the GenerateFlowFile processor, with the exact sample data above, and a ConvertRecord processor set to Infer Schema, to make NiFi generate the schema so I can spot any errors, but when I do that, I get the same error message.

The last piece of information I have is that this error started when I added the "terminal" and the "driver" JSON attributes. Without it, everything seems to work correctly.

Re: Can't redefine org.apache.nifi.dataType - Schema error

araujo — Thu, 10 Feb 2022 03:47:11 GMT

@hjfigueira ,

I ran a test with your schema and data and it worked without problems for me.

I think there's something else in your environment that hasn't been described here that's contributing to the problem.

André

Re: Can't redefine org.apache.nifi.dataType - Schema error

hjfigueira — Thu, 10 Feb 2022 20:13:34 GMT

Thank you very much @araujo.

Finally, I manage to solve that, I'm just unsure about the true cause tho. What I did was, on the service that writes the JSON, in the "Schema Access Strategy" instead of using "Inherit Schema" I used the "Schema Text" property, and used that same schema. ( I already had it into the reader ).

I suspect that somehow, the schemas that NiFi was generating, contained duplicated data, but I have no hard evidence on that so, it's just speculation.

Re: Can't redefine org.apache.nifi.dataType - Schema error

araujo — Thu, 10 Feb 2022 23:33:01 GMT

Which version of NiFi are you using?

Re: Can't redefine org.apache.nifi.dataType - Schema error

hjfigueira — Fri, 11 Feb 2022 15:29:25 GMT

1.13.2

Re: Can't redefine org.apache.nifi.dataType - Schema error

araujo — Fri, 11 Feb 2022 21:52:06 GMT

Thanks!

question Re: Can't redefine org.apache.nifi.dataType - Schema error in Support Questions

Can't redefine org.apache.nifi.dataType - Schema error

Re: Can't redefine org.apache.nifi.dataType - Schema error

Re: Can't redefine org.apache.nifi.dataType - Schema error

Re: Can't redefine org.apache.nifi.dataType - Schema error

Re: Can't redefine org.apache.nifi.dataType - Schema error

Re: Can't redefine org.apache.nifi.dataType - Schema error