Member since
06-21-2021
4
Posts
0
Kudos Received
0
Solutions
02-10-2022
12:13 PM
Thank you very much @araujo. Finally, I manage to solve that, I'm just unsure about the true cause tho. What I did was, on the service that writes the JSON, in the "Schema Access Strategy" instead of using "Inherit Schema" I used the "Schema Text" property, and used that same schema. ( I already had it into the reader ). I suspect that somehow, the schemas that NiFi was generating, contained duplicated data, but I have no hard evidence on that so, it's just speculation.
... View more
02-09-2022
10:32 AM
Hi Guys,
I'm currently having an issue where a schema that I'm using is erroring out in a RecordLookup processor ( with a Rest Service ). The message I get is the one in the title of this post, "Can't redefine org.apache.nifi.dataType". Below you can find the schema that I'm using.
{
"type": "record",
"name": "FullTicketData",
"fields": [
{
"name" : "terminal",
"type": {
"type": "record",
"name" : "terminal_array",
"fields": [
{
"name": "success",
"type": "string"
},
{
"name": "data",
"type": {
"type": "array",
"items": {
"name": "data_array",
"type": "record",
"fields": [
{
"name": "ID",
"type": "int"
}
]
}
}
},
{
"name": "count",
"type": "int"
}
]
}
},
{
"name" : "driver",
"type": {
"type": "record",
"name" : "driver",
"fields": [
{
"name": "success",
"type": "string"
},
{
"name": "data",
"type": {
"type": "array",
"items": {
"name": "data_record",
"type": "record",
"fields": [
{
"name": "ID",
"type": "int"
}
]
}
}
},
{
"name": "count",
"type": "int"
}
]
}
},
{
"name": "ticket",
"type": {
"name" : "ticket",
"type" : "record",
"fields": [
{
"name": "success",
"type": "string"
},
{
"name": "data",
"type": {
"name": "data",
"type": "record",
"fields": [
{
"name": "ID",
"type": "int"
},
{
"name": "job",
"type": {
"name": "job",
"type": "record",
"fields": [
{
"name": "ID",
"type": "int"
}
]
}
}
]
}
}
]
}
},
{
"name": "order",
"type": {
"name": "order",
"type": "record",
"fields": [
{
"name": "externalJobNo",
"type": "string"
},
{
"name": "jobNo",
"type": "string"
},
{
"name": "driverName",
"type": "string"
},
{
"name": "startTime",
"type": "string"
},
{
"name": "endTime",
"type": "string"
},
{
"name": "comment",
"type": "string",
"default": "empty"
},
{
"name": "readStart",
"type": "string",
"default": "0"
},
{
"name": "readEnd",
"type": "string",
"default": "0"
},
{
"name": "removed",
"type": "string"
},
{
"name": "installed",
"type": "string"
},
{
"name": "endTopCut",
"type": "string"
},
{
"name": "endBottomCut",
"type": "string"
},
{
"name": "startTemp",
"type": "string"
},
{
"name": "endTemp",
"type": "string"
},
{
"name": "swPer",
"type": "string"
},
{
"name": "gravity",
"type": "string"
},
{
"name": "volume",
"type": "string"
},
{
"name": "grossVolume",
"type": "string"
},
{
"name": "temp",
"type": "string"
},
{
"name": "dist",
"type": "string"
},
{
"name": "explanation",
"type": "string"
},
{
"name": "status",
"type": "string"
},
{
"name": "driverID",
"type": "int"
},
{
"name": "billingQuestionComment",
"type": "string",
"default": "empty"
}
]
}
},
{
"name": "jobTimeType",
"type": {
"name": "jobTimeType",
"type": "record",
"fields": [
{
"name": "startTime",
"type": "string"
},
{
"name": "endTime",
"type": "string"
},
{
"name": "startlatitude",
"type": "string"
},
{
"name": "startlongitude",
"type": "string"
},
{
"name": "endlatitude",
"type": "string"
},
{
"name": "endlongitude",
"type": "string"
}
]
}
},
{
"name": "jobTimeTypeTerminal",
"type": {
"name": "jobTimeTypeTerminal",
"type": "record",
"fields": [
{
"name": "terminal",
"type": "string"
},
{
"name": "startTime",
"type": "string"
},
{
"name": "endTime",
"type": "string"
},
{
"name": "startlatitude",
"type": "string"
},
{
"name": "startlongitude",
"type": "string"
},
{
"name": "endlatitude",
"type": "string"
},
{
"name": "endlongitude",
"type": "string"
}
]
}
}
]
}
And the data, that it maps. BTW, this is just an example data.
{
"terminal": {
"success": "string",
"data": [
{
"ID": 0
}
],
"count": 0
},
"driver": {
"success": "string",
"data": [
{
"ID": 0
}
],
"count": 0
},
"ticket": {
"success": "string",
"data": {
"ID": 0,
"job": {
"ID": 0
}
}
},
"order": {
"externalJobNo": "string",
"jobNo": "string",
"driverName": "string",
"startTime": "string",
"endTime": "string",
"comment": "string",
"readStart": "string",
"readEnd": "string",
"removed": "string",
"installed": "string",
"endTopCut": "string",
"endBottomCut": "string",
"startTemp": "string",
"endTemp": "string",
"swPer": "string",
"gravity": "string",
"volume": "string",
"grossVolume": "string",
"temp": "string",
"dist": "string",
"explanation": "string",
"status": "string",
"driverID": 0,
"billingQuestionComment": "string"
},
"jobTimeType": {
"startTime": "string",
"endTime": "string",
"startlatitude": "string",
"startlongitude": "string",
"endlatitude": "string",
"endlongitude": "string"
},
"jobTimeTypeTerminal": {
"terminal": "string",
"startTime": "string",
"endTime": "string",
"startlatitude": "string",
"startlongitude": "string",
"endlatitude": "string",
"endlongitude": "string"
}
}
So, for testing, I tried to use the GenerateFlowFile processor, with the exact sample data above, and a ConvertRecord processor set to Infer Schema, to make NiFi generate the schema so I can spot any errors, but when I do that, I get the same error message.
The last piece of information I have is that this error started when I added the "terminal" and the "driver" JSON attributes. Without it, everything seems to work correctly.
... View more
Labels:
- Labels:
-
Apache NiFi
06-21-2021
11:35 AM
I'm having a similar issue. The SFTP processors are also not being able to connect. (also on jre 😎 One thing that might help, is that running nifi using the official docker image, works fine. And, the JRE on that container is also the jre8 but not the same exact version, the version on the container is a bit older than mine. Haven't had the time to test this out, but it might help; Also, on local docker and on local host, "it works on my machine". The faulty one is on AWS Have you guys achieved any progress on the issue?
... View more