Member since
09-18-2018
92
Posts
5
Kudos Received
0
Solutions
04-10-2019
02:26 PM
24:40,136 ERROR [Timer-Driven Process Thread-5] o.a.n.p.jolt.record.JoltTransformRecord JoltTransformRecord[id=06fd25cf-016a-1000-780f-11032c386125] Unable to write transformed records StandardFlowFileRecord[uuid=6626d078-5d3d-4156-988f-af57bd8efd7e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1554901130806-97, container=default, section=97], offset=171543, length=86903],offset=0,name=6626d078-5d3d-4156-988f-af57bd8efd7e,size=86903] due to org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [MapRecord[{timeAtPosition=2019-04-09T18:24:27Z, reportedAltitude=MapRecord[{assignedAltitude=MapRecord[{simpleAltitude=350}]}], qualifiedAircraftId=MapRecord[{computerId=MapRecord[{facilityIdentifier=CCZM}], gufi=KN44418300, aircraftId=ICL922, departurePoint=MapRecord[{airport=KJFK}], arrivalPoint=MapRecord[{airport=EBLG}], igtd=2019-04-09T17:00:00Z}], ncsmTrackData=MapRecord[{eta=MapRecord[{etaType=ESTIMATED, timeValue=2019-04-09T23:18:14Z}], departureFixAndTime=MapRecord[{fixName=MERIT, arrTime=2019-04-09T17:03:10Z}], nextEvent=MapRecord[{longitudeDecimal=-55.45632194020566, latitudeDecimal=46.91005273931668}], rvsmData=MapRecord[{futureCompliance=true, equipped=true, currentCompliance=true}]}], position=MapRecord[{latitude=MapRecord[{latitudeDMS=MapRecord[{minutes=38, degrees=46, direction=NORTH}]}], longitude=MapRecord[{longitudeDMS=MapRecord[{minutes=32, degrees=056, direction=WEST}]}]}], speed=550}]] of type CHOICE[RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD] to Map for field trackInformation because the type is not supported: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [MapRecord[{timeAtPosition=2019-04-09T18:24:27Z, reportedAltitude=MapRecord[{assignedAltitude=MapRecord[{simpleAltitude=350}]}], qualifiedAircraftId=MapRecord[{computerId=MapRecord[{facilityIdentifier=CCZM}], gufi=KN44418300, aircraftId=ICL922, departurePoint=MapRecord[{airport=KJFK}], arrivalPoint=MapRecord[{airport=EBLG}], igtd=2019-04-09T17:00:00Z}], ncsmTrackData=MapRecord[{eta=MapRecord[{etaType=ESTIMATED, timeValue=2019-04-09T23:18:14Z}], departureFixAndTime=MapRecord[{fixName=MERIT, arrTime=2019-04-09T17:03:10Z}], nextEvent=MapRecord[{longitudeDecimal=-55.45632194020566, latitudeDecimal=46.91005273931668}], rvsmData=MapRecord[{futureCompliance=true, equipped=true, currentCompliance=true}]}], position=MapRecord[{latitude=MapRecord[{latitudeDMS=MapRecord[{minutes=38, degrees=46, direction=NORTH}]}], longitude=MapRecord[{longitudeDMS=MapRecord[{minutes=32, degrees=056, direction=WEST}]}]}], speed=550}]] of type CHOICE[RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD] to Map for field trackInformation because the type is not supported
org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [MapRecord[{timeAtPosition=2019-04-09T18:24:27Z, reportedAltitude=MapRecord[{assignedAltitude=MapRecord[{simpleAltitude=350}]}], qualifiedAircraftId=MapRecord[{computerId=MapRecord[{facilityIdentifier=CCZM}], gufi=KN44418300, aircraftId=ICL922, departurePoint=MapRecord[{airport=KJFK}], arrivalPoint=MapRecord[{airport=EBLG}], igtd=2019-04-09T17:00:00Z}], ncsmTrackData=MapRecord[{eta=MapRecord[{etaType=ESTIMATED, timeValue=2019-04-09T23:18:14Z}], departureFixAndTime=MapRecord[{fixName=MERIT, arrTime=2019-04-09T17:03:10Z}], nextEvent=MapRecord[{longitudeDecimal=-55.45632194020566, latitudeDecimal=46.91005273931668}], rvsmData=MapRecord[{futureCompliance=true, equipped=true, currentCompliance=true}]}], position=MapRecord[{latitude=MapRecord[{latitudeDMS=MapRecord[{minutes=38, degrees=46, direction=NORTH}]}], longitude=MapRecord[{longitudeDMS=MapRecord[{minutes=32, degrees=056, direction=WEST}]}]}], speed=550}]] of type CHOICE[RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD, RECORD] to Map for field trackInformation because the type is not supported
at org.apache.nifi.serialization.record.util.DataTypeUtils.convertRecordFieldtoObject(DataTypeUtils.java:693)
at org.apache.nifi.serialization.record.util.DataTypeUtils.convertRecordArrayToJavaArray(DataTypeUtils.java:732)
at org.apache.nifi.serialization.record.util.DataTypeUtils.convertRecordFieldtoObject(DataTypeUtils.java:691)
at org.apache.nifi.serialization.record.util.DataTypeUtils.convertRecordFieldtoObject(DataTypeUtils.java:686)
at org.apache.nifi.processors.jolt.record.JoltTransformRecord.transform(JoltTransformRecord.java:380)
at org.apache.nifi.processors.jolt.record.JoltTransformRecord.onTrigger(JoltTransformRecord.java:333)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
... View more
04-10-2019
02:17 PM
I have seen a similar problem reported here. Worthy of note, is that I think the problem is data specific. There are XML records that seem to work fine in the test flow I've provided. This does not make sense to me however since as I said the JoltTransformJSON works fine for the generated JSON (even for the problem data).
... View more
04-10-2019
01:44 PM
The specific error I see is as follows: Here is the associated template for the flow above. You can use it to verify the problem I am having: Test_Jolt_Transform_Record.xml
... View more
04-10-2019
01:05 PM
Hi @Matt Burgess. I've tried JoltTransformRecord. Its not behaving as I'd expect. In the following you'll see that I generate a single XML record, I convert it using JoltTransformRecord (that fails). I also convert it using the same XMLReader and JSONSetWriter using ConvertRecord. I then pipe that converted JSON to a separate JoltTransformJSON, that is using the same JOLT Transform as the original JoltTransformRecord. The JoltTransformJSON succeeds. The configuration of the JoltTransformRecord is as follows: The overall flow is as follows: What am I missing with the use of the JoltTransformRecord?
... View more
04-10-2019
10:11 AM
Hi @Matt Burgess All your points make sense. In terms of 3 above, I only thought that I might extract the inferred schema for performance reasons. As it stands the XMLReader will be forced to infer the schema with each received record. The nature of my incoming XML data is that it contains data with optional fields, and hence the inferred schema will change. My thought was that if I can determine the "superset schema" and apply that then I'd eliminate the need for inference (buying back the associated performance). In terms of 4 above, it sounds like what I will want to do is JoltTransformRecord (instead of ConvertRecord). The end result will be that I both convert to JSON and change the structure to a more suitable form. Thanks very much for your detailed replies. They are hugely useful. With feedback from folks such as you I am bringing my comfort with NIFI up. My role currently is to architect our future solutions, and in that context I am looking for the technologies that we might integrate to solve problems in new and better ways. I'm very bullish about NIFI especially when I see folks from Hortonworks support it so well. I'm keen to try some of the other components that Hortonworks also supports. I do have a separate posted question about streaming messages into buffered queues that I'd love your architectural perspective on.
... View more
04-09-2019
06:43 PM
Hi @Matt Burgess I've tried 1.9.2. Sure enough the XMLRecordReader works with message inference. Using this as part of ConvertRecord allowed me to easily convert from XML to JSON (avoding the use of TransformXML with an xslt sheet). Four questions for you. 1] Would you expect ConvertRecord or TransformXML to be more flexible? 2] Would you expect ConvertRecord or TransformXML to provide better performance? Performance is key for me - i have a lot of data. 3] How can I inspect the generated flowfiles to see the actual AVRO format? I thought it might be placed on an attribute within the generated flowfile, however I did not see this. 4] You also suggested trying JoltTransformRecord. Would this be instead of ConvertRecord and TransformXML (or in addition to)?
... View more
04-08-2019
06:23 PM
Ty @Matt Burgess. I can try that. I have a continuous large stream of XML to handle. I've been assuming that using ConvertRecord in order to convert the XML into JSON is faster than using XMLTransform to do the same. Is this your experience? Also I am hoping that ConvertRecord (and AVRO output specs) will give me more flexibility in trimming down the output. i dont need all the input data in my output JSON format (I only need some of it).
... View more
04-08-2019
05:49 PM
The XML that I have is as follows: <?xml version="1.0" encoding="UTF-8"?>
<dataService> <dataOutput> <fltdMessage acid="QXE2828" airline="QXE" arrArpt="KSJC" cdmPart="false" depArpt="KPAE" fdTrigger="HCS_TRACK_MSG" flightRef="97606500" msgType="trackInformation" sensitivity="A" sourceFacility="KZSE" sourceTimeStamp="2019-04-05T16:44:30Z"> <trackInformation> <qualifiedAircraftId> <aircraftId>QXE2828</aircraftId> <computerId> <facilityIdentifier>KZSE</facilityIdentifier> <idNumber>657</idNumber> </computerId> <gufi>KS49063601</gufi> <igtd>2019-04-05T15:30:00Z</igtd> <departurePoint> <airport>KPAE</airport> </departurePoint> <arrivalPoint> <airport>KSJC</airport> </arrivalPoint> </qualifiedAircraftId> <speed>378</speed> <reportedAltitude> <assignedAltitude> <simpleAltitude>330C</simpleAltitude> </assignedAltitude> </reportedAltitude> <position> <latitude> <latitudeDMS degrees="41" direction="NORTH" minutes="14" seconds="14" /> </latitude> <longitude> <longitudeDMS degrees="122" direction="WEST" minutes="49" seconds="03" /> </longitude> </position> <timeAtPosition>2019-04-05T16:44:30Z</timeAtPosition> <ncsmTrackData> <eta etaType="ESTIMATED" timeValue="2019-04-05T17:29:51Z" /> <rvsmData currentCompliance="true" equipped="true" futureCompliance="true" /> <arrivalFixAndTime arrTime="2019-04-05T17:13:34Z" fixName="ZINNN" /> <nextEvent latitudeDecimal="39.125619878405274" longitudeDecimal="-122.59562140202789" /> </ncsmTrackData> </trackInformation> </fltdMessage> </dataOutput>
</dataService> I'd like to generate the AVRO spec for this so that I can use ConvertRecord with an XMLReader. Is there an easy way to do this? The InferAvroSchema processor does not seem to support this structure, as can be seen by the "unsupported content" in the following. I belive this processor only supports csv and json. So I think I need to find another means to generate the AVRO schema. I think this may be close. I converted XML to JSON then I inferred the AVRO schema from the JSON. However this does not work as the AVRO schema for converting the XML directly. {
"type" : "record",
"name" : "nice",
"fields" : [ {
"name" : "dataService",
"type" : {
"type" : "record",
"name" : "dataService",
"fields" : [ {
"name" : "dataOutput",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "dataOutput",
"fields" : [ {
"name" : "fltdMessage",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "fltdMessage.dataOutput",
"fields" : [ {
"name" : "acid",
"type" : "string",
"doc" : "Type inferred from '\"QXE2828\"'"
}, {
"name" : "airline",
"type" : "string",
"doc" : "Type inferred from '\"QXE\"'"
}, {
"name" : "arrArpt",
"type" : "string",
"doc" : "Type inferred from '\"KSJC\"'"
}, {
"name" : "cdmPart",
"type" : "boolean",
"doc" : "Type inferred from 'false'"
}, {
"name" : "depArpt",
"type" : "string",
"doc" : "Type inferred from '\"KPAE\"'"
}, {
"name" : "fdTrigger",
"type" : "string",
"doc" : "Type inferred from '\"HCS_TRACK_MSG\"'"
}, {
"name" : "flightRef",
"type" : "int",
"doc" : "Type inferred from '97606500'"
}, {
"name" : "msgType",
"type" : "string",
"doc" : "Type inferred from '\"trackInformation\"'"
}, {
"name" : "sensitivity",
"type" : "string",
"doc" : "Type inferred from '\"A\"'"
}, {
"name" : "sourceFacility",
"type" : "string",
"doc" : "Type inferred from '\"KZSE\"'"
}, {
"name" : "sourceTimeStamp",
"type" : "string",
"doc" : "Type inferred from '\"2019-04-05T16:44:30Z\"'"
}, {
"name" : "trackInformation",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "qualifiedAircraftId",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "qualifiedAircraftId.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "aircraftId",
"type" : "string",
"doc" : "Type inferred from '\"QXE2828\"'"
}, {
"name" : "computerId",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "computerId.qualifiedAircraftId.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "facilityIdentifier",
"type" : "string",
"doc" : "Type inferred from '\"KZSE\"'"
}, {
"name" : "idNumber",
"type" : "int",
"doc" : "Type inferred from '657'"
} ]
},
"doc" : "Type inferred from '{\"facilityIdentifier\":\"KZSE\",\"idNumber\":657}'"
}, {
"name" : "gufi",
"type" : "string",
"doc" : "Type inferred from '\"KS49063601\"'"
}, {
"name" : "igtd",
"type" : "string",
"doc" : "Type inferred from '\"2019-04-05T15:30:00Z\"'"
}, {
"name" : "departurePoint",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "departurePoint.qualifiedAircraftId.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "airport",
"type" : "string",
"doc" : "Type inferred from '\"KPAE\"'"
} ]
},
"doc" : "Type inferred from '{\"airport\":\"KPAE\"}'"
}, {
"name" : "arrivalPoint",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "arrivalPoint.qualifiedAircraftId.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "airport",
"type" : "string",
"doc" : "Type inferred from '\"KSJC\"'"
} ]
},
"doc" : "Type inferred from '{\"airport\":\"KSJC\"}'"
} ]
},
"doc" : "Type inferred from '{\"aircraftId\":\"QXE2828\",\"computerId\":{\"facilityIdentifier\":\"KZSE\",\"idNumber\":657},\"gufi\":\"KS49063601\",\"igtd\":\"2019-04-05T15:30:00Z\",\"departurePoint\":{\"airport\":\"KPAE\"},\"arrivalPoint\":{\"airport\":\"KSJC\"}}'"
}, {
"name" : "speed",
"type" : "int",
"doc" : "Type inferred from '378'"
}, {
"name" : "reportedAltitude",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "reportedAltitude.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "assignedAltitude",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "assignedAltitude.reportedAltitude.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "simpleAltitude",
"type" : "string",
"doc" : "Type inferred from '\"330C\"'"
} ]
},
"doc" : "Type inferred from '{\"simpleAltitude\":\"330C\"}'"
} ]
},
"doc" : "Type inferred from '{\"assignedAltitude\":{\"simpleAltitude\":\"330C\"}}'"
}, {
"name" : "position",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "position.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "latitude",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "latitude.position.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "latitudeDMS",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "latitudeDMS.latitude.position.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "degrees",
"type" : "int",
"doc" : "Type inferred from '41'"
}, {
"name" : "direction",
"type" : "string",
"doc" : "Type inferred from '\"NORTH\"'"
}, {
"name" : "minutes",
"type" : "int",
"doc" : "Type inferred from '14'"
}, {
"name" : "seconds",
"type" : "int",
"doc" : "Type inferred from '14'"
} ]
},
"doc" : "Type inferred from '{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}'"
} ]
},
"doc" : "Type inferred from '{\"latitudeDMS\":{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}}'"
}, {
"name" : "longitude",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "longitude.position.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "longitudeDMS",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "longitudeDMS.longitude.position.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "degrees",
"type" : "int",
"doc" : "Type inferred from '122'"
}, {
"name" : "direction",
"type" : "string",
"doc" : "Type inferred from '\"WEST\"'"
}, {
"name" : "minutes",
"type" : "int",
"doc" : "Type inferred from '49'"
}, {
"name" : "seconds",
"type" : "string",
"doc" : "Type inferred from '\"03\"'"
} ]
},
"doc" : "Type inferred from '{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}'"
} ]
},
"doc" : "Type inferred from '{\"longitudeDMS\":{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}}'"
} ]
},
"doc" : "Type inferred from '{\"latitude\":{\"latitudeDMS\":{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}},\"longitude\":{\"longitudeDMS\":{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}}}'"
}, {
"name" : "timeAtPosition",
"type" : "string",
"doc" : "Type inferred from '\"2019-04-05T16:44:30Z\"'"
}, {
"name" : "ncsmTrackData",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "ncsmTrackData.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "eta",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "eta.ncsmTrackData.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "etaType",
"type" : "string",
"doc" : "Type inferred from '\"ESTIMATED\"'"
}, {
"name" : "timeValue",
"type" : "string",
"doc" : "Type inferred from '\"2019-04-05T17:29:51Z\"'"
} ]
},
"doc" : "Type inferred from '{\"etaType\":\"ESTIMATED\",\"timeValue\":\"2019-04-05T17:29:51Z\"}'"
}, {
"name" : "rvsmData",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "rvsmData.ncsmTrackData.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "currentCompliance",
"type" : "boolean",
"doc" : "Type inferred from 'true'"
}, {
"name" : "equipped",
"type" : "boolean",
"doc" : "Type inferred from 'true'"
}, {
"name" : "futureCompliance",
"type" : "boolean",
"doc" : "Type inferred from 'true'"
} ]
},
"doc" : "Type inferred from '{\"currentCompliance\":true,\"equipped\":true,\"futureCompliance\":true}'"
}, {
"name" : "arrivalFixAndTime",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "arrivalFixAndTime.ncsmTrackData.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "arrTime",
"type" : "string",
"doc" : "Type inferred from '\"2019-04-05T17:13:34Z\"'"
}, {
"name" : "fixName",
"type" : "string",
"doc" : "Type inferred from '\"ZINNN\"'"
} ]
},
"doc" : "Type inferred from '{\"arrTime\":\"2019-04-05T17:13:34Z\",\"fixName\":\"ZINNN\"}'"
}, {
"name" : "nextEvent",
"type" : {
"type" : "record",
"name" : "dataService",
"namespace" : "nextEvent.ncsmTrackData.trackInformation.fltdMessage.dataOutput",
"fields" : [ {
"name" : "latitudeDecimal",
"type" : "double",
"doc" : "Type inferred from '39.125619878405274'"
}, {
"name" : "longitudeDecimal",
"type" : "double",
"doc" : "Type inferred from '-122.59562140202789'"
} ]
},
"doc" : "Type inferred from '{\"latitudeDecimal\":39.125619878405274,\"longitudeDecimal\":-122.59562140202789}'"
} ]
},
"doc" : "Type inferred from '{\"eta\":{\"etaType\":\"ESTIMATED\",\"timeValue\":\"2019-04-05T17:29:51Z\"},\"rvsmData\":{\"currentCompliance\":true,\"equipped\":true,\"futureCompliance\":true},\"arrivalFixAndTime\":{\"arrTime\":\"2019-04-05T17:13:34Z\",\"fixName\":\"ZINNN\"},\"nextEvent\":{\"latitudeDecimal\":39.125619878405274,\"longitudeDecimal\":-122.59562140202789}}'"
} ]
},
"doc" : "Type inferred from '{\"qualifiedAircraftId\":{\"aircraftId\":\"QXE2828\",\"computerId\":{\"facilityIdentifier\":\"KZSE\",\"idNumber\":657},\"gufi\":\"KS49063601\",\"igtd\":\"2019-04-05T15:30:00Z\",\"departurePoint\":{\"airport\":\"KPAE\"},\"arrivalPoint\":{\"airport\":\"KSJC\"}},\"speed\":378,\"reportedAltitude\":{\"assignedAltitude\":{\"simpleAltitude\":\"330C\"}},\"position\":{\"latitude\":{\"latitudeDMS\":{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}},\"longitude\":{\"longitudeDMS\":{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}}},\"timeAtPosition\":\"2019-04-05T16:44:30Z\",\"ncsmTrackData\":{\"eta\":{\"etaType\":\"ESTIMATED\",\"timeValue\":\"2019-04-05T17:29:51Z\"},\"rvsmData\":{\"currentCompliance\":true,\"equipped\":true,\"futureCompliance\":true},\"arrivalFixAndTime\":{\"arrTime\":\"2019-04-05T17:13:34Z\",\"fixName\":\"ZINNN\"},\"nextEvent\":{\"latitudeDecimal\":39.125619878405274,\"longitudeDecimal\":-122.59562140202789}}}'"
} ]
},
"doc" : "Type inferred from '{\"acid\":\"QXE2828\",\"airline\":\"QXE\",\"arrArpt\":\"KSJC\",\"cdmPart\":false,\"depArpt\":\"KPAE\",\"fdTrigger\":\"HCS_TRACK_MSG\",\"flightRef\":97606500,\"msgType\":\"trackInformation\",\"sensitivity\":\"A\",\"sourceFacility\":\"KZSE\",\"sourceTimeStamp\":\"2019-04-05T16:44:30Z\",\"trackInformation\":{\"qualifiedAircraftId\":{\"aircraftId\":\"QXE2828\",\"computerId\":{\"facilityIdentifier\":\"KZSE\",\"idNumber\":657},\"gufi\":\"KS49063601\",\"igtd\":\"2019-04-05T15:30:00Z\",\"departurePoint\":{\"airport\":\"KPAE\"},\"arrivalPoint\":{\"airport\":\"KSJC\"}},\"speed\":378,\"reportedAltitude\":{\"assignedAltitude\":{\"simpleAltitude\":\"330C\"}},\"position\":{\"latitude\":{\"latitudeDMS\":{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}},\"longitude\":{\"longitudeDMS\":{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}}},\"timeAtPosition\":\"2019-04-05T16:44:30Z\",\"ncsmTrackData\":{\"eta\":{\"etaType\":\"ESTIMATED\",\"timeValue\":\"2019-04-05T17:29:51Z\"},\"rvsmData\":{\"currentCompliance\":true,\"equipped\":true,\"futureCompliance\":true},\"arrivalFixAndTime\":{\"arrTime\":\"2019-04-05T17:13:34Z\",\"fixName\":\"ZINNN\"},\"nextEvent\":{\"latitudeDecimal\":39.125619878405274,\"longitudeDecimal\":-122.59562140202789}}}}'"
} ]
},
"doc" : "Type inferred from '{\"fltdMessage\":{\"acid\":\"QXE2828\",\"airline\":\"QXE\",\"arrArpt\":\"KSJC\",\"cdmPart\":false,\"depArpt\":\"KPAE\",\"fdTrigger\":\"HCS_TRACK_MSG\",\"flightRef\":97606500,\"msgType\":\"trackInformation\",\"sensitivity\":\"A\",\"sourceFacility\":\"KZSE\",\"sourceTimeStamp\":\"2019-04-05T16:44:30Z\",\"trackInformation\":{\"qualifiedAircraftId\":{\"aircraftId\":\"QXE2828\",\"computerId\":{\"facilityIdentifier\":\"KZSE\",\"idNumber\":657},\"gufi\":\"KS49063601\",\"igtd\":\"2019-04-05T15:30:00Z\",\"departurePoint\":{\"airport\":\"KPAE\"},\"arrivalPoint\":{\"airport\":\"KSJC\"}},\"speed\":378,\"reportedAltitude\":{\"assignedAltitude\":{\"simpleAltitude\":\"330C\"}},\"position\":{\"latitude\":{\"latitudeDMS\":{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}},\"longitude\":{\"longitudeDMS\":{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}}},\"timeAtPosition\":\"2019-04-05T16:44:30Z\",\"ncsmTrackData\":{\"eta\":{\"etaType\":\"ESTIMATED\",\"timeValue\":\"2019-04-05T17:29:51Z\"},\"rvsmData\":{\"currentCompliance\":true,\"equipped\":true,\"futureCompliance\":true},\"arrivalFixAndTime\":{\"arrTime\":\"2019-04-05T17:13:34Z\",\"fixName\":\"ZINNN\"},\"nextEvent\":{\"latitudeDecimal\":39.125619878405274,\"longitudeDecimal\":-122.59562140202789}}}}}'"
} ]
},
"doc" : "Type inferred from '{\"dataOutput\":{\"fltdMessage\":{\"acid\":\"QXE2828\",\"airline\":\"QXE\",\"arrArpt\":\"KSJC\",\"cdmPart\":false,\"depArpt\":\"KPAE\",\"fdTrigger\":\"HCS_TRACK_MSG\",\"flightRef\":97606500,\"msgType\":\"trackInformation\",\"sensitivity\":\"A\",\"sourceFacility\":\"KZSE\",\"sourceTimeStamp\":\"2019-04-05T16:44:30Z\",\"trackInformation\":{\"qualifiedAircraftId\":{\"aircraftId\":\"QXE2828\",\"computerId\":{\"facilityIdentifier\":\"KZSE\",\"idNumber\":657},\"gufi\":\"KS49063601\",\"igtd\":\"2019-04-05T15:30:00Z\",\"departurePoint\":{\"airport\":\"KPAE\"},\"arrivalPoint\":{\"airport\":\"KSJC\"}},\"speed\":378,\"reportedAltitude\":{\"assignedAltitude\":{\"simpleAltitude\":\"330C\"}},\"position\":{\"latitude\":{\"latitudeDMS\":{\"degrees\":41,\"direction\":\"NORTH\",\"minutes\":14,\"seconds\":14}},\"longitude\":{\"longitudeDMS\":{\"degrees\":122,\"direction\":\"WEST\",\"minutes\":49,\"seconds\":\"03\"}}},\"timeAtPosition\":\"2019-04-05T16:44:30Z\",\"ncsmTrackData\":{\"eta\":{\"etaType\":\"ESTIMATED\",\"timeValue\":\"2019-04-05T17:29:51Z\"},\"rvsmData\":{\"currentCompliance\":true,\"equipped\":true,\"futureCompliance\":true},\"arrivalFixAndTime\":{\"arrTime\":\"2019-04-05T17:13:34Z\",\"fixName\":\"ZINNN\"},\"nextEvent\":{\"latitudeDecimal\":39.125619878405274,\"longitudeDecimal\":-122.59562140202789}}}}}}'"
} ]
}
... View more
Labels:
- Labels:
-
Apache NiFi
10-31-2018
03:28 PM
Thank you @Matt Burgess for the key operation that i was missing. Keep in mind, I still need the shift operation since I am reducing a more complex JSON as part of the shift So the entire spec contains two operations as follows.. [
{
"operation": "shift",
"spec": {
"textNOTAM": {
"NOTAM": {
"id": "&0",
"number": "&0",
"year": "&0",
"type": "&0",
"issued": "&0",
"affectedFIR": "&0",
"minimumFL": "&0",
"maximumFL": "&0",
"radius": "&0",
"location": "&0",
"effectiveStart": "&0",
"effectiveEnd": "&0",
"schedule": "&0",
"text": "&0"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"effectiveStart": "=toString",
"effectiveEnd": "=toString"
}
}
]
... View more
10-30-2018
12:59 PM
1 Kudo
The following code is the current JOLT transformation that I am using in a NIFI JoltTransformJSON processor: [
{
"operation": "shift",
"spec": {
"textNOTAM": {
"NOTAM": {
// simple match. Put the value '4' in the output under the "Rating" field
"id": "&0",
"number": "&0",
"year": "&0",
"type": "&0",
"issued": "&0",
"affectedFIR": "&0",
"minimumFL": "&0",
"maximumFL": "&0",
"radius": "&0",
"location": "&0",
"effectiveStart": "&0",
"effectiveEnd": "&0",
"schedule": "&0",
"text": "&0"
}
}
}
}]
A typical flow file coming out of my JOLT transformation looks like this: {"id":"NOTAM_1_51227850","number":376,"year":2018,"type":"N","issued":"2018-10-30T12:29:00.000Z","affectedFIR":"LIMM","minimumFL":"000","maximumFL":999,"radius":5,"location":"LIPA","effectiveStart":201810301229,"effectiveEnd":201811032359,"text":"NON-STANDARD BARRIER CONFIGURATION. BARRIER 2 (BAK-12) UNSERVICEABLE"} The problem I am having is with the effectiveStart and effectiveEnd fields. I need the value to be a string, not a number. How can I adjust the JOLT transformation above to ensure that it is a string? The JSON that comes out of this processor, I then inject into MONGO using a PutMongo processor. Because of this input format problem, with effectiveStart and effectiveEnd, they are coming out of mongo as follows: { "_id" : ObjectId("5bc8b2808aad6206ababaa4b"), "id" : "NOTAM_1_51105982", "number" : 7, "year" : 2018, "type" : "N", "issued" : "2018-10-16T00:13:00.000Z", "affectedFIR" : "ZBW", "minimumFL" : "000", "maximumFL" : 999, "radius" : 5, "location" : "OXC", "effectiveStart" : NumberLong("201810160013"), "effectiveEnd" : "201810312359EST", "text" : "SVC AUTOMATED WX BCST SYSTEM PRECIPITATION OUT OF SERVICE" } Notice that the effectiveStart is a NumberLong. How can I adjust the JOLT transform to ensure that the effectiveStart and effectiveEnd are seen as strings when the JSON flow file is sent to PutMongo?
... View more
Labels:
- Labels:
-
Apache NiFi