Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

ConvertJSONToAvro not working despite successful with avro-tools validation

New Contributor

jsontoavroissue.zip

Hi -

Problem statement: I have a XML file which is converted to JSON. This JSON is sent to ConvertJSONToAvro where it fails. This same JSON output when run against same .asvc file in avro-tools works fine and generates the output. Can someone please look at the attached artifacts and help.

command tried:

java -jar avro-tools-1.8.1.jar fromjson --schema-file TMSDataModelFlat.avsc 90573445988483.json > validated.avro

5 REPLIES 5

Hi,

Try a SplitJson processor before the ConvertJsonToAvro one, it works like this for me !

New Contributor

Thank you, Quentin. Can you please share what should be the JsonXpath Expression for the suggested processor in this case?

If you want all the data and fields, it's : $

(corresponding to the root)

New Contributor

Thank you so much Quentin. I have attached the error from the log that I received after I added 'SplitJSON' processor with $ Xpath value.

2017-08-10 12:28:17,383 ERROR [Timer-Driven Process Thread-9] o.a.nifi.processors.standard.SplitJson SplitJson[id=cc6c6cd5-015d-1000-1de2-6f84dcd57e85] The evaluated value {DataModel={Header={TMS_ID=1093984501, PROCESS_CD=WvWellHeader, BATCH_ID=25679, RECORD_COUNT=1, SOURCE_SYSTEM=WELLVIEW, SOURCE_RECORD_ID=1093984501, TARGET_SYSTEM=EDW, TRANSACTION_TYPE=NEW, SOURCE_CREATE_DATE=2017-03-29T10:18:00-05:00, KEY={ELEMENT_NAME=KEY_PAIRS, ELEMENT_VALUE=WvWellHeader?Idrec=BA6300E67D2A4F63898DDC50290A03F2}, DELETE_FL=N}, Data=H4sIAAAAAAAAA42TTY/bIBCGz+2v4NgeHAP+aCwhJAIki2pDBDhuellF3Uh7Slbb/f/qGCd2emoPljXvPO8wjMfMu4F//sSM8lryjagLjHX9TVFRbuti3ayVkhWmDRa42FKWT1wyDLpt/8uRQLCMb6MEJzijTYbrJiN4qFh+12+IFZ3mQrZIRyRaFAZh0ZNp24Ca0ZdskymR4OqD9vFHJP92zSS43F57EZ3n4u306/WM5OlyejmhNiqWz7kHTjqluXs7v58+zi8LkmTAQhRR7707cLFh+RJBSrreRn/k0wksv8eQUuZggnF2zs0CJLdGtypd8slB56gbfcJY1FsTEWH5AgDdimhiD81U1QrjqqxZPktj2tndFGSEVKtyXRYAzCIQuj3sPByhOC2L1bpk+aKMBbQIUBxTgimGw6d4vEOMXdj3ipOmrjJMsgImPYtTYefN7vsGCkNvVUNT6ZuWepPQqbPxuOfu8vv1+n4ee1vEcb7HELTsvYnHUemt1yF6I6NWz1JYocSz0noPo/8bnKzSa/ggCh4OzdcZphkuEvuQmYY4zgTCvuNQdF2kKS7awgTXe6n5z+vljAix6AuhGA3o4wpRiYavs/EGzqukYVL3/dHqvsC27wq+quqaUlJU06om7WFL0q6JZUmm3WP5+BP/AaZyjR3KAwAA}} of $ was not a JSON Array compatible type and cannot be split.

Have you seen this before? Any clue what could be wrong? One thing that I noticed is: '=' instead of ':' as we are used to seeing in JSON data. But when I export the output from previous processor in a 'PutFile', I get a valid JSON object (shared below:

{ "DataModel" :{ "Header" :{ "TMS_ID" :"1093984501", "PROCESS_CD" :"WvWellHeader", "BATCH_ID" :"25679", "RECORD_COUNT" :"1", "SOURCE_SYSTEM" :"WELLVIEW", "SOURCE_RECORD_ID" :"1093984501", "TARGET_SYSTEM" :"EDW", "TRANSACTION_TYPE" :"NEW", "SOURCE_CREATE_DATE" :"2017-03-29T10:18:00-05:00", "KEY" :{ "ELEMENT_NAME" :"KEY_PAIRS", "ELEMENT_VALUE" :"WvWellHeader?Idrec=BA6300E67D2A4F63898DDC50290A03F2" }, "DELETE_FL" :"N" }, "Data" :"H4sIAAAAAAAAA42TTY/bIBCGz+2v4NgeHAP+aCwhJAIki2pDBDhuellF3Uh7Slbb/f/qGCd2emoPljXvPO8wjMfMu4F//sSM8lryjagLjHX9TVFRbuti3ayVkhWmDRa42FKWT1wyDLpt/8uRQLCMb6MEJzijTYbrJiN4qFh+12+IFZ3mQrZIRyRaFAZh0ZNp24Ca0ZdskymR4OqD9vFHJP92zSS43F57EZ3n4u306/WM5OlyejmhNiqWz7kHTjqluXs7v58+zi8LkmTAQhRR7707cLFh+RJBSrreRn/k0wksv8eQUuZggnF2zs0CJLdGtypd8slB56gbfcJY1FsTEWH5AgDdimhiD81U1QrjqqxZPktj2tndFGSEVKtyXRYAzCIQuj3sPByhOC2L1bpk+aKMBbQIUBxTgimGw6d4vEOMXdj3ipOmrjJMsgImPYtTYefN7vsGCkNvVUNT6ZuWepPQqbPxuOfu8vv1+n4ee1vEcb7HELTsvYnHUemt1yF6I6NWz1JYocSz0noPo/8bnKzSa/ggCh4OzdcZphkuEvuQmYY4zgTCvuNQdF2kKS7awgTXe6n5z+vljAix6AuhGA3o4wpRiYavs/EGzqukYVL3/dHqvsC27wq+quqaUlJU06om7WFL0q6JZUmm3WP5+BP/AaZyjR3KAwAA" }}

Sorry, the splitJson processor is useless in your case. I assumed you have an array of several objects..

Please, remove the splitJson processor and share the full error you have.