question Convert Json to Avro processor -- Failed to Parse Record Schema in Support Questions

Convert Json to Avro processor -- Failed to Parse Record Schema

ankurkapoor_wor — Sat, 11 Feb 2017 21:18:12 GMT

Hi Friends,

Needed a help here.

I have picked up one of the xmls lying in the trafficLocs_data_for_simulator.zip.

My objective is to convert XML to Avro format. The tutorials has shown us how to convert XML to JSON format by using AttributesToJSON processor.

I am facing issue while configuring ConvertJSONToAvro processor.

I have used the Record Schema as mentioned below -

{ "type": "record", "name" : "first", "namespace" : "xml", "fields" : [{"name" : "Direction_of_Travel", "type" : "string" }, { "name" : "Latitude", "type" : "float" }, { "name" : "Longitude", "type" : "float" }, { "name" : "Vehicle_ID", "type" : "integer" }, { "name" : "Vehicle_Speed"," type" : "integer" }] }

I am getting red alert symbol mentioning it as Failed to Parse Record Schema.

Any pointers will help.

Thanks,

Ankur K

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

mqureshi — Sat, 11 Feb 2017 23:36:52 GMT

@Ankur Kapoor

In the last "type" after "Vehicle Speed", there is a space. Please remove that space and it will work.

 { "name" : "Vehicle_Speed"," type" : "integer" } --> wrong

 { "name" : "Vehicle_Speed", "type" : "integer" } --> right

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

ankurkapoor_wor — Sun, 12 Feb 2017 01:47:08 GMT

Hi @mqureshi,

Many thanks for the reply. I have removed space but it is still showing the same issue Failed to Parse Record Schema.

The Record schema is mentioned below -

{ "type" : "record", "name" : "first", "namespace" : "xml", "fields" : [{"name" : "Direction_of_Travel", "type" : "string" }, { "name" : "Latitude", "type" : "float" }, { "name" : "Longitude", "type" : "float" }, { "name" : "Vehicle_ID", "type" : "integer" }, { "name" : "Vehicle_Speed","type" : "integer" }] }

Please advise.

Thanks,

Ankur

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

mqureshi — Sun, 12 Feb 2017 03:35:27 GMT

@Ankur Kapoor

I used your record and it works. Do you have a inferavroschema before calling convertJsontoAvro? Look at my screen shot. My second screen shot shows details of convertjsontoavro processor details. Third screenshot shows details of InferAvroSchema details.screen-shot-2017-02-11-at-13434-pm.png

screen-shot-2017-02-11-at-13242-pm.png screen-shot-2017-02-11-at-13302-pm.png

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

ankurkapoor_wor — Sun, 12 Feb 2017 07:52:47 GMT

Hi @mqureshi,

Many thanks for your reply.

I had actually tried this option of using infer avro schema before but did not work. I wanted to check whether you have used XML file as a source or CSV file. My requirement is to convert the XML file into Avro format.

If you see tutorials on Nifi , I had picked up one xml file lying in the zip - trafficLocs_data_for_simulator.zip.

I am attaching some snapshots also warning message which is coming. Please have a look.

1-entire-flow.gif 2-getfile.gif 4-evaluatexpath.gif 7-inferavroschema.gif warning-message.png

Thanks,

Ankur

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

mqureshi — Sun, 12 Feb 2017 12:47:39 GMT

@Ankur Kapoor

I just did the same thing and it still works. Where do you write your output of "AttributesToJson"? I am creating a new flow file because this gives me a clean json record.

entireflow.png evaluate-xpath.png attributestojson.png inferavroschema.png convertjsontoavro.png

I can see the file being created on my machine.

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

mqureshi — Sun, 12 Feb 2017 12:48:57 GMT

@Ankur Kapoor

Also attaching the template of my workflow.hcc.xml

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

ankurkapoor_wor — Sun, 12 Feb 2017 17:25:19 GMT

Hi @mqureshi ,

Many thanks brother. I am pleased to say it is working now.

You have really been helpful.

Now , I want to solve some bigger questions , the xml file lying in the zip - trafficLocs_data_for_simulator.zip was a small xml.

Hence we are able to define its structure in EvaluteXPath , suppose we are having bigger xmlxmls.zip

How we process these big xml's which really exist in the real life. Please advise.

Attached some xmls for your reference.

Many Thanks,

Ankur

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

rajatpuchnanda1 — Tue, 26 Jun 2018 01:45:39 GMT

Hi @mqureshi,

Can you please tell me how to conver JSON to csv in nifi..?

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

Former Member — Fri, 29 Jun 2018 23:57:34 GMT

@jmedel

Any thoughts?

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

jmedel — Sat, 30 Jun 2018 03:46:40 GMT

Hey @rajat puchnanda

First I would recommend extracting the attributes from JSON using EvaluateJSONPath processor, then utilize AttributesToCSV processor to represent the FlowFile attributes as CSV format.

For seeing how to use evaluateJSONPath processor, check out this part of the tutorial series:

https://hortonworks.com/tutorial/analyze-transit-patterns-with-apache-nifi/section/5/

AttributesToCSV processor is similar to AttributesToJSON processor, the following tutorial shows how to use AttributesToJSON, apply the similar idea to using AttributesToCSV processor:

https://hortonworks.com/tutorial/analyze-transit-patterns-with-apache-nifi/section/6/

For info on AttributesToCSV processor:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.AttributesToCSV/index.html

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

bhide — Wed, 21 Aug 2019 17:39:17 GMT

@ankurkapoor_wor Hi, Even I am facing the same issue as @mqureshi. I am trying to fetch data from SQL server in Avro fomat through NiFi and load it to Redshift through copy command. But the generated Avro file is converting the date and timestamp datatypes to string because of which copy command is loading all NULL values in the target table. So I tried to follow your approach, In my case I'm using ExecuteSQLRecord processor to fetch the data from SQL server and writing it to json format and then trying to convert it to Avro format using ConvertJsonToAvro processor but then I am unable to parse Record Schema. Could you please help me also to resolve this issue.

Thanks in advance!

Anusha