Convert Json to Avro processor -- Failed to Parse Record Schema

[#1]

Hi Friends,

Needed a help here.

I have picked up one of the xmls lying in the trafficLocs_data_for_simulator.zip.

My objective is to convert XML to Avro format. The tutorials has shown us how to convert XML to JSON format by using AttributesToJSON processor.

I am facing issue while configuring ConvertJSONToAvro processor.

I have used the Record Schema as mentioned below -

{ "type": "record", "name" : "first", "namespace" : "xml", "fields" : [{"name" : "Direction_of_Travel", "type" : "string" }, { "name" : "Latitude", "type" : "float" }, { "name" : "Longitude", "type" : "float" }, { "name" : "Vehicle_ID", "type" : "integer" }, { "name" : "Vehicle_Speed"," type" : "integer" }] }

I am getting red alert symbol mentioning it as Failed to Parse Record Schema.

Any pointers will help.

Thanks,

Ankur K

[#2]

@Ankur Kapoor

In the last "type" after "Vehicle Speed", there is a space. Please remove that space and it will work.

 { "name" : "Vehicle_Speed"," type" : "integer" } --> wrong

 { "name" : "Vehicle_Speed", "type" : "integer" } --> right

[#3]

Hi @mqureshi,

Many thanks for the reply. I have removed space but it is still showing the same issue Failed to Parse Record Schema.

The Record schema is mentioned below -

{ "type" : "record", "name" : "first", "namespace" : "xml", "fields" : [{"name" : "Direction_of_Travel", "type" : "string" }, { "name" : "Latitude", "type" : "float" }, { "name" : "Longitude", "type" : "float" }, { "name" : "Vehicle_ID", "type" : "integer" }, { "name" : "Vehicle_Speed","type" : "integer" }] }

Please advise.

Thanks,

Ankur

[#4]

@Ankur Kapoor

I used your record and it works. Do you have a inferavroschema before calling convertJsontoAvro? Look at my screen shot. My second screen shot shows details of convertjsontoavro processor details. Third screenshot shows details of InferAvroSchema details.screen-shot-2017-02-11-at-13434-pm.png

screen-shot-2017-02-11-at-13242-pm.pngscreen-shot-2017-02-11-at-13302-pm.png

[#5]

Hi @mqureshi,

Many thanks for your reply.

I had actually tried this option of using infer avro schema before but did not work. I wanted to check whether you have used XML file as a source or CSV file. My requirement is to convert the XML file into Avro format.

If you see tutorials on Nifi , I had picked up one xml file lying in the zip - trafficLocs_data_for_simulator.zip.

I am attaching some snapshots also warning message which is coming. Please have a look.

1-entire-flow.gif2-getfile.gif4-evaluatexpath.gif7-inferavroschema.gifwarning-message.png

Thanks,

Ankur

[#6]

@ankurkapoor_wor Hi, Even I am facing the same issue as @mqureshi. I am trying to fetch data from SQL server in Avro fomat through NiFi and load it to Redshift through copy command. But the generated Avro file is converting the date and timestamp datatypes to string because of which copy command is loading all NULL values in the target table. So I tried to follow your approach, In my case I'm using ExecuteSQLRecord processor to fetch the data from SQL server and writing it to json format and then trying to convert it to Avro format using ConvertJsonToAvro processor but then I am unable to parse Record Schema. Could you please help me also to resolve this issue. 

Thanks in advance!

Anusha

[#7]

@Ankur Kapoor

I just did the same thing and it still works. Where do you write your output of "AttributesToJson"? I am creating a new flow file because this gives me a clean json record.

entireflow.pngevaluate-xpath.pngattributestojson.pnginferavroschema.pngconvertjsontoavro.png

I can see the file being created on my machine.

[#8 · Accepted solution]

@Ankur Kapoor

Also attaching the template of my workflow.hcc.xml

[#9]

Hi @mqureshi ,

Many thanks brother. I am pleased to say it is working now.

You have really been helpful.

Now , I want to solve some bigger questions , the xml file lying in the zip - trafficLocs_data_for_simulator.zip was a small xml.

Hence we are able to define its structure in EvaluteXPath , suppose we are having bigger xmlxmls.zip

How we process these big xml's which really exist in the real life. Please advise.

Attached some xmls for your reference.

Many Thanks,

Ankur

[#10]

Hi @mqureshi,

Can you please tell me how to conver JSON to csv in nifi..?