Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Convert Json to Avro processor -- Failed to Parse Record Schema

Solved Go to solution
Highlighted

Convert Json to Avro processor -- Failed to Parse Record Schema

New Contributor

Hi Friends,

Needed a help here.

I have picked up one of the xmls lying in the trafficLocs_data_for_simulator.zip.

My objective is to convert XML to Avro format. The tutorials has shown us how to convert XML to JSON format by using AttributesToJSON processor.

I am facing issue while configuring ConvertJSONToAvro processor.

I have used the Record Schema as mentioned below -

{ "type": "record", "name" : "first", "namespace" : "xml", "fields" : [{"name" : "Direction_of_Travel", "type" : "string" }, { "name" : "Latitude", "type" : "float" }, { "name" : "Longitude", "type" : "float" }, { "name" : "Vehicle_ID", "type" : "integer" }, { "name" : "Vehicle_Speed"," type" : "integer" }] }

I am getting red alert symbol mentioning it as Failed to Parse Record Schema.

Any pointers will help.

Thanks,

Ankur K

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

Super Guru

@Ankur Kapoor

Also attaching the template of my workflow.hcc.xml

11 REPLIES 11

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

Super Guru
@Ankur Kapoor

In the last "type" after "Vehicle Speed", there is a space. Please remove that space and it will work.

 { "name" : "Vehicle_Speed"," type" : "integer" } --> wrong

 { "name" : "Vehicle_Speed", "type" : "integer" } --> right

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

New Contributor

Hi @mqureshi,

Many thanks for the reply. I have removed space but it is still showing the same issue Failed to Parse Record Schema.

The Record schema is mentioned below -

{ "type" : "record", "name" : "first", "namespace" : "xml", "fields" : [{"name" : "Direction_of_Travel", "type" : "string" }, { "name" : "Latitude", "type" : "float" }, { "name" : "Longitude", "type" : "float" }, { "name" : "Vehicle_ID", "type" : "integer" }, { "name" : "Vehicle_Speed","type" : "integer" }] }

Please advise.

Thanks,

Ankur

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

Super Guru

@Ankur Kapoor

I used your record and it works. Do you have a inferavroschema before calling convertJsontoAvro? Look at my screen shot. My second screen shot shows details of convertjsontoavro processor details. Third screenshot shows details of InferAvroSchema details.screen-shot-2017-02-11-at-13434-pm.png

screen-shot-2017-02-11-at-13242-pm.pngscreen-shot-2017-02-11-at-13302-pm.png

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

New Contributor

Hi @mqureshi,

Many thanks for your reply.

I had actually tried this option of using infer avro schema before but did not work. I wanted to check whether you have used XML file as a source or CSV file. My requirement is to convert the XML file into Avro format.

If you see tutorials on Nifi , I had picked up one xml file lying in the zip - trafficLocs_data_for_simulator.zip.

I am attaching some snapshots also warning message which is coming. Please have a look.

1-entire-flow.gif2-getfile.gif4-evaluatexpath.gif7-inferavroschema.gifwarning-message.png

Thanks,

Ankur

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

New Contributor

@ankurkapoor_wor Hi, Even I am facing the same issue as @mqureshi. I am trying to fetch data from SQL server in Avro fomat through NiFi and load it to Redshift through copy command. But the generated Avro file is converting the date and timestamp datatypes to string because of which copy command is loading all NULL values in the target table. So I tried to follow your approach, In my case I'm using ExecuteSQLRecord processor to fetch the data from SQL server and writing it to json format and then trying to convert it to Avro format using ConvertJsonToAvro processor but then I am unable to parse Record Schema. Could you please help me also to resolve this issue. 

Thanks in advance!

Anusha

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

Super Guru

@Ankur Kapoor

I just did the same thing and it still works. Where do you write your output of "AttributesToJson"? I am creating a new flow file because this gives me a clean json record.

entireflow.pngevaluate-xpath.pngattributestojson.pnginferavroschema.pngconvertjsontoavro.png

I can see the file being created on my machine.

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

Super Guru

@Ankur Kapoor

Also attaching the template of my workflow.hcc.xml

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

New Contributor

Hi @mqureshi ,

Many thanks brother. I am pleased to say it is working now.

You have really been helpful.

Now , I want to solve some bigger questions , the xml file lying in the zip - trafficLocs_data_for_simulator.zip was a small xml.

Hence we are able to define its structure in EvaluteXPath , suppose we are having bigger xmlxmls.zip

How we process these big xml's which really exist in the real life. Please advise.

Attached some xmls for your reference.

Many Thanks,

Ankur

Re: Convert Json to Avro processor -- Failed to Parse Record Schema

New Contributor

Hi @mqureshi,

Can you please tell me how to conver JSON to csv in nifi..?

Don't have an account?
Coming from Hortonworks? Activate your account here