Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Convert Json to Avro in Nifi

avatar
Explorer

Hi,
I want to convert json data to avro.
I have used GenerateFlowFile and put dummy json value [{"firstname":"prathik","age":21},{"firstname":"arun","age":22}].
I have then used ConvertRecord processor and set JsonTreeReader and AvroRecordSetWriter with AvroSchemaRegistry which has the following schema:
{
"name": "person",
"namespace": "nifi",
"type": "record",
"fields": [
{"name": "firstname", "type": "string" },
{ "name": "age", "type": "int" }
]
}

But i am getting this as my output : prathik*arun,

where  is a red dot


I am new to Apache Nifi.

1 ACCEPTED SOLUTION

avatar
Contributor

Where are you viewing the data ? Is it in Nifi? Avro is in binary format so its not readable in any editor. If you are viewing in Nifi, then you should check if the you embedded the schema or not. If the schema is embedded then something in the flow is wrong. If not, i suppose it can look the way it shows you. 

A quick test could be to dump it in HDFS and create an external hive table to check if the data looks correct.

Hope this helps. If the comment helps you to find a solution or move forward, please accept it as a solution for other community members.

View solution in original post

7 REPLIES 7

avatar
Contributor

Where are you viewing the data ? Is it in Nifi? Avro is in binary format so its not readable in any editor. If you are viewing in Nifi, then you should check if the you embedded the schema or not. If the schema is embedded then something in the flow is wrong. If not, i suppose it can look the way it shows you. 

A quick test could be to dump it in HDFS and create an external hive table to check if the data looks correct.

Hope this helps. If the comment helps you to find a solution or move forward, please accept it as a solution for other community members.

avatar
Explorer

Hi @SagarKanani Thanks for your input.

I have set the write strategy in AvroRecordSetWriter as Embedded Avro. And i have received the following output  :

Objavro.schema€{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}avro.codecnull¢ÈFB¡-Ž`0„Ûv‚L@prathik*arun,¢ÈFB¡-Ž`0„Ûv‚L@

 

Again  is red dot.

One more Doubt, how can one view avro data if we can't view them on nifi.

thanks.

avatar
Contributor

Hi....you should be able to see the avro in Nifi. When you open the file, there is an option on the top left "View as". It is by default Original. Change it to formatted and you should be able to see the avro correctly. Hope this helps.

avatar
Explorer

Thanks @SagarKanani i am now able to view the avro data.

Sorry for the inconvenience, but i had another doubt. If i try to convert this Avro data to Parquet by using ConverAvroToParquet processor then i am getting an output like this:

PAR1&&,6(prathikarunprathikarun,(<H nifi.person % firstname%%age,& % firstnamenn&<6(prathikarun&v%agejj&v<(Ø,parquet.avro.schema€{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}writer.model.nameavroJparquet-mr version 1.10.0 (build 031a6654009e3b82020012a18434c582bd74c73a),ÃPAR1

 

If i try to put the "view as" option as formatted i get the following message: Unable to generate view of data: Not a data file.
Thanks in Advance. 

avatar
Contributor

Unfortunately, i havent used parquet at all...I would assume 'not a data file' could mean either the file doesnt have schema embedded or file is not in correct format (conversion didnt work)

avatar
Super Guru

@P_Rat98  You need parquet tools to read parquet files from command line.

There is no method to view parquet in nifi.

 

https://pypi.org/project/parquet-tools/

 

 

avatar
Moderator

Hello @P_Rat98 ,

 

thank you for raising the question about the red dot. Please see this thread that might answer your inquiry.

 

In short: it is a "non-breaking space character to the file that displays as a red dot".

 

Please let us know if it helped by pressing the "Accept as Solution" button. Should you need further information, please do not hesitate to reach out to the Community.

 

Best regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community: