Skip to main content
P_Rat98
P_Rat98
Silver ⭐️⭐️
August 25, 2020
Solved

Convert Json to Avro in Nifi

Hi,
I want to convert json data to avro.
I have used GenerateFlowFile and put dummy json value [{"firstname":"prathik","age":21},{"firstname":"arun","age":22}].
I have then used ConvertRecord processor and set JsonTreeReader and AvroRecordSetWriter with AvroSchemaRegistry which has the following schema:
{
"name": "person",
"namespace": "nifi",
"type": "record",
"fields": [
{"name": "firstname", "type": "string" },
{ "name": "age", "type": "int" }
]
}

But i am getting this as my output : prathik*arun,

where  is a red dot


I am new to Apache Nifi.

SagarKanani
SagarKananiAnswer
Gold ⭐️
August 25, 2020

Where are you viewing the data ? Is it in Nifi? Avro is in binary format so its not readable in any editor. If you are viewing in Nifi, then you should check if the you embedded the schema or not. If the schema is embedded then something in the flow is wrong. If not, i suppose it can look the way it shows you. 

A quick test could be to dump it in HDFS and create an external hive table to check if the data looks correct.

Hope this helps. If the comment helps you to find a solution or move forward, please accept it as a solution for other community members.

View in thread

7 replies

SagarKanani
SagarKananiAnswer
Gold ⭐️
August 25, 2020

Where are you viewing the data ? Is it in Nifi? Avro is in binary format so its not readable in any editor. If you are viewing in Nifi, then you should check if the you embedded the schema or not. If the schema is embedded then something in the flow is wrong. If not, i suppose it can look the way it shows you. 

A quick test could be to dump it in HDFS and create an external hive table to check if the data looks correct.

Hope this helps. If the comment helps you to find a solution or move forward, please accept it as a solution for other community members.

Translate
P_Rat98
P_Rat98Author
Silver ⭐️⭐️
August 25, 2020

Hi @SagarKanani Thanks for your input.

I have set the write strategy in AvroRecordSetWriter as Embedded Avro. And i have received the following output  :

Objavro.schema€{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}avro.codecnull¢ÈFB¡-Ž`0„Ûv‚L@prathik*arun,¢ÈFB¡-Ž`0„Ûv‚L@

 

Again  is red dot.

One more Doubt, how can one view avro data if we can't view them on nifi.

thanks.

Translate
SagarKanani
SagarKanani
Gold ⭐️
August 25, 2020

Hi....you should be able to see the avro in Nifi. When you open the file, there is an option on the top left "View as". It is by default Original. Change it to formatted and you should be able to see the avro correctly. Hope this helps.

Translate
Bender
Gold ⭐️⭐️⭐️
Bender
Gold ⭐️⭐️⭐️
August 25, 2020

Hello @P_Rat98 ,

 

thank you for raising the question about the red dot. Please see this thread that might answer your inquiry.

 

In short: it is a "non-breaking space character to the file that displays as a red dot".

 

Please let us know if it helped by pressing the "Accept as Solution" button. Should you need further information, please do not hesitate to reach out to the Community.

 

Best regards:

Ferenc

Translate
Ferenc Erdelyi, Technical Solutions Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. Learn more about the Cloudera Community: - Community Guidelines (https://community.cloudera.com/t5/custom/page/page-id/Community_Guidelines) - How to use the forum (/t5/help/faqpage)
    Terms & ConditionsAccessibility statement