Convert Json to Avro in Nifi

[#1]

Hi,
I want to convert json data to avro.
I have used GenerateFlowFile and put dummy json value [{"firstname":"prathik","age":21},{"firstname":"arun","age":22}].
I have then used ConvertRecord processor and set JsonTreeReader and AvroRecordSetWriter with AvroSchemaRegistry which has the following schema:
{
"name": "person",
"namespace": "nifi",
"type": "record",
"fields": [
{"name": "firstname", "type": "string" },
{ "name": "age", "type": "int" }
]
}

But i am getting this as my output : prathik*arun,

where  is a red dot

I am new to Apache Nifi.

[#2 · Accepted solution]

Where are you viewing the data ? Is it in Nifi? Avro is in binary format so its not readable in any editor. If you are viewing in Nifi, then you should check if the you embedded the schema or not. If the schema is embedded then something in the flow is wrong. If not, i suppose it can look the way it shows you. 

A quick test could be to dump it in HDFS and create an external hive table to check if the data looks correct.

Hope this helps. If the comment helps you to find a solution or move forward, please accept it as a solution for other community members.

[#3]

Hi @SagarKanani Thanks for your input.

I have set the write strategy in AvroRecordSetWriter as Embedded Avro. And i have received the following output  :

Obj
avro.schema€{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}avro.codecnull¢ÈFB¡-Ž`0„Ûv‚L@prathik*arun,¢ÈFB¡-Ž`0„Ûv‚L@

Again  is red dot.

One more Doubt, how can one view avro data if we can't view them on nifi.

thanks.

[#4]

Hi....you should be able to see the avro in Nifi. When you open the file, there is an option on the top left "View as". It is by default Original. Change it to formatted and you should be able to see the avro correctly. Hope this helps.

[#5]

Thanks @SagarKanani i am now able to view the avro data.

Sorry for the inconvenience, but i had another doubt. If i try to convert this Avro data to Parquet by using ConverAvroToParquet processor then i am getting an output like this:

PAR1&&,6(prathikarunprathikarun,(<Hnifi.person% firstname%%age,&% firstnamenn&<6(prathikarun&v%agejj&v<(Ø
,parquet.avro.schema€
{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}writer.model.nameavroJparquet-mr version 1.10.0 (build 031a6654009e3b82020012a18434c582bd74c73a),Ã
PAR1

If i try to put the "view as" option as formatted i get the following message: Unable to generate view of data: Not a data file.
Thanks in Advance. 

[#6]

Unfortunately, i havent used parquet at all...I would assume 'not a data file' could mean either the file doesnt have schema embedded or file is not in correct format (conversion didnt work)

[#7]

@P_Rat98  You need parquet tools to read parquet files from command line.

There is no method to view parquet in nifi.

https://pypi.org/project/parquet-tools/

[#8]

Hello @P_Rat98 ,

thank you for raising the question about the red dot. Please see this thread that might answer your inquiry.

In short: it is a "non-breaking space character to the file that displays as a red dot".

Please let us know if it helped by pressing the "Accept as Solution" button. Should you need further information, please do not hesitate to reach out to the Community.

Best regards:

Ferenc