question Re: Convert Json to Avro in Nifi in Support Questions

Convert Json to Avro in Nifi

P_Rat98 — Tue, 25 Aug 2020 06:27:09 GMT

Hi,
I want to convert json data to avro.
I have used GenerateFlowFile and put dummy json value [{"firstname":"prathik","age":21},{"firstname":"arun","age":22}].
I have then used ConvertRecord processor and set JsonTreeReader and AvroRecordSetWriter with AvroSchemaRegistry which has the following schema:
{
"name": "person",
"namespace": "nifi",
"type": "record",
"fields": [
{"name": "firstname", "type": "string" },
{ "name": "age", "type": "int" }
]
}

But i am getting this as my output : prathik*arun,

where is a red dot

I am new to Apache Nifi.

Re: Convert Json to Avro in Nifi

SagarKanani — Tue, 25 Aug 2020 07:39:47 GMT

Where are you viewing the data ? Is it in Nifi? Avro is in binary format so its not readable in any editor. If you are viewing in Nifi, then you should check if the you embedded the schema or not. If the schema is embedded then something in the flow is wrong. If not, i suppose it can look the way it shows you.

A quick test could be to dump it in HDFS and create an external hive table to check if the data looks correct.

Hope this helps. If the comment helps you to find a solution or move forward, please accept it as a solution for other community members.

Re: Convert Json to Avro in Nifi

Bender — Tue, 25 Aug 2020 07:45:52 GMT

Hello @P_Rat98 ,

thank you for raising the question about the red dot. Please see this thread that might answer your inquiry.

In short: it is a "non-breaking space character to the file that displays as a red dot".

Please let us know if it helped by pressing the "Accept as Solution" button. Should you need further information, please do not hesitate to reach out to the Community.

Best regards:

Ferenc

Re: Convert Json to Avro in Nifi

P_Rat98 — Tue, 25 Aug 2020 10:41:44 GMT

Hi @SagarKanani Thanks for your input.

I have set the write strategy in AvroRecordSetWriter as Embedded Avro. And i have received the following output :

Objavro.schema€{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}avro.codecnull¢ÈFB¡-Ž`0„Ûv‚L@prathik*arun,¢ÈFB¡-Ž`0„Ûv‚L@

Again is red dot.

One more Doubt, how can one view avro data if we can't view them on nifi.

thanks.

Re: Convert Json to Avro in Nifi

SagarKanani — Tue, 25 Aug 2020 10:46:28 GMT

Hi....you should be able to see the avro in Nifi. When you open the file, there is an option on the top left "View as". It is by default Original. Change it to formatted and you should be able to see the avro correctly. Hope this helps.

Re: Convert Json to Avro in Nifi

P_Rat98 — Tue, 25 Aug 2020 11:23:17 GMT

Thanks @SagarKanani i am now able to view the avro data.

Sorry for the inconvenience, but i had another doubt. If i try to convert this Avro data to Parquet by using ConverAvroToParquet processor then i am getting an output like this:

PAR1&&,6(prathikarunprathikarun,(<Hnifi.person% firstname%%age,&% firstnamenn&<6(prathikarun&v%agejj&v<(Ø,parquet.avro.schema€{"type":"record","name":"person","namespace":"nifi","fields":[{"name":"firstname","type":"string"},{"name":"age","type":"int"}]}writer.model.nameavroJparquet-mr version 1.10.0 (build 031a6654009e3b82020012a18434c582bd74c73a),ÃPAR1

If i try to put the "view as" option as formatted i get the following message: Unable to generate view of data: Not a data file.
Thanks in Advance.

Re: Convert Json to Avro in Nifi

SagarKanani — Tue, 25 Aug 2020 13:33:39 GMT

Unfortunately, i havent used parquet at all...I would assume 'not a data file' could mean either the file doesnt have schema embedded or file is not in correct format (conversion didnt work)

Re: Convert Json to Avro in Nifi

stevenmatison — Fri, 04 Sep 2020 13:25:10 GMT

@P_Rat98 You need parquet tools to read parquet files from command line.

There is no method to view parquet in nifi.

https://pypi.org/project/parquet-tools/