question Re: How can i convert a any file in to Parquet format using NiFi? in Support Questions

How can i convert a any file in to Parquet format using NiFi?

mohamedashraf_e — Sun, 08 Oct 2017 17:39:43 GMT

How can i convert a any file in to Parquet format using NiFi?

i want convert for example CSV File to Parquet file using Apache NiFi

Re: How can i convert a any file in to Parquet format using NiFi?

ahadjidj — Mon, 09 Oct 2017 04:51:59 GMT

Have tried PutParquet : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.4.0/org.apache.nifi.processors.parquet.PutParquet/index.html

It's a record based processor that you can use to read CSV file with CSVReader and write it as parquet

I hope this helps

Re: How can i convert a any file in to Parquet format using NiFi?

mohamedashraf_e — Mon, 09 Oct 2017 13:16:40 GMT

@Abdelkrim Hadjidj

I did it but no convert happens

Re: How can i convert a any file in to Parquet format using NiFi?

mburgess — Tue, 10 Oct 2017 00:58:11 GMT

What do you mean by "no convert happens?" PutParquet should write Parquet file(s) to the HDFS directory you configured in the processor. I believe the incoming flow file is the one transferred to the "success" relationship once the converted file has been successfully written to HDFS, not the converted file. For that I imagine there would have to be a ParquetRecordSetWriter and you'd use ConvertRecord instead of PutParquet.

Re: How can i convert a any file in to Parquet format using NiFi?

ahadjidj — Tue, 10 Oct 2017 13:28:43 GMT

@Mohamed Ashraf

What configuration are you using? what error do you have?

Re: How can i convert a any file in to Parquet format using NiFi?

mohamedashraf_e — Sun, 18 Aug 2019 04:19:12 GMT

@Abdelkrim Hadjidj

I have CSV File , i want convert this file to Parquet.

I did these steps and the operation did not work.

1- GetFile

2- PutParquet

3- PutHDFS

After these steps, the file put into the HDFS but is not converted.

Re: How can i convert a any file in to Parquet format using NiFi?

mohamedashraf_e — Tue, 10 Oct 2017 15:10:50 GMT

I have CSV File , I want convert this file to parquet

this is my steps:-

1- GetFile

2-PutParquet

3-PutHDFS

After these steps, the file put into the HDFS but is not converted.

Re: How can i convert a any file in to Parquet format using NiFi?

ahadjidj — Tue, 10 Oct 2017 21:28:45 GMT

@Mohamed Ashraf I don't have the possibility to test your scenario right now but the PutParquet should write the parquet file directly on HDFS so no need to PutHDFS.

What do you have in /user/nifi ? what directory have you configured with PutHDFS ?

Re: How can i convert a any file in to Parquet format using NiFi?

ahadjidj — Wed, 11 Oct 2017 04:14:45 GMT

@Mohamed Ashraf

Following your question, I wrote an article on how to use PutParquet to convert data. Check it out to have a better understanding on the process.

https://community.hortonworks.com/articles/140422/convert-data-from-jsoncsvavro-to-parquet-with-nifi.html

I hope this helps