Created 10-08-2017 10:39 AM
How can i convert a any file in to Parquet format using NiFi?
i want convert for example CSV File to Parquet file using Apache NiFi
Created 10-10-2017 09:14 PM
Following your question, I wrote an article on how to use PutParquet to convert data. Check it out to have a better understanding on the process.
I hope this helps
Created 10-08-2017 09:51 PM
Have tried PutParquet : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.4.0/org.apache....
It's a record based processor that you can use to read CSV file with CSVReader and write it as parquet
I hope this helps
Created 10-09-2017 06:16 AM
I did it but no convert happens
Created 10-09-2017 05:58 PM
What do you mean by "no convert happens?" PutParquet should write Parquet file(s) to the HDFS directory you configured in the processor. I believe the incoming flow file is the one transferred to the "success" relationship once the converted file has been successfully written to HDFS, not the converted file. For that I imagine there would have to be a ParquetRecordSetWriter and you'd use ConvertRecord instead of PutParquet.
Created 10-10-2017 08:10 AM
I have CSV File , I want convert this file to parquet
this is my steps:-
1- GetFile
2-PutParquet
3-PutHDFS
After these steps, the file put into the HDFS but is not converted.
Created 10-10-2017 06:28 AM
What configuration are you using? what error do you have?
Created on 10-10-2017 07:48 AM - edited 08-17-2019 09:19 PM
I have CSV File , i want convert this file to Parquet.
I did these steps and the operation did not work.
1- GetFile
2- PutParquet
3- PutHDFS
After these steps, the file put into the HDFS but is not converted.
Created 10-10-2017 02:28 PM
@Mohamed Ashraf I don't have the possibility to test your scenario right now but the PutParquet should write the parquet file directly on HDFS so no need to PutHDFS.
What do you have in /user/nifi ? what directory have you configured with PutHDFS ?
Created 10-10-2017 09:14 PM
Following your question, I wrote an article on how to use PutParquet to convert data. Check it out to have a better understanding on the process.
I hope this helps