question Re: Apache nifi - how to convert a file .txt into Parquet (to save into HDFS) with Nifi? in Support Questions

Apache nifi - how to convert a file .txt into Parquet (to save into HDFS) with Nifi?

Lallagreta — Fri, 08 Jan 2021 17:08:35 GMT

Hi, i can't compress so many files.txt into a Parquet format to save in HDFS.

How can i do that? @ApacheNifi

Re: Apache nifi - how to convert a file .txt into Parquet (to save into HDFS) with Nifi?

stevenmatison — Fri, 08 Jan 2021 17:55:52 GMT

@Lallagreta The solution you are looking for is to leverage NiFi Parquet Processors w/ Parquet Record Reader/Writer

Some fun links:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-parquet-nar/1.11.4/org.apache.nifi.parquet.ParquetRecordSetWriter/index.html

https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-1-10-Support-for-Parquet-RecordReader/ta-p/282390

The Parquet procs are part of Nifi1.10 and up, but you can also install the nars into any older nifi versions:

https://community.cloudera.com/t5/Support-Questions/Can-I-put-the-NiFi-1-10-Parquet-Record-Reader-in-NiFi-1-9/m-p/286465

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven

Re: Apache nifi - how to convert a file .txt into Parquet (to save into HDFS) with Nifi?

Lallagreta — Sat, 09 Jan 2021 10:03:33 GMT

Hi, thank you so much for your answer.

I understand that i had to treat the data as CSV format with a “tab” delimiter rather than a “,”.

For my project i use this flow:

GetFile -> UpdateAttributo -> PutParquet but something go wrong.

The error that compare is: "Unable to create record reader".

This is my processor configuration:

THANK YOU @ApacheNifi

Re: Apache nifi - how to convert a file .txt into Parquet (to save into HDFS) with Nifi?

stevenmatison — Mon, 11 Jan 2021 13:55:59 GMT

You must have the reader incorrectly configured for your CSV schema.