Skip to main content
derek_calderon
derek_calderon
Bronze ⭐️⭐️⭐️
April 2, 2019
Solved

Can I store more than one record in a parquet file using ConvertAvroToParquet?

I have created a flow to read json messages from kafka, and covert them to avro so that I can use ConvertAvroToParquet. My issue is that I can only store one avro message in a parquet file. Is there away to store multiple records within a parquet file using ConvertAvroToParquet?

mburgess
mburgessAnswer
Gold ⭐️⭐️⭐️
April 2, 2019

You can use MergeContent or MergeRecord for this, it can take flow files each with a single record and combine them together to make a flow file containing many Avro records, then you can use ConvertAvroToParquet or PutParquet.

View in thread

1 reply

mburgess
Gold ⭐️⭐️⭐️
mburgessAnswer
Gold ⭐️⭐️⭐️
April 2, 2019

You can use MergeContent or MergeRecord for this, it can take flow files each with a single record and combine them together to make a flow file containing many Avro records, then you can use ConvertAvroToParquet or PutParquet.

Translate
    Terms & ConditionsAccessibility statement