Skip to main content
medloh
medloh
Silver ⭐️⭐️⭐️
February 5, 2021
Solved

Nifi PutParquet processor, how to enter json schema info?

I'm trying to convert local json file with getfile processor into parquet file on HDFS.  I'm following this guide:

 

https://medium.com/@abdelkrim.hadjidj/using-apache-nifi-for-json-to-parquet-conversion-945219d5caba

 

I choose Record Reader of JsonTreeReader, but I don't see any of the schema properties in the putparquet processor? (Schema Access Strategy, Schema Text).  How do I get those to show up, or how do I enter them manually?

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
February 5, 2021

@medloh 

The article you are using for reference is old and bit out of date.
As part of the  work that went into NIFI-3921 the schema properties within the putParquet processor were removed.  Before these changes were made at the time of that article you referenced, you had to set the schema properties and they had to match the schema properties set in the recordReader.   With the changes the processor simply gets them from the reader so they do not need to be configured a second time in the processor properties.

Also at time of that article there was no parquetReader or ParquetRecordSetWriter controller services.  Now that NiFi has a ParquetReader and writer, you can use the ConvertRecord processor to read a source FlowFiles and convert it parquet within your dataflow and then have freedom to use whatever processor you want downstream in your dataflow to write out the parquet content.  You can think of the putParquet as a combination of ParquetRecordSetWriter and putHDFS with a selectable recordReader only.

Hope this helps,

Matt

View in thread

5 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
February 5, 2021

@medloh 

The article you are using for reference is old and bit out of date.
As part of the  work that went into NIFI-3921 the schema properties within the putParquet processor were removed.  Before these changes were made at the time of that article you referenced, you had to set the schema properties and they had to match the schema properties set in the recordReader.   With the changes the processor simply gets them from the reader so they do not need to be configured a second time in the processor properties.

Also at time of that article there was no parquetReader or ParquetRecordSetWriter controller services.  Now that NiFi has a ParquetReader and writer, you can use the ConvertRecord processor to read a source FlowFiles and convert it parquet within your dataflow and then have freedom to use whatever processor you want downstream in your dataflow to write out the parquet content.  You can think of the putParquet as a combination of ParquetRecordSetWriter and putHDFS with a selectable recordReader only.

Hope this helps,

Matt

Translate
    medloh
    medlohAuthor
    Silver ⭐️⭐️⭐️
    February 5, 2021

    Thanks Matt,

     

    The author of that article recommended this, I'll give it a try.

     

    " Also, NiFi has now a Parquet Record that you can use outside of the PutParquet. I advise you to use ConvertRecord then PutHDFS directly. This is better."

     

    I'm a little unsure of how and where I need to configure the schemas but hopefully I'll figure it out.

     

    Translate
    MattWho
    Titanium ⭐️⭐️⭐️
    MattWho
    Titanium ⭐️⭐️⭐️
    February 8, 2021

    @medloh 

    The Schema only needs to be defined in the RecordReader configured in the PutParquet processor.

    In the case of the ConvertRecord processor there exists both a Record reader and a Record Writer.  You can have the RecordReader get the Schema from the Record Writer or define its own Schema.

    Hope this helps,

    Matt

    Translate
    Terms & ConditionsAccessibility statement