Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how could I convert avro format (result of executesq) to parquet using putparquet ?

Solved Go to solution
Highlighted

how could I convert avro format (result of executesq) to parquet using putparquet ?

Explorer

Hi,

I am going to convert the result of Nifi ExecuteSQL procecessor and save the result as putparquet into hdfs.

Any suggestion on how to convert avro format (result of executesql) to parquet using putparquet ?

I will use avroreader in putparquet processor, but it requires Schema Registry. could I auto generate the Schema Registry of the result from executesql.?

Thanks

Andy

1 ACCEPTED SOLUTION

Accepted Solutions

Re: how could I convert avro format (result of executesq) to parquet using putparquet ?

Super Guru
@Andy Liang

ExecuteSql processor will result output flowfile with embedded avro schema in it.

So in your PutParquet processor configure the Record Reader as Avro Reader and use the embedded avro schema as Schema access strategy

80581-avro-reader.png

By using Embedded avro schema you don't have to set up any avro schema registry and this processor will work dynamically based on the embedded avro schema.

1 REPLY 1

Re: how could I convert avro format (result of executesq) to parquet using putparquet ?

Super Guru
@Andy Liang

ExecuteSql processor will result output flowfile with embedded avro schema in it.

So in your PutParquet processor configure the Record Reader as Avro Reader and use the embedded avro schema as Schema access strategy

80581-avro-reader.png

By using Embedded avro schema you don't have to set up any avro schema registry and this processor will work dynamically based on the embedded avro schema.

Don't have an account?
Coming from Hortonworks? Activate your account here