Support Questions
Find answers, ask questions, and share your expertise

Convert Fixed length text file into an Avro file using Nifi

New Contributor


I have a requirement where i need to convert a nested fixed length text file into a nested avro file using Nifi. My source file will be having data like below

01 - Header

02 - Account Details -1

03 - Transaction Details 1

03 - Transaction Details 2

04 - Transaction Count 1

02 - Account Details -2

03 - Transaction Details 1

04 - Transaction Count 2

05 - Trailer

In the above file we need to ignore Header and Trailer . We will need to read each line and determine the type of data based on the first 2 characters in this case '02' and nest all the lines after this line with the same block until we encounter another line with '02'. Record '02' will be the mail block and '03' will be a nested array within '02' and '04' will be nested record in '02' as well . The output will be an avro file.

The data volume is quite big so would like to build a solution with performance also in mind.

We are fine with the approach of building this with a custom processor as well but would like to do this with the existing resources to have the option of re-usability for other similar requirements.