Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Split Large Json file into multiple files with a specified number of records

avatar
Explorer

Hi, 

I am a newbie to Nifi and would like some guidance please. 

We want to split a large Json file into multiple files with a specified number of records. I am able to split a file into individual records using SplitJson and the Json Path Expression set as $.<objectthatcontainsthearray>.* I have also added an UpdateAttribute Processor with filename set to ${filename}_${fragment.index} so that we have the sequence of the files as order is important.

However, we might want to have say a 100,000 records split into 100 files of 1000 records each . What is the easiest way to do this ?

Thanks very much in advance

 

1 REPLY 1

avatar
Super Mentor

@pxm 

 

You'll want to use the SplitRecord processor to accomplish what you are looking to do.

 

Configure it with a RecordReader and RecordWriter  for you input data and desired output data formats along with the desired 1000 records per split.

 

Hope this helps,

Matt