How to split large json file into multiple json files in Nifi?

Question

We have a large json file which is more than 100GB and we want to split this json file into multiple files. We used Split Text processor to split this json file into mutliple files by specifying Line Split Count. Is there any way we can pass attribute/variable in Line Split Count and then split the records based on the attribute/variable as currently Line Split Count does not support attributes/variables.

Kindly suggest if there is another approach to split these json files based on attribute/variables

Sample Json File

{"name": "John","lastName": "Wick","phoneNumber": "123123123"} {"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"} {"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}

DigitalPlumber · Accepted Answer

Yes SplitRecord is what you should use.Attached is a flow definition as an example.Note that I had to rename the file with a "txt" extension once you download it rename it to a .json extensionYou can then drag a processor group and it gives you an option to upload the flow definition. That example generates a file with 102 records and on SlitRecord we use a JsontTreeReader that will split by 3 records and writes the flowfiles out, In this case per 3 per flowFile generating 34 FlowFiles.1-2 / 3 = 34 In your case and based on your screenshot I would change split count to be 1500000 ( or another number based on your needs )

SAMSAL · Answer

Hi,Try to look into QueryRecord or PartitionRecord Processors. Those might help.Thanks

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.