question How to split large json file into multiple json files in Nifi? in Support Questions

How to split large json file into multiple json files in Nifi?

rahul_loke — Fri, 17 Feb 2023 17:47:56 GMT

We have a large json file which is more than 100GB and we want to split this json file into multiple files. We used Split Text processor to split this json file into mutliple files by specifying Line Split Count. Is there any way we can pass attribute/variable in Line Split Count and then split the records based on the attribute/variable as currently Line Split Count does not support attributes/variables.

Kindly suggest if there is another approach to split these json files based on attribute/variables

Sample Json File

{"name": "John","lastName": "Wick","phoneNumber": "123123123"} 
{"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}
{"name": "Paul","lastName": "Jackson","phoneNumber": "123123123"}

Re: How to split large json file into multiple json files in Nifi?

SAMSAL — Sun, 19 Feb 2023 16:16:22 GMT

Hi,

Try to look into QueryRecord or PartitionRecord Processors. Those might help.

Thanks

Re: How to split large json file into multiple json files in Nifi?

rahul_loke — Thu, 23 Feb 2023 16:37:24 GMT

Both QueryRecord and PartitionRecord do not fit this use case, I have tried it. Can SplitRecord processor be used this purpose, is yes can you provide an example based on the above sample records?

Re: How to split large json file into multiple json files in Nifi?

DigitalPlumber — Mon, 27 Feb 2023 17:25:20 GMT

Yes SplitRecord is what you should use.
Attached is a flow definition as an example.

Note that I had to rename the file with a "txt" extension once you download it rename it to a .json extension

You can then drag a processor group and it gives you an option to upload the flow definition.

That example generates a file with 102 records and on SlitRecord we use a JsontTreeReader that will split by 3 records and writes the flowfiles out, In this case per 3 per flowFile generating 34 FlowFiles.

1-2 / 3 = 34

In your case and based on your screenshot I would change split count to be 1500000 ( or another number based on your needs )

Re: How to split large json file into multiple json files in Nifi?

DianaTorres — Thu, 02 Mar 2023 17:42:37 GMT

@rahul_loke Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks