Member since
08-22-2019
4
Posts
0
Kudos Received
0
Solutions
12-04-2019
06:23 AM
@pxm You'll want to use the SplitRecord processor to accomplish what you are looking to do. Configure it with a RecordReader and RecordWriter for you input data and desired output data formats along with the desired 1000 records per split. Hope this helps, Matt
... View more
11-04-2019
06:00 AM
1 Kudo
@pxm NiFi sets not restriction on the data size that can be processed. Ingested data becomes the content portion of a NiFi FlowFile and is written to the content repository. The data is not read again unless a processor needs to read the content; otherwise, only the FlowFile attributes/metadata is passed from processor component to another component. So you need to make sure you have sufficient storage space for the NIFi content_repository. It is also strongly recommended that this dedicated storage separate from any other NiFi repository. Beyond that, any limitation here will be on network and disk IO. Thanks, Matt
... View more