Member since
08-22-2019
4
Posts
0
Kudos Received
0
Solutions
11-27-2019
12:05 PM
Hi,
I am a newbie to Nifi and would like some guidance please.
We want to split a large Json file into multiple files with a specified number of records. I am able to split a file into individual records using SplitJson and the Json Path Expression set as $.<objectthatcontainsthearray>.* I have also added an UpdateAttribute Processor with filename set to ${filename}_${fragment.index} so that we have the sequence of the files as order is important.
However, we might want to have say a 100,000 records split into 100 files of 1000 records each . What is the easiest way to do this ?
Thanks very much in advance
... View more
Labels:
- Labels:
-
Apache NiFi
11-02-2019
01:47 PM
I am quite new to nifi. We want to download a csv file from an FtpServer and insert in into HDFS.
I have got this working with a small test file by using the getSFTP and putHDFS processors. We now want to work with our first actual data file of around 15Gb, and later date files of possibly in the 200-500 GB range. Are there any restrictions on the size of the files that can be ingested, and if so, how can these be overcome ?
Thanks very much in advance
... View more
Labels:
- Labels:
-
Apache NiFi