https://community.cloudera.com/t5/Support-Questions/Split-huge-file-one-file-for-each-day-based-on-date-column/m-p/203146#M165149 <A rel="user" href="https://community.cloudera.com/users/98268/ramgood.html" nodeid="98268">@Ram G</A><P>In NiFi we are having <STRONG>partition record</STRONG> processor, based on the content of the flowfile processor creates <STRONG>dynamic partitions</STRONG> and adds the partition_field_name and value as the attribute to the flowfile.</P><P>By using these attributes we can store the data into HDFS directories dynamically.</P><P>To read the content of the flowfile you need to define </P><P><STRONG>RecordReader Controller service</STRONG> as <STRONG>CSV Reader </STRONG>and <STRONG>value seperator </STRONG>as <B>\t(</B>as you are having tab delimited file), define RecordWriter controller service as per your requirements(like avro,json..etc)</P><P>But keep in mind as you mentioned you are having more than 100 GB file and thinking to split the file, For this case <STRONG>i believe Hive will work much better to create Dynamic partitions.</STRONG>Store the file into HDFS then create Hive External table with tab delimiter and create partition table and insert into Partition table select from non_partition_table.</P><P>How ever if you want to do this in NiFi make sure you are having <A href="https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html" target="_blank">sufficient memory </A>in your NiFi instance once you pull the file into NiFi use <STRONG>SplitRecord</STRONG> processor to Split the <STRONG>Huge file</STRONG> into reasonable smaller chunks then feed the <STRONG>splitted</STRONG> flowfiles to <STRONG>PartitionRecord</STRONG> processor.Once you have created partitions then store the flowfiles into HDFS.</P><P>Refer to <A href="https://community.hortonworks.com/articles/191760/create-dynamic-partitions-based-on-flowfile-conten.html" target="_blank">this</A> link for more details regards to PartitionRecord processor Usage/Configurations.</P><P>Refer to <A href="https://community.hortonworks.com/articles/85234/how-to-address-jvm-outofmemory-errors-in-nifi.html" target="_blank">this</A> link for Jvm OutofMemory issues in NiFi.</P><P>-</P><P>If the Answer helped to resolve your issue, <STRONG>Click on Accept button below to accept the answer,</STRONG> That would be great help to Community users to find solution quickly for these kind of issues.</P> Thu, 27 Sep 2018 05:44:44 GMT Shu_ashu 2018-09-27T05:44:44Z