https://community.cloudera.com/t5/Support-Questions/Converting-Large-CSV-into-JSON/m-p/242775#M204574 <P>I have a relatively large CSV (~80GB) I need to transform into multiple JSON documents/records. I'm using a ConvertRecord processor with a CSVReader and AvroRecordSetWriter and that's where my CSV gets stuck. What's the best approach? Break up the CSV prior to converting it or try to get more horsepower on the server? </P><UL><LI>Server Mem: 16GB</LI><LI>Cores: 4</LI><LI>Maximum Timer Driven Thread Count : 16</LI><LI>Java Min/Max Heap: 2GB/10GB</LI></UL> Tue, 18 Jun 2019 19:05:48 GMT millerbill3 2019-06-18T19:05:48Z https://community.cloudera.com/t5/Support-Questions/Converting-Large-CSV-into-JSON/m-p/242775#M204574 <P>I have a relatively large CSV (~80GB) I need to transform into multiple JSON documents/records. I'm using a ConvertRecord processor with a CSVReader and AvroRecordSetWriter and that's where my CSV gets stuck. What's the best approach? Break up the CSV prior to converting it or try to get more horsepower on the server? </P><UL><LI>Server Mem: 16GB</LI><LI>Cores: 4</LI><LI>Maximum Timer Driven Thread Count : 16</LI><LI>Java Min/Max Heap: 2GB/10GB</LI></UL> Tue, 18 Jun 2019 19:05:48 GMT https://community.cloudera.com/t5/Support-Questions/Converting-Large-CSV-into-JSON/m-p/242775#M204574 millerbill3 2019-06-18T19:05:48Z https://community.cloudera.com/t5/Support-Questions/Converting-Large-CSV-into-JSON/m-p/242776#M204575 <P><A rel="user" href="https://community.cloudera.com/users/114774/millerbill3.html">@Bill Miller</A></P><P>Try with series of <A rel="noopener noreferrer noopener noreferrer noopener noreferrer noopener noreferrer noopener noreferrer noopener noreferrer" href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.SplitRecord/index.html" target="_blank">SplitRecord</A> processors to create smaller chunks of files.</P><P>Follow the similar approach mentioned in <A rel="noopener noreferrer noopener noreferrer noopener noreferrer noopener noreferrer noopener noreferrer noopener noreferrer" href="https://community.hortonworks.com/questions/122858/nifi-splittext-big-file.html" target="_blank">this</A> thread and see if you get any performance with this approach.</P><P><BR /></P> Wed, 19 Jun 2019 08:21:33 GMT https://community.cloudera.com/t5/Support-Questions/Converting-Large-CSV-into-JSON/m-p/242776#M204575 Shu_ashu 2019-06-19T08:21:33Z