https://community.cloudera.com/t5/Support-Questions/Out-of-Memory-Heap-Error-When-Processing-Bulk-File-1-26-0/m-p/399007#M250370 <P>Welcome to our community! To help you get the best possible answer, I have tagged our Airflow expert&nbsp;<a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/40384">@smdas</a>&nbsp;who may be able to assist you further.<BR /><BR />Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.</P> Wed, 18 Dec 2024 05:56:12 GMT VidyaSargur 2024-12-18T05:56:12Z https://community.cloudera.com/t5/Support-Questions/Out-of-Memory-Heap-Error-When-Processing-Bulk-File-1-26-0/m-p/398975#M250351 <DIV class="flex-1 overflow-hidden @container/thread"> <DIV class="h-full"> <DIV class="react-scroll-to-bottom--css-lndsn-79elbk h-full"> <DIV class="react-scroll-to-bottom--css-lndsn-1n7m0yu"> <DIV class="flex flex-col text-sm"> <DIV class="m-auto text-base py-[18px] px-3 md:px-4 w-full md:px-5 lg:px-4 xl:px-5"> <DIV class="mx-auto flex flex-1 gap-4 text-base md:gap-5 lg:gap-6 md:max-w-3xl"> <DIV class="group/conversation-turn relative flex w-full min-w-0 flex-col agent-turn"> <DIV class="flex-col gap-1 md:gap-3"> <DIV class="flex max-w-full flex-col flex-grow"> <DIV class="min-h-8 text-message flex w-full flex-col items-end gap-2 whitespace-normal break-words text-start [.text-message+&amp;]:mt-5"> <DIV class="flex w-full flex-col gap-1 empty:hidden first:pt-[3px]"> <DIV class="markdown prose w-full break-words dark:prose-invert light"> <P>Hello Team,</P> <P><SPAN>I am&nbsp;</SPAN><STRONG><EM>Sachin Duggal</EM></STRONG><SPAN>&nbsp;, and I am the owner of a bags manufacturing business.&nbsp;</SPAN>I am encountering an “Out of memory heap” error while processing a large text file (26k+ rows) through SFTP. The file contains | separated values, and I'm using the following processors for data manipulation. However, the process halts after the <STRONG>splitJSON</STRONG> processor.</P> <P>Has anyone else faced a similar issue or has any suggestions on how to optimize the memory usage or address this issue? I would appreciate any insights or alternative approaches that could help resolve this.</P> <P>Looking forward to your suggestions!</P> <P><STRONG>Sachin Duggal&nbsp;</STRONG></P> </DIV> </DIV> </DIV> </DIV> <DIV class="mb-2 flex gap-3 empty:hidden -ml-2"> <DIV class="items-center justify-start rounded-xl p-1 flex"> <DIV class="flex items-center">&nbsp;</DIV> </DIV> </DIV> <DIV class="pr-2 lg:pr-0">&nbsp;</DIV> <DIV class="mt-3 w-full empty:hidden"> <DIV class="text-center">&nbsp;</DIV> </DIV> </DIV> <DIV class="absolute"> <DIV class="flex items-center justify-center">&nbsp;</DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> </DIV> <DIV class="md:pt-0 dark:border-white/20 md:border-transparent md:dark:border-transparent w-full"> <DIV> <DIV class="m-auto text-base px-3 md:px-4 w-full md:px-5 lg:px-4 xl:px-5"> <DIV class="mx-auto flex flex-1 gap-4 text-base md:gap-5 lg:gap-6 md:max-w-3xl"> <DIV class="flex justify-center">&nbsp;</DIV> </DIV> </DIV> </DIV> </DIV> Thu, 19 Dec 2024 05:46:53 GMT https://community.cloudera.com/t5/Support-Questions/Out-of-Memory-Heap-Error-When-Processing-Bulk-File-1-26-0/m-p/398975#M250351 sdbags 2024-12-19T05:46:53Z https://community.cloudera.com/t5/Support-Questions/Out-of-Memory-Heap-Error-When-Processing-Bulk-File-1-26-0/m-p/399007#M250370 <P>Welcome to our community! To help you get the best possible answer, I have tagged our Airflow expert&nbsp;<a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/40384">@smdas</a>&nbsp;who may be able to assist you further.<BR /><BR />Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.</P> Wed, 18 Dec 2024 05:56:12 GMT https://community.cloudera.com/t5/Support-Questions/Out-of-Memory-Heap-Error-When-Processing-Bulk-File-1-26-0/m-p/399007#M250370 VidyaSargur 2024-12-18T05:56:12Z