Out of Memory Heap Error When Processing Bulk File (1.26.0)

Question

Hello Team, I am Sachin Duggal , and I am the owner of a bags manufacturing business. I am encountering an “Out of memory heap” error while processing a large text file (26k+ rows) through SFTP. The file contains | separated values, and I'm using the following processors for data manipulation. However, the process halts after the splitJSON processor. Has anyone else faced a similar issue or has any suggestions on how to optimize the memory usage or address this issue? I would appreciate any insights or alternative approaches that could help resolve this. Looking forward to your suggestions! Sachin Duggal

VidyaSargur · Answer

Welcome to our community! To help you get the best possible answer, I have tagged our Airflow expert @smdas who may be able to assist you further.

Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.