Out of Memory Heap Error When Processing Bulk File (1.26.0)

[#1]

Hello Team,

I am Sachin Duggal , and I am the owner of a bags manufacturing business. I am encountering an “Out of memory heap” error while processing a large text file (26k+ rows) through SFTP. The file contains | separated values, and I'm using the following processors for data manipulation. However, the process halts after the splitJSON processor.

Has anyone else faced a similar issue or has any suggestions on how to optimize the memory usage or address this issue? I would appreciate any insights or alternative approaches that could help resolve this.

Looking forward to your suggestions!

Sachin Duggal 

[#2]

Welcome to our community! To help you get the best possible answer, I have tagged our Airflow expert @smdas who may be able to assist you further.

Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.