env -i spark2-submit --keytab svc.keytab --principal svc@CORP.COM sample.py   We submit the jobs someothing like this.   ... View more

Thanks for help  ... View more

sorry forgot to mention.. I have been using executor memory of 14GB and driver memory of 10GB. None of my tasks spill memory to disk. this is so strange and its shaking my fundamental understanding of spark.  I have memory overhead of 3G.   Again, the same setting in CDSW are used but it never failed from there. Its when I run the job in Oozie it fails. It restarts on its own and that one completes without any failures.   When would spark use physical memory and virtual memory? ... View more

I applied this property and increased the limit to 6 GB. It still fails with exact same error message.   ... View more

Hi, We see the error as below. "The container showed error ...Container [pid=68208,containerID=container_e59_1543621459332_7731_01_000002] is running beyond physical memory limits. Current usage: 4.0 GB of 4 GB physical memory used; 26.5 GB of 8.4 GB virtual memory used. Killing container..."   When I run in CDSW or through Oozie both have same memory and configurations for my spark application(executor memory,core,driver memory,memory overhead etc). From CDSW it never failed but when I run from Oozie Shell Action (caloing spark2-submit) it randomly fails.  Trying to understand what is different in Oozie, how do I set this memory limit. ... View more

Most of the logs in spark UI are showing "No logs available for container" ... View more

Thanks for reply. I tried to see logs from SparkUI but it was blank. It will be great if you can guide me on checking yarn logs.  I tried below command and it gave me blank too yarn logs -applicationId application_1543621459332_8094 > spark_app.log     ... View more