question Re: Spark Job long GC pauses in Support Questions

question Re: Spark Job long GC pauses in Support Questions https://community.cloudera.com/t5/Support-Questions/Spark-Job-long-GC-pauses/m-p/282805#M210208 <P>Hi <a href="https://community.cloudera.com/t5/user/viewprofilepage/user-id/23013">w@leed</a> </P><P> </P><P>Thanks for Replying.</P><P> </P><P>I did test the Job with all the three Collectors - ParallelGC, CMS and G1GC:</P><P> </P><P>I has tested following options with the G1GC:</P><P> </P><TABLE border="1"><TBODY><TR><TD><SPAN>-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps</SPAN></TD></TR></TBODY></TABLE><P> </P><P>and with CMS:</P><P> </P><DIV><TABLE border="1"><TBODY><TR><TD><SPAN>-XX:+UseConcMarkSweepGC </SPAN><SPAN>-XX:+PrintGCTimeStamps</SPAN><SPAN> -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseParNewGC  -XX:+CMSConcurrentMTEnabled </SPAN><SPAN>-XX:ParallelCMSThreads=10 </SPAN><SPAN>-XX:ConcGCThreads=8 -XX:ParallelGCThreads=16</SPAN></TD></TR></TBODY></TABLE><SPAN><BR /></SPAN></DIV><DIV><SPAN><SPAN>With G1GC defaults, I could see following: </SPAN></SPAN></DIV><DIV><SPAN><SPAN><BR /></SPAN></SPAN><TABLE border="1"><TBODY><TR><TD>Desired survivor size 1041235968 bytes, new threshold 5 (max 15)<BR />[PSYoungGen: 1515304K->782022K(3053056K)] 2750361K->2017087K(6371840K), 1.5875321 secs] [Times: user=4.72 sys=0.74, real=1.59 secs]<BR />Heap after GC invocations=9 (full 3):<BR />PSYoungGen total 3053056K, used 782022K [0x0000000580000000, 0x000000068ef80000, 0x0000000800000000)<BR />eden space 2270720K, 0% used [0x0000000580000000,0x0000000580000000,0x000000060a980000)<BR />from space 782336K, 99% used [0x000000065f380000,0x000000068ef31ab0,0x000000068ef80000)<BR />to space 1016832K, 0% used [0x0000000612d80000,0x0000000612d80000,0x0000000650e80000)<BR />ParOldGen total 3318784K, used 1235064K [0x0000000080000000, 0x000000014a900000, 0x0000000580000000)<BR />object space 3318784K, 37% used [0x0000000080000000,0x00000000cb61e318,0x000000014a900000)<BR />Metaspace used 55055K, capacity 55638K, committed 55896K, reserved 1097728K<BR />class space used 7049K, capacity 7207K, committed 7256K, reserved 1048576K<BR />}<BR />{Heap before GC invocations=10 (full 3):<BR />PSYoungGen total 3053056K, used 3052742K [0x0000000580000000, 0x000000068ef80000, 0x0000000800000000)<BR />eden space 2270720K, 100% used [0x0000000580000000,0x000000060a980000,0x000000060a980000)<BR />from space 782336K, 99% used [0x000000065f380000,0x000000068ef31ab0,0x000000068ef80000)<BR />to space 1016832K, 0% used [0x0000000612d80000,0x0000000612d80000,0x0000000650e80000)<BR />ParOldGen total 3318784K, used 1235064K [0x0000000080000000, 0x000000014a900000, 0x0000000580000000)<BR />object space 3318784K, 37% used [0x0000000080000000,0x00000000cb61e318,0x000000014a900000)<BR />Metaspace used 55108K, capacity 55702K, committed 55896K, reserved 1097728K<BR />class space used 7049K, capacity 7207K, committed 7256K, reserved 1048576K<BR />42.412: [GC (Allocation Failure)<BR />Desired survivor size 1653080064 bytes, new threshold 4 (max 15)<BR />[PSYoungGen: 3052742K->1016800K(3422720K)] 4287807K->2985385K(6741504K), 4.0304873 secs] [Times: user=11.87 sys=1.77, real=4.03 secs]<BR />Heap after GC invocations=10 (full 3):<BR />PSYoungGen total 3422720K, used 1016800K [0x0000000580000000, 0x0000000727a80000, 0x0000000800000000)<BR />eden space 2405888K, 0% used [0x0000000580000000,0x0000000580000000,0x0000000612d80000)<BR />from space 1016832K, 99% used [0x0000000612d80000,0x0000000650e78240,0x0000000650e80000)<BR />to space 1614336K, 0% used [0x00000006c5200000,0x00000006c5200000,0x0000000727a80000)<BR />ParOldGen total 3318784K, used 1968584K [0x0000000080000000, 0x000000014a900000, 0x0000000580000000)<BR />object space 3318784K, 59% used [0x0000000080000000,0x00000000f8272318,0x000000014a900000)<BR />Metaspace used 55108K, capacity 55702K, committed 55896K, reserved 1097728K<BR />class space used 7049K, capacity 7207K, committed 7256K, reserved 1048576K</TD></TR></TBODY></TABLE></DIV><DIV> </DIV><DIV><SPAN>With all the Collectors only difference I could see was that, a delayed full GC.</SPAN></DIV><DIV><SPAN>I am considering to changing the YoungGen now. Will update if I do see a difference.</SPAN></DIV><DIV> </DIV><DIV><SPAN>On a parallel note -</SPAN></DIV><DIV><SPAN>1. I did also see that there are some of the objects in the memory which remain persistent across GC cycles - for example : scala.Tuple2 and java.lang.Long </SPAN></DIV><DIV><SPAN>2. These are Java RDD's</SPAN></DIV><DIV> </DIV><DIV><SPAN>Regards</SPAN></DIV><DIV><SPAN> </SPAN></DIV><DIV> </DIV><DIV> </DIV> Wed, 13 Nov 2019 00:45:14 GMT PARTOMIA09 2019-11-13T00:45:14Z