https://community.cloudera.com/t5/Support-Questions/Hive-LLAP-containers-are-getting-killed-by-Yarn-and-new/m-p/335671#M231993 <P>Hi Everyone,</P><P>We have a situation where yarn is killing llap application containers and then requesting to launch new ones. This causes a brief unavailability in llap daemons and running applications fail because of this.</P><P>When we reviewed some of the container logs, saw following message:</P><P>&nbsp;</P><P><SPAN>2022-01-26 03:15:48,339</SPAN><SPAN> [Component&nbsp; dispatcher] </SPAN><SPAN>ERROR</SPAN><SPAN> instance.ComponentInstance - [COMPINSTANCE llap-0 : container_e127_1642817883045_7610_01_000002]: </SPAN><SPAN>container_e127_1642817883045_7610_01_000002 completed. Reinsert back to pending list and requested a new container</SPAN><SPAN>.</SPAN></P><P><SPAN>&nbsp;exitStatus=-104, diagnostics=[2022-01-26 03:15:47.314]Container [pid=8434,containerID=container_e127_1642817883045_7610_01_000002] i</SPAN><SPAN>s running 665411584B beyond the 'PHYSICAL' memory limit. Current usage: 75.6 GB of 75 GB physical memory used</SPAN><SPAN>; 77.6 GB of 157.5 GB virtual memory used. Killing container.</SPAN></P><P>&nbsp;</P><P><SPAN>I don't understand from where this 75.6GB of 75 GB limit is coming from? I have tried increasing the memory per llap daemon but it doesn't help either. Parameters:</SPAN></P><P><SPAN>1. Memory allocated for all yarn containers on a node is = 95 GB </SPAN></P><P><SPAN>2. llap memory per daemon = 75 Gb</SPAN></P><P><SPAN>3. memory cache per daemon = 20 Gb</SPAN></P><P><SPAN>4. llap_daemon_overhead= 6 GB</SPAN></P><P>&nbsp;</P><P><SPAN>Hive servers2 or hive-interacative-server logs don't provide much detail either. What other properties I can fine tune to fix this? Any help is appreciated.</SPAN></P><P>&nbsp;</P> Fri, 04 Feb 2022 20:30:53 GMT Priyanka26 2022-02-04T20:30:53Z