Support Questions
Find answers, ask questions, and share your expertise

LLAP Start Error after increase memory parameters

Re: LLAP Start Error after increase memory parameters

Contributor

Hi! all Do not understand why this happen : Error : is running beyond physical memory limits. Current usage: 21.3 GB of 21 GB physical memory used;

/usr/hdp/current/hive-server2-hive2/bin/hive --service llapstatus


"hostname" : "serv06.kyivstar.ua",
    "containerId" : "container_e122_1512988591180_3785_01_000030",
    "logUrl" : "http://serv06.kyivstar.ua:8042/node/containerlogs/container_e122_1512988591180_3785_01_000030/hive",
    "diagnostics" : "Container [pid=33120,containerID=container_e122_1512988591180_3785_01_000030] 
	is 	running beyond physical memory limits. Current usage: 21.3 GB of 21 GB physical memory used; 30.8 GB of 44.1 GB virtual memory used. Killing container.\nDump 
	of the process-tree for container_e122_1512988591180_3785_01_000030 :\n\t|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) 
	RSSMEM_USAGE(PAGES) FULL_CMD_LINE\n\t|- 33132 33120 33120 33120 (python) 30 6 458362880 4302 python ./infra/agent/slider-agent/agent/main.py --label 
	container_e122_1512988591180_3785_01_000030___LLAP --zk-quorum serv03.kyivstar.ua:2181,serv01.kyivstar.ua:2181,serv02.kyivstar.ua:2181 -
	-zk-reg-path /registry/users/hive/services/org-apache-slider/llap0 \n\t|- 33120 33118 33120 33120 (bash) 0 0 118026240 369 /bin/bash -c python 
	./infra/agent/slider-agent/agent/main.py --label container_e122_1512988591180_3785_01_000030___LLAP --zk-quorum serv03.kyivstar.ua:2181,
	serv01.kyivstar.ua:2181,serv02.kyivstar.ua:2181 --zk-reg-path /registry/users/hive/services/org-apache-slider/llap0 > 
	/grid/11/hadoop/yarn/log/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/slider-agent.out 2>&1  
	\n\t|- 33184 1 33120 33120 (java) 10487 1162 32513224704 5572712 /usr/jdk64/jdk1.8.0_77/bin/java -Dproc_llapdaemon -Xms18251m -Xmx18251m -Dhttp.maxConnections=19 
	-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:TLABSize=8m -XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:MetaspaceSize=1024m -XX:InitiatingHeapOccupancyPercent=90 
	-XX:MaxGCPauseMillis=200 -Xmx25251m -XX:MetaspaceSize=1024m -server -Djava.net.preferIPv4Stack=true -XX:+UseNUMA -XX:+PrintGCDetails -verbose:gc 
	-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=4 -XX:GCLogFileSize=100M -XX:+PrintGCDateStamps 
	-Xloggc:/grid/11/hadoop/yarn/log/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030//gc_2017-12-13-17.log -Djava.io.tmpdir=/grid/5/hadoop/yarn/local/
	usercache/hive/appcache/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/tmp/ -Dlog4j.configurationFile=llap-daemon-log4j2.properties 
	-Dllap.daemon.log.dir=/grid/11/hadoop/yarn/log/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/ 
	-Dllap.daemon.log.file=llap-daemon-hive-serv06.kyivstar.ua.log -Dllap.daemon.root.logger=query-routing -Dllap.daemon.log.level=ERROR -classpath 
	/grid/5/hadoop/yarn/local/usercache/hive/appcache/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/app/install//conf/:
	/grid/5/hadoop/yarn/local/usercache/hive/appcache/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/app/install//lib/*:
	/grid/5/hadoop/yarn/local/usercache/hive/appcache/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/app/install//lib/tez/*:
	/grid/5/hadoop/yarn/local/usercache/hive/appcache/application_1512988591180_3785/container_e122_1512988591180_3785_01_000030/app/install//lib/udfs/*:.: 
	org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon \n\nContainer killed on request. Exit code is 143\nContainer exited with a non-zero exit code 143. \n",
    "yarnContainerExitStatus" : 0
  } ]
}




LLAP start command: /usr/hdp/current/hive-server2-hive2/bin/hive --service llap --slider-am-container-mb 1024 --size 21251m 
                    --cache 2094m --xmx 18251m --loglevel ERROR  --output /var/lib/ambari-agent/tmp/llap-slider2017-12-13_15-28-50 
					--slider-placement 4 --skiphadoopversion --skiphbasecp --instances 18 --logger query-routing --args " -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:TLABSize=8m 
					-XX:+ResizeTLAB -XX:+UseNUMA -XX:+AggressiveOpts -XX:MetaspaceSize=1024m -XX:InitiatingHeapOccupancyPercent=90 -XX:MaxGCPauseMillis=200 -Xmx25251m 
					-XX:MetaspaceSize=1024m"


Re: LLAP Start Error after increase memory parameters

Contributor
##### LDAP Memory ##########
HiveServer Interactive Heap Size    23251
Memory per Daemon                   21251  
LLAP Daemon Heap Size (MB)          18251
In-Memory Cache per Daemon          2094
LLAP Daemon Container Max Headroom  1024