Memory Utilization is high unable to find what causing this

[#1]

[#2 · Accepted solution]

Hi @deepak rathod Click on the Scheduler link on the Resource Manager page. It will tell you what is the resource utilization for each queue and and you can drill down to identify which jobs are consuming total of 279 containers ? Appears that you have 279 containers running with avg. of 12 GB , with total of 3.41 TB memory reserved.

[#3]

Hi @Saumil Mayani i found jobs their container comsumption but unable to drill down to see the memory used by each job. is there any way to drill down to memory consumption level?

[#4]

Hi @deepak rathod Could you please share the screen capture of the scheduler page. It should have for each application, Allocated Memory and vCores, allong with Running Containers. you can sort and see which application has the most Allocated Memory MB. Sample attached.

[#5]

This is my scheduler @Saumil Mayani

[#6]

Hi @deepak rathod appears that the hdp version you are running, does not have this information on YARN Resource Manager UI. Sample screenshot I attached earlier was from HDP-2.6.2.0-205. You may need to upgrade HDP stack.

[#7]

@Saumil Mayani but my version is Hadoop 2.7.1.2.3.2.0-2950

[#8]

Hi @deepak rathod yes, you are using HDP-2.3.2.0. You need to upgrade to HDP-2.6.2.0.

Here is the doc:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/index.html

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-upgrade/content/ambari_upgrade_gu...