"ActivityAnalyzerFacade" consuming high ammount of memory resources for a server and launched by root

[#1]

We are observing mentioned process is consuming high CPU resources for a particular server and going through the clutser looking for same process, I 've founded on other 2 servers, running under a different user and they are not even close to the consumpltion level like the one I am having the doubt with.

What I want to know is if this is a normal behavior and if I should't worry abou it.

server_1_[root@server_1 ~]# top -p 16193
top - 18:04:58 up 479 days, 21:34,  2 users,  load average: 5.79, 3.53, 2.51
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.4 us,  4.0 sy,  0.0 ni, 75.1 id,  0.0 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem : 26384640+total, 11807649+free, 26398576 used, 11937132+buff/cache
KiB Swap:  4194556 total,  4194556 free,        0 used. 23283172+avail Mem


  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
16193 alipiah   20   0 15.861g 2.012g  30100 S   0.7  0.8  14:42.70 java


**********************************************************************************


server_2_[root@server_2 ~]# top -p 9864
top - 18:06:52 up 479 days, 21:36,  2 users,  load average: 0.39, 0.47, 0.49
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.8 us,  0.3 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 26384640+total, 66057520 free, 26081160 used, 17170771+buff/cache
KiB Swap:  4194556 total,  4194556 free,        0 used. 23298083+avail Mem


  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 9864 alipiah   20   0 15.857g 438752  30048 S   0.0  0.2   3:12.83 java


**********************************************************************************


server_3_[root@server_3 ~]# top -p 344095
top - 18:08:43 up 479 days, 21:38,  2 users,  load average: 38.80, 35.37, 32.46
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s): 81.3 us,  1.9 sy,  0.0 ni, 16.8 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 26384640+total, 19629884 free, 42436532 used, 20177996+buff/cache
KiB Swap:  4194556 total,  4194556 free,        0 used. 21663824+avail Mem


   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
344095 root      20   0 29.954g 8.888g  23968 S  2129  3.5  82702:17 java

[#2 · Accepted solution]

@Jorge Luis Hernandez Olmos,

We have seen this happen in some versions. There are some Tez analyzers that consume a lot of resources. Do this config change to disable the analyzers that consume more resources. Disabling these does not impact your normal processing or analysis.

Ambari > SmartSenseConfigs> custom activity-conf :

tez_job.activity.analyzers= com.hortonworks.smartsense.activity.job.tez.analyzer.SlowNodeAnalyzer, com.hortonworks.smartsense.activity.job.tez.analyzer.LocalityAnalyzer, com.hortonworks.smartsense.activity.job.tez.analyzer.SkewAnalyzer, com.hortonworks.smartsense.activity.job.tez.analyzer.SlowestVertexAnalyzer, com.hortonworks.smartsense.activity.job.tez.analyzer.SpillAnalyzer

This disables ContainerReuseAnalyzer, SlowTaskIdentifier, CriticalPathAnalyzerandShuffleTimeAnalyzer.

Pls note, by default activity analysis consumes minimum of 8 GB of heap even if above analyzers are disabled. So, it is ok to see such consumption even after disabling these tez analyzers.