About michelle_lu

michelle_lu · ‎11-21-2018

@Jalender were you able to find out what was eating your resource manager heap? We are also experiencing the same thing but not daily. and our cluster is very much smaller than yours (only 29 data nodes) and when it is idle, RM heap should more or less be under 30% only. but on some days, we get very high RM heap when cluster is totally idle. also checked "yarn.resourcemanager.state-store.max-completed-applications" and it just means value is less than or equal to "yarn.resourcemanager.max-completed-applications" which is default at 10,000.

michelle_lu · ‎11-06-2018

hi @aengineer, azure agent is only on the juggernaut cluster (see below): is there going to be an issue with having azure storage account on juggernaut?

michelle_lu · ‎11-05-2018

Hello. We are trying to set up connection between our local HDP 2.5.6 cluster (datalab) and an external HDP 2.6.5 cluster (juggernaut). The idea is that we want to run spark (scala) jobs on yarn mode and it will call juggernaut cluster as data source. We have set up all possible connections but even simple "hdfs dfs -ls" command from datalab to juggernaut is giving out java.io.IOException: Connection reset by peer error. I have attached debug log when I ran ls command from datalab to juggernaut. datalab-debug-log.txt Both datalab and juggernaut are able to ping each other, telnet ok. We have also added all juggernaut nodes to /etc/hosts on datalab. We have also observed that we are able to run "hdfs dfs -ls" command from juggernaut to datalab. In summary: HDP 2.5.6 (datalab) --> HDP 2.6.5 (juggernaut) = NOT OK HDP 2.6.5 (juggernaut) --> HDP 2.5.6 (datalab) = OK Has anyone experienced something similar to this setup ang getting this issue?

Online	Offline
Last Visited	‎03-13-2019 03:29 AM

Date Registered	‎10-30-2018 08:43 AM
Last Visited	‎03-13-2019 03:29 AM
Total Messages Posted	3

Re: Resource manager heap usage

Re: Unable to establish connection between 2 clust...

Unable to establish connection between 2 clusters ...