Hello HCC, We have 16 node cluster running on HDP 2.5.3.0, Ambari 2.5.1.0, on this cluster we allocated resource manager heap to 4G and when there is no single job running in the cluster for almost 5-6hrs, but still resource manager heap usage is going to 80-85% always. We have another big production cluster with 800 data nodes in that cluster RM usage will be under 25-30% always.On what basis heap usage is utilized on the cluster ? and how can we control heap usage when cluster is idle. i also checked this property "yarn.resourcemanager.max-completed-applications" which is hard coded to "${yarn.resourcemanager.max-completed-applications}" but this hard coded property not defined anywhere in Ambari. Thanks in advance HCC. ... View more

our jobs are failing with below exception we are using HDP 2.5.3.0 , Tez 0.7.0.2.5 , Hive 1.2.1.2.5 and Ambari 2.4.0 failing with below exception Status: Failed Vertex failed, vertexName=Reducer 2, vertexId=vertex_**, diagnostics=[Exception in VertexManager, vertex:vertex_ [Reducer 2],org.apache.tez.dag.api.TezUncheckedException: Atleast 1 bipartite source should exist at org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.onVertexStarted(ShuffleVertexManager.java:456) at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEventOnVertexStarted.invoke(VertexManager.java:587) at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:658) at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent$1.run(VertexManager.java:653) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:653) at org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerEvent.call(VertexManager.java:642) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) , Vertex did not succeed due to AM_USERCODE_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_** [Reducer 2] killed/failed due to:AM_USERCODE_FAILURE] Vertex did not succeed due to AM_USERCODE_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_ [Reducer 2] killed/failed due to:AM_USERCODE_FAILURE]Vertex killed, vertexName=Reducer 3, vertexId=vertex_, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_ [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 4, vertexId=vertex_, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_ [Map 4] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2 After Staging execution is it something related to https://issues.apache.org/jira/browse/TEZ-3239 https://issues.apache.org/jira/browse/TEZ-2107 Thanks. ... View more