Support Questions

Find answers, ask questions, and share your expertise

Unable to open application master UI in spark1.6.1 in cluster mode

New Contributor

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2.

when i submit the application in cluster mode. the driver Container can be in any of the 8 nodes. If driver Container goto the node 3 (where Active ResourceManager is service is running) then i am able to open application master UI, but in other cases it is not opening, after some time ambari will show critical alert with message Connection failed to resource manager host url.

if i check the resource manager logs. It shows some access exception for spark user while calling getServiceState.

Here is the full stack strace :

2018-08-25 05:02:30,209 WARN  resourcemanager.AdminService ( - User spark doesn't have permission to call 'getServiceState'
2018-08-25 05:02:30,210 WARN  resourcemanager.RMAuditLogger ( - USER=spark IP=  OPERATION=getServiceState TARGET=AdminService RESULT=FAILURE  DESCRIPTION=Unauthorized user PERMISSIONS=
2018-08-25 05:02:30,210 INFO  ipc.Server ( - IPC Server handler 0 on 8033, call org.apache.hadoop.ha.HAServiceProtocol.getServiceStatus from Call#51845 Retry#0 User spark doesn't have permission to call 'getServiceState'
  at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(
  at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(
  at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAccess(
  at org.apache.hadoop.yarn.server.resourcemanager.AdminService.getServiceStatus(
  at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.getServiceStatus(
  at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
  at org.apache.hadoop.ipc.RPC$
  at org.apache.hadoop.ipc.Server$Handler$
  at org.apache.hadoop.ipc.Server$Handler$
  at Method)
  at org.apache.hadoop.ipc.Server$
2018-08-25 05:05:43,300 INFO  client.DefaultHttpClient ( - I/O exception (org.apache.http.NoHttpResponseException) caught when processing request: The target server failed to respond

Hi @Shashi Vk,

Can you add user 'spark' in yarn.admin.acl via ambari Ambari->service>Yarn->configs>yarn.admin.acl comma seperated and restart resourcemanager and try.

Please accept answer if this helped.

New Contributor

Hi Akhil, updated the yarn.admin.acl with yarn,spark and restarted all required components. But still facing the same issue. This time if I click on application master. Even resource manager UI is not opening for some time. resource manager lists below log for many times.

2018-08-28 06:24:17,048 INFO  webproxy.WebAppProxyServlet ( - dr.who is accessing unchecked which is the app master GUI of application_1535435544017_0008 owned by spark