Support Questions
Find answers, ask questions, and share your expertise

AMS in distributed mode with Kerberos

AMS in distributed mode with Kerberos

New Contributor

Hi all.

 

Our cluster is running on HDP 3.1.4 with Kerberos enabled. An Active Directory server is used as the Kerberos host. We initially installed AMS with the default embedded mode and this worked without any issues.

 

After we configured the AMS to use the distributed mode  Ambari Metric Collector started up but in the hbase-ams-regionserver.log we got this warning/error:

 

2020-05-13 20:18:10,253 WARN [regionserver/xxxxx-cbs-hb-wk-kr1:61320] regionserver.HRegionServer: error telling master we are up
org.apache.hbase.thirdparty.com.google.protobuf.ServiceException: java.io.IOException: Call to xxxxx-cbs-hb-wk-kr1.reddog.cluster.net/172.17.6.133:61300 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: ams
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:13578)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2611)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:941)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Call to xxxxx-cbs-hb-wk-kr1.reddog.cluster.net/172.17.6.133:61300 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException): Kerberos principal name does NOT have the expected hostname part: ams
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:185)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)

 

Ambari is alerting with:

Connection failed: [Errno 111] Connection refused to xxxxx-cbs-hb-wk-kr1.reddog.cluster.net:6188

 

We also checked the server and no process is listing on port 6188.

 

Can this great community please help us with this error?

What does the log statement "Kerberos principal name does NOT have the expected hostname part: ams" mean?

 

Thanks in advance