User is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM causing

[#1]

Hi All,

After the integration of AD with kerberos we are not able to start the RM due the "User impersonate" issue.

[Error: User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM]

Loggers for RM:

2021-04-21 16:10:08,878 INFO service.AbstractService (AbstractService.java:noteFailure(267)) - Service ResourceManager failed in state STARTED
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1324)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1513)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
at org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter.<init>(FileSystemTimelineWriter.java:119)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.createTimelineWriter(TimelineClientImpl.java:152)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:143)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
... 6 more
2021-04-21 16:10:08,881 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.w.WebAppContext@5e1a986c{/,null,UNAVAILABLE}{/cluster}
2021-04-21 16:10:08,888 INFO server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped ServerConnector@2de9ca6{HTTP/1.1,[http/1.1]}{0.0.0.0:8088}
2021-04-21 16:10:08,891 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.w.WebAppContext@1e545821{/ui2,file:///usr/hdp/3.0.1.0-187/hadoop-yarn/webapps/ui2/,UNAVAILABLE}
2021-04-21 16:10:08,891 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@3d904e9c{/static,jar:file:/usr/hdp/3.0.1.0-187/hadoop-yarn/hadoop-yarn-common-3.1.1.3.0.1.0-187.jar!/webapps/static,UNAVAILABLE}
2021-04-21 16:10:08,891 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@658255aa{/logs,file:///u01/var/log/hadoop-yarn/yarn/,UNAVAILABLE}
2021-04-21 16:10:08,893 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,894 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,894 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,894 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,894 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,894 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,894 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,895 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,895 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,895 INFO event.AsyncDispatcher (AsyncDispatcher.java:serviceStop(155)) - AsyncDispatcher is draining to stop, ignoring any new events.
2021-04-21 16:10:08,895 INFO storage.HBaseTimelineWriterImpl (HBaseTimelineWriterImpl.java:serviceStop(580)) - closing the entity table
2021-04-21 16:10:08,906 INFO storage.HBaseTimelineWriterImpl (HBaseTimelineWriterImpl.java:serviceStop(585)) - closing the app_flow table
2021-04-21 16:10:08,906 INFO storage.HBaseTimelineWriterImpl (HBaseTimelineWriterImpl.java:serviceStop(590)) - closing the application table
2021-04-21 16:10:08,906 INFO storage.HBaseTimelineWriterImpl (HBaseTimelineWriterImpl.java:serviceStop(594)) - closing the flow run table
2021-04-21 16:10:08,906 INFO storage.HBaseTimelineWriterImpl (HBaseTimelineWriterImpl.java:serviceStop(599)) - closing the flowActivityTable table
2021-04-21 16:10:08,906 INFO storage.HBaseTimelineWriterImpl (HBaseTimelineWriterImpl.java:serviceStop(607)) - closing the hbase Connection
2021-04-21 16:10:08,906 INFO zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:clse(342)) - Close zookeeper connection 0x4bff2185 to hostname1.EXAMPLE.COM:2181,hostname2.EXAMPLE.COM:2181,hostname3.EXAMPLE.COM:2181
2021-04-21 16:10:08,907 INFO ipc.Server (Server.java:stop(3082)) - Stopping server on 8141
2021-04-21 16:10:08,909 INFO ipc.Server (Server.java:run(1185)) - Stopping IPC Server listener on 8141
2021-04-21 16:10:08,910 INFO resourcemanager.ResourceManager (ResourceManager.java:transitionToStandby(1302)) - Transitioning to standby state
2021-04-21 16:10:08,911 INFO resourcemanager.ResourceManager (ResourceManager.java:transitionToStandby(1309)) - Transitioned to standby state
2021-04-21 16:10:08,910 INFO ipc.Server (Server.java:run(1319)) - Stopping IPC Server Responder
2021-04-21 16:10:08,911 FATAL resourcemanager.ResourceManager (ResourceManager.java:main(1516)) - Error starting ResourceManager
org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1324)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1513)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: $UN3000-7G7U66I5CC6J@EXAMPLE.COM is not allowed to impersonate rm/hostname.EXAMPLE.COM@EXAMPLE.COM
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497)
at org.apache.hadoop.ipc.Client.call(Client.java:1443)
at org.apache.hadoop.ipc.Client.call(Client.java:1353)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:900)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1654)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
at org.apache.hadoop.yarn.client.api.impl.FileSystemTimelineWriter.<init>(FileSystemTimelineWriter.java:119)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.createTimelineWriter(TimelineClientImpl.java:152)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:143)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
... 6 more
2021-04-21 16:10:08,912 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(524)) - EventThread shut down
2021-04-21 16:10:08,911 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x278f31f01d00022 closed
2021-04-21 16:10:08,917 INFO resourcemanager.ResourceManager (LogAdapter.java:info(49)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down ResourceManager at hostname.EXAMPLE.COM/10.X.X.X

[#2]

@somesh Can you check Ambari => HDFS => Core-site proxy setting

try setting the proxy values to * as shown below

hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*

[#3]

Hi @Scharan ,
I have added below proxy values but getting same error. [that user name start with $OP something as normal user doesn't start with Alpha-numeric values] 

[#4]

Any new updates for fix? We are facing same issue yarn resource manager failed to start after kerbero and ad integration. yarn resource manager was able to start before kerbero. 

$Q0JU00-0VODD7BP1PAK@EXAMPLE.COM is not allowed to impersonate rm/hostname@EXAMPLE.COM

[#5 · Accepted solution]

Just check what Java version you are using. 

Important:

For JDK 8u241 and higher versions running on Kerberized clusters, you must disable referrals by setting sun.security.krb5.disableReferrals=true.

For example, with OpenJDK 1.8.0u242:

1. Open /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/lib/security/java.security with a text editor.
2. Add sun.security.krb5.disableReferrals=true (it can be at the bottom of the file).
3. Add this property on each node that has the impacted JDK version.
4. Restart the applications using the JDK so the change takes effect.

For more information, see the KB article.

[#6]

@GangWar Thanks, this worked for me.