I have a kerberized cluster which is running Ranger KMS. I am trying to run HAWQ on the cluster however I am running into issues when HAWQ segments request containers through YARN Resource Manager. The secure HAWQ installation uses the user "postgres" to request containers. The following is the error message reported in YARN Resource Manager
2017-03-23 10:56:30,816 INFO hdfs.DFSClient (DFSClient.java:getDelegationToken(1043)) - Created HDFS_DELEGATION_TOKEN token 20049 for postgres on 192.168.59.104:8020
2017-03-23 10:56:30,889 WARN security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1032)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2298)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.obtainSystemTokensForUser(DelegationTokenRenewer.java:679)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.requestNewHdfsDelegationToken(DelegationTokenRenewer.java:643)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:488)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:868)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1014)
... 16 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%4..., status: 403, message: Forbidden
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1019)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
... 17 more
I see the following in the Ranger KMS log (kms.log)
2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting up[http-bio-9292-Acceptor-0] latch=7
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has value [/kms/v1/]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has value [-1]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value [ISO-8859-1]
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking request OPTIONS /kms/v1/
2017-03-23 11:02:00,738 DEBUG RealmBase - No applicable constraints defined
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Not subject to any constraint
2017-03-23 11:02:00,738 TRACE StandardWrapper - Returning non-STM instance
2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket: [org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State out: [OPEN]
2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request header
java.io.EOFException: Unexpected EOF read on the socket
at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:169)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:990)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket: [org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State out: [CLOSE
D]
2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/192.168.59.104,port=58547,localport=9292]The following is from the Ranger KMS access log:
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres HTTP/1.1" 403 258
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 403 258
The following is from the Ranger KMS audit log (kms-audit.log)
2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104 Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres ErrorMsg:'Authentication required'
2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104 Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres&user.name=yarn ErrorMsg:'Authentication required'
I have added the following proxyuser configuration in Ranger KMS as well:
hadoop.kms.proxyuser.postgres.users=*
hadoop.kms.proxyuser.postgres.hosts=*
hadoop.kms.proxyuser.yarn.users=*
hadoop.kms.proxyuser.yarn.hosts=*
The core-site.xml has the required proxyuser configuration as well:
hadoop.proxyuser.postgres.groups=*
hadoop.proxyuser.postgres.hosts=*
hadoop.proxyuser.yarn.groups=*
hadoop.proxyuser.yarn.hosts=*
But nothing seem to be working in this case here.
