Support Questions

Find answers, ask questions, and share your expertise

HAWQ Issues with Ranger KMS

avatar
Contributor

I have a kerberized cluster which is running Ranger KMS. I am trying to run HAWQ on the cluster however I am running into issues when HAWQ segments request containers through YARN Resource Manager. The secure HAWQ installation uses the user "postgres" to request containers. The following is the error message reported in YARN Resource Manager

2017-03-23 10:56:30,816 INFO  hdfs.DFSClient (DFSClient.java:getDelegationToken(1043)) - Created HDFS_DELEGATION_TOKEN token 20049 for postgres on 192.168.59.104:8020
2017-03-23 10:56:30,889 WARN  security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(895)) - Unable to add the application to the delegation token renewer.
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1032)
        at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
        at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2298)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:685)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$2.run(DelegationTokenRenewer.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.obtainSystemTokensForUser(DelegationTokenRenewer.java:679)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.requestNewHdfsDelegationToken(DelegationTokenRenewer.java:643)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:488)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:77)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:891)
        at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:868)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1014)
        ... 16 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%4..., status: 403, message: Forbidden
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:278)
        at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:170)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1019)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        ... 17 more

I see the following in the Ranger KMS log (kms.log)

2017-03-23 11:02:00,734 DEBUG LimitLatch - Counting up[http-bio-9292-Acceptor-0] latch=7
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [uriBC] has value [/kms/v1/]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [semicolon] has value [-1]
2017-03-23 11:02:00,738 DEBUG CoyoteAdapter - The variable [enc] has value [ISO-8859-1]
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase - Security checking request OPTIONS /kms/v1/
2017-03-23 11:02:00,738 DEBUG RealmBase -   No applicable constraints defined
2017-03-23 11:02:00,738 DEBUG AuthenticatorBase -  Not subject to any constraint
2017-03-23 11:02:00,738 TRACE StandardWrapper -   Returning non-STM instance
2017-03-23 11:02:00,739 DEBUG Http11Protocol - Socket: [org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State out: [OPEN]
2017-03-23 11:02:00,758 DEBUG Http11Processor - Error parsing HTTP request header
java.io.EOFException: Unexpected EOF read on the socket
        at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:169)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:990)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
2017-03-23 11:02:00,758 DEBUG Http11Protocol - Socket: [org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/192.168.59.104,port=58547,localport=9292]], Status in: [OPEN_READ], State out: [CLOSE
D]
2017-03-23 11:02:00,758 TRACE JIoEndpoint - Closing socket:org.apache.tomcat.util.net.SocketWrapper@24800623:Socket[addr=/192.168.59.104,port=58547,localport=9292]

The following is from the Ranger KMS access log:

192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres HTTP/1.1" 403 258
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 401 997
192.168.59.104 - - [23/Mar/2017:11:02:00 -0700] "OPTIONS /kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=postgres&user.name=yarn HTTP/1.1" 403 258

The following is from the Ranger KMS audit log (kms-audit.log)

2017-03-23 11:02:00,738 UNAUTHENTICATED RemoteHost:192.168.59.104 Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres ErrorMsg:'Authentication required'
2017-03-23 11:02:00,786 UNAUTHENTICATED RemoteHost:192.168.59.104 Method:OPTIONS URL:http://hdp-hdb-200.gagan.com:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fhdp-hdb-200.gagan.com%40gagan.com&doAs=pos
tgres&user.name=yarn ErrorMsg:'Authentication required'

I have added the following proxyuser configuration in Ranger KMS as well:

hadoop.kms.proxyuser.postgres.users=*

hadoop.kms.proxyuser.postgres.hosts=*

hadoop.kms.proxyuser.yarn.users=*

hadoop.kms.proxyuser.yarn.hosts=*

The core-site.xml has the required proxyuser configuration as well:

hadoop.proxyuser.postgres.groups=*

hadoop.proxyuser.postgres.hosts=*
hadoop.proxyuser.yarn.groups=*
hadoop.proxyuser.yarn.hosts=*
But nothing seem to be working in this case here.
1 ACCEPTED SOLUTION

avatar
Rising Star

A colleague once ran into something similar on HDP 2.3.x (seen in this older version only), and did the following:

===

After we added the following properties to Ranger KMS (via Ambari, Ranger KMS Config tab) and restarted cluster services, it appears that HAWQ is running successfully on YARN.

hadoop.kms.proxyuser.rm.users=*

hadoop.kms.proxyuser.rm.hosts=*

This allowed us to get past the AUTHORIZATION error in Ranger KMS logs due to renewer rm user not being allowed as a proxy to get delegation token for accounts.

View solution in original post

3 REPLIES 3

avatar
Rising Star

A colleague once ran into something similar on HDP 2.3.x (seen in this older version only), and did the following:

===

After we added the following properties to Ranger KMS (via Ambari, Ranger KMS Config tab) and restarted cluster services, it appears that HAWQ is running successfully on YARN.

hadoop.kms.proxyuser.rm.users=*

hadoop.kms.proxyuser.rm.hosts=*

This allowed us to get past the AUTHORIZATION error in Ranger KMS logs due to renewer rm user not being allowed as a proxy to get delegation token for accounts.

avatar
Contributor

hadoop.kms.proxyuser.rm.users=*

hadoop.kms.proxyuser.rm.hosts=*

avatar
Contributor

That worked like a charm Vineet. Appreciate that tip. It worked for me as well.