Reply
Highlighted
Explorer
Posts: 17
Registered: ‎09-09-2014

KMS - AuthenticationToken ignored - Invalid signature - A009: HTTP status [403], message [Forbidden]

Dear Friends

 

Based on my original post from http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/Pig-script-in-Hue-START-RETRY-status/m-p/23161

and the last response (This is probably related to kms, as stated somewhere else, I hope they will be able to help about that!), I think the issue (Start_Retry halt status in Pig Editor in Hue) is due to a recent bug mentioned at https://issues.apache.org/jira/browse/HADOOP-11151


We have recently upgraded both CM and CDH to from 5.2 to 5.3 but the bug still exists. So seems it has not completely been fixed as the error messages mentioned in the URL above are exact the same as our error messages we got as below (highlighed in Bold).

 

Would you please let us know if my understanding is correct and if not, what how do you suggest to fix the issue. If yes, does anybody know if it is going to be fixed in the future release i.e. 5.3.1. 


Thanks much for you attention.

 

P.S. more on KMS: http://hadoop.apache.org/docs/current/hadoop-kms/index.html 

Kind regards
Andy

----------
from /var/log/hadoop-kms/kms.log

2015-01-01 16:25:46,866 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter:
AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
---------
from pig job's log in hue (from Oozie dahsboar/workflows --> log tab)

2015-01-01 16:25:46,875 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[our_name_node.com] USER[my_active_dir_user_name] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000012-141228125521164-oozie-oozi-W] ACTION[0000012-141228125521164-oozie-oozi-W@pig] Error starting action [pig]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: HTTP status [403], message [Forbidden]]
org.apache.oozie.action.ActionExecutorException: JA009: HTTP status [403], message [Forbidden]
...
Caused by: java.io.IOException: HTTP status [403], message [Forbidden]
at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:223)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:145)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:346)

at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:799)

at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)

Announcements
New solutions