28725
DISCUSSIONS
101723
MEMBERS
3157
ARTICLES
Created 01-01-2015 05:38 PM
Dear Friends
Based on my original post from http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/Pig-script-in-Hue-START-RETRY-status/m-p/23161
and the last response (This is probably related to kms, as stated somewhere else, I hope they will be able to help about that!), I think the issue (Start_Retry halt status in Pig Editor in Hue) is due to a recent bug mentioned at https://issues.apache.org/jira/browse/HADOOP-11151
We have recently upgraded both CM and CDH to from 5.2 to 5.3 but the bug still exists. So seems it has not completely been fixed as the error messages mentioned in the URL above are exact the same as our error messages we got as below (highlighed in Bold).
Would you please let us know if my understanding is correct and if not, what how do you suggest to fix the issue. If yes, does anybody know if it is going to be fixed in the future release i.e. 5.3.1.
Thanks much for you attention.
P.S. more on KMS: http://hadoop.apache.org/docs/current/hadoop-kms/index.html
Kind regards
Andy
----------
from /var/log/hadoop-kms/kms.log
2015-01-01 16:25:46,866 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter:
AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
---------
from pig job's log in hue (from Oozie dahsboar/workflows --> log tab)
2015-01-01 16:25:46,875 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[our_name_node.com] USER[my_active_dir_user_name] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000012-141228125521164-oozie-oozi-W] ACTION[0000012-141228125521164-oozie-oozi-W@pig] Error starting action [pig]. ErrorType [TRANSIENT], ErrorCode [JA009], Message [JA009: HTTP status [403], message [Forbidden]]
org.apache.oozie.action.ActionExecutorException: JA009: HTTP status [403], message [Forbidden]
...
Caused by: java.io.IOException: HTTP status [403], message [Forbidden]
at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:223)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:145)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:346)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:799)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)