Could you please tell me which user you are using while firing this command ? If this is some user other then hdfs then it will not work. Try using hdfs and fire this command. I was also facing this issue but then after firing this command using hdfs user, it worked like a charm. ... View more

Some more info on this: The node manage logs shows that it is trying to enable log aggregation and download resources uploaded when the spark job was submitted. I believe that the node manager does not have a valid HDFS delegation token. How is the token transmitted from the spark submit job to the node managers? Thanks ... View more

Sorry I was trying to get clarification for this *second* issue, as I was also hitting this same exact scenario. For others in the future, the response by @Josh Elser that is upvoted at the top (when sorted by Votes) also worked for me to correct this java.net.SocketTimeoutException when connecting to PQS. I was missing the PQS host in hadoop.proxyuser.HTTP.hosts. I didn't realize the upvoted response was for this second issue because the comment sorting was showing things out of order for me. I never did track down an impersonation error message, but I didn't increase tracing at all to try real hard at capturing the error. ... View more