Upgraded HDP stack from 2.3.2 to 2.4.2. webhdfs curl failed to obtain user group information: java.io.IOException: Usernames not matched: name=hdfs != expected=hdfs-

[#1]

HI,

I recently upgraded from HDP 2.3.2 to HDP 2.4.2 on a secure cluster.

When I try to start MR History server, yarn Timeline service and HBase master service, all three are throwing the same error as

failed to obtain user group information: java.io.IOException: Usernames not matched: name=hdfs != expected=hdfs-REALM
Below is the stacktrace:

 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 195, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://znlhacdt0002.amer.zurich.corp:50070/webhdfs/v1/app-logs?op=GETFILESTATUS&user.name=hdfs'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "SecurityException", 
    "javaClassName": "java.lang.SecurityException", 
    "message": "Failed to obtain user group information: java.io.IOException: Usernames not matched: name=hdfs != expected=hdfs-SANDBOX"
  }
}

Not sure why the user.name its using hdfs-sandox instead hdfs. When I tried the curl, it works with hdfs-sandbox, but doesnt work with hdfs. Please let me know where I have to change the config to make it work..
Any help is greatly appreciated.
thanks,
Sreeram

[#2 · Accepted solution]

@Sreeram Chintalapudi This could happen if there is an incorrect RULE in "hadoop.security.auth_to_local" in /etc/hadoop/conf/core-site.xml. You can review/modify the contents by navigating to Ambari -> HDFS -> Configs -> Advanced -> Advanced core-site.xml -> hadoop.security.auth_to_local.

It should look similar to this;

RULE:[1:$1@$0]([email protected])s/.*/ambari-qa/
RULE:[1:$1@$0]([email protected])s/.*/hbase/ 
RULE:[1:$1@$0]([email protected])s/.*/hdfs/ 
RULE:[1:$1@$0]([email protected])s/.*/spark/ 
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// 
RULE:[1:$1@$0](.*@.*EXAMPLE.COM)s/@.*// 
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*// 
RULE:[1:$1@$0](.*@EXAMPLE.COM)s/@.*///L 
RULE:[2:$1@$0]([email protected])s/.*/ams/ 
RULE:[2:$1@$0]([email protected])s/.*/ams/ 
RULE:[2:$1@$0]([email protected])s/.*/atlas/ 
RULE:[2:$1@$0]([email protected])s/.*/hdfs/ 
RULE:[2:$1@$0]([email protected])s/.*/falcon/ 
RULE:[2:$1@$0]([email protected])s/.*/hbase/ 
RULE:[2:$1@$0]([email protected])s/.*/hive/ 
RULE:[2:$1@$0]([email protected])s/.*/mapred/ 
RULE:[2:$1@$0]([email protected])s/.*/hdfs/ 
RULE:[2:$1@$0]([email protected])s/.*/knox/ 
RULE:[2:$1@$0]([email protected])s/.*/yarn/ 
RULE:[2:$1@$0]([email protected])s/.*/hdfs/ 
RULE:[2:$1@$0]([email protected])s/.*/oozie/ 
RULE:[2:$1@$0]([email protected])s/.*/yarn/ 
RULE:[2:$1@$0]([email protected])s/.*/yarn/ 
DEFAULT

If you make any changes to the rules you will need to restart the affected services.

Hope this helps,

Steve

[#3]

Thanks Steve for the article..that did the trick...thanks again for all ur help

Sreeram

[#4]

Hi Sreeram,

I am also facing the same issue. Can you please tell me exactly what you did to solve the issue.

Thanks in advance