HDFS service failing with python error

[#1]

SYMPTOM: HDFS service is not able to start and throwing below error in logs -

python error " File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 124, in create_ranger_repository"

ERROR: Ambari operation logs shows below message -

stderr:   /var/lib/ambari-agent/data/errors-22280.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 433, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 524, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 102, in start
    namenode(action="start", hdfs_binary=hdfs_binary, upgrade_type=upgrade_type, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 60, in namenode
    setup_ranger_hdfs(upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py", line 61, in setup_ranger_hdfs
    hdp_version_override = hdp_version, skip_if_rangeradmin_down= not params.retryAble)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/setup_ranger_plugin_xml.py", line 78, in setup_ranger_plugin
    policy_user)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 124, in create_ranger_repository
    repo = self.get_repository_by_name_urllib2(repo_name, component, 'true', ambari_username_password_for_ranger)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 82, in wrapper
    return function(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 77, in get_repository_by_name_urllib2
    response = json.loads(result.read())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

ROOT CAUSE: This is due to issue with "amb_ranger_admin" user password and hence ranger service are not able to communicate with Ranger admin.

RESOLUTION: Below were steps performed for resolution -

1. Tried disabling the HDFS plugin for ranger and restarting HDFS worked well. 
2. Removed the HDFS repository policy cache files from both namenodes. 
3. Enabled HDFS plugin and restarted standby namenode which got failed again with same error. 
4. Checked in Ranger UI in Audit -> Access->login tab was displaying wrong credentials for ambari admin user 
5. We Reset the password for amb_ranger_admin from Ranger UI also modified same value in Ambari -> Services->Ranger->Configs 
6. Restarted ranger