NN / HS2 /HMaster not starting after Ambari upgrade : "Ambari admin username and password are blank"

[#1]

Hi,

after upgrading Ambari from 2.0.1 to 2.1.2.1 on HDP2.2.4 the services with enabled Ranger plugins are not starting.

Error message is:

2016-04-22 17:23:39,688 - HTTP Code: 401
2016-04-22 17:23:39,688 - HTTP Data: {"statusCode":401,"msgDesc":"Authentication Failed"}
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 317, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 484, in restart
    self.start(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 82, in start
    namenode(action="start", rolling_restart=rolling_restart, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 50, in namenode
    setup_ranger_hdfs(rolling_upgrade = rolling_restart)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/setup_ranger_hdfs.py", line 51, in setup_ranger_hdfs
    hdp_version_override = hdp_version)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/setup_ranger_plugin.py", line 73, in setup_ranger_plugin
    policy_user)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 108, in create_ranger_repository
    raise Fail('Ambari admin username and password are blank ') 

resource_management.core.exceptions.Fail: Ambari admin username and password are blank

If I disable the Ranger-plugin the services are starting up, but enabling the plugin results in the same error at service restart. Before the Ambari upgrade the services were running incl. Ranger-plugin enabled.

What happened here?!?!

[#2]

@Gerd Koenig

Please refer

https://community.hortonworks.com/questions/10015/namenode-unable-to-start-ranger-issue-in-namenode....

https://community.hortonworks.com/questions/14329/hiveserver2-wont-start-after-changing-rangers-admi...

[#3]

Hi @Kuldeep Kulkarni , thanks for the URLs, they are quite interesting.

Unfortunately my error message is nothing with JSON decode, but

resource_management.core.exceptions.Fail:Ambari admin username and password are blank

I update all the admin passwords to be the same, for sure, but the problem remains.

I updated the following (Ambari 2.1.2.1) properties:

Ambari => Ranger => Config: Ranger Admin user's password for Ambari

Ambari => Ranger => Config: admin_password

Ranger WebUI => password of user admin

What I don't understand is the error message, which properties exactly shall be blank ?!?!

[#4]

@Gerd Koenig - Thank you for your reply, can you please check answer given by @hkropp

[#5]

To me it looks like in the Ranger admin config your

ambari_ranger_admin

and

ambari_ranger_password

are empty. Could you please check if they exists and are not blank?

Please note, that the ambari_ranger_admin is different from the Ranger admin_user. But both are defined in the Ranger configuration in Ambari.

[#6]

@Gerd Koenig

[#7]

Hi @hkropp , thanks for this hint.

I cross-checked with another cluster (same HDP-, same Ambari version) where Ranger is working as expected. The above mentioned properties are missing in both. I checked this doc to verify which user settings belongs to whom (I have to admit, it is a bit confusing 😉 ).

The properties in Ambari - Ranger - Config looks like

Therefore the users and password are in place (this config looks the same in another cluster where Ranger-plugins are working). The only thing I noticed is, that in Ranger-Web-Admin the user 'amb_ranger_admin' was missing.

I created that user as 'new user' in Ranger Web UI with the same password as in Ambari.

Unfortunately that didn't solve the problem, restarting HDFS/Hive/HBase still results in the same error message

...kind of guess-less right now. Maybe I will drop service Ranger and re-install it, since the cluster is not yet in "real use"

Any further hints..?!?!

[#8 · Accepted solution]

Hi,

just as a "cleanup" of this thread, at the end I didn't figure out what was the real error, but after re-installing Ranger (incl. dropping db 'ranger') now it works.

Sorry, not really a solution, but I just need a working env. asap 😉

Regards...

[#9]

A few days ago, I happened to run into this issue myself. The root cause is, when installing Ranger, in the "External URL" property, the administrator entered "http://hostname.example.com:6080/", instead of the expected "http://hostname.example.com:6080" (WITHOUT the trailing slash character). Even though the Ranger installation would go through, Ranger's Usersync would log errors in /var/log/ranger/usersync/usersync.log due to this extraneous character. Also, any attempt to enable any of Ranger's plugins would fail, with the error message "Ambari admin username and password are blank", because Ranger is indeed missing many users, including the important one amb_ranger_admin. To fix this, just edit this property and remove any character after port 6080, and everything will start working.