question Re: HiveServer2 won't start after changing Ranger's admin password in Archives of Support Questions (Read Only)

HiveServer2 won't start after changing Ranger's admin password

Adija1 — Thu, 04 Feb 2016 15:38:01 GMT

Hello

HDP 2.3.2

Ambari 2.1.2.1

After changing the admin password of Ranger - both in the ranger UI and in the ambari under Advanced ranger-env the HiveServer2 fails to start. Ranger starts flawlessly - but HiveServer2 (which has Ranger authorization enabled) doesn't. If i disable the Ranger authorization from Hive (switching to none) HiveServer2 will start.

The error in the ambari is:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 193, in <module>
    HiveServer().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 85, in start
    setup_ranger_hive(rolling_upgrade=rolling_restart)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive.py", line 50, in setup_ranger_hive
    hdp_version_override = hdp_version)                 
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/setup_ranger_plugin_xml.py", line 78, in setup_ranger_plugin
    policy_user)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 92, in create_ranger_repository
    repo = self.get_repository_by_name_urllib2(repo_name, component, 'true', ambari_username_password_for_ranger)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", line 57, in get_repository_by_name_urllib2
    response = json.loads(result.read())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
    raise ValueError("No JSON object could be decoded")

I've noticed there's a registered bug around this issue but it is unassigned and unresolved -https://issues.apache.org/jira/browse/AMBARI-13168

Any ideas how to workaround this issue ??

Re: HiveServer2 won't start after changing Ranger's admin password

rmani — Thu, 04 Feb 2016 17:02:05 GMT

Did you reset the password in Ranger config in Ambari after ranger admin password change? Ranger property in Advanced ranger-env -> admin_username and admin_password needs to be changed after reseting the Ranger admin password

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Thu, 04 Feb 2016 17:24:42 GMT

Hi rmani and thank you for replying.

As stated in my question - yes i did reset the password in Ranger UI >>>admin >> profile >> change password and also in Ambari >>>> Ranger >>>Advance ranger-env >>>> admin_password.

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Sun, 07 Feb 2016 22:40:45 GMT

I have simulated this issue in a different cluster and it is persistent - after changing Ranger's admin's password - in ranger UI & in Ambari >>> Ranger >> Advanced ranger-env - HiveServer2 won't start.

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Sun, 07 Feb 2016 22:57:42 GMT

@Adi Jabkowsky

Changing the password process is sensitive as per my experience

See the following doc.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_Ranger_Install_Guide/content/updating_ranger_admin_passwords.html

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Sun, 07 Feb 2016 23:16:41 GMT

Hi Neeraj and thank you for your reply !

I've went through this doc and did set the password in both of these places in Ambari >> Ranger. (Both under "Admin Settings" >>> Ranger Admin user's password for Ambari" & under Advanced ranger-env >>> admin password).However - problem remains.

It is very easy to reproduce - just change Ranger's admin password in UI + both places in Ambari and see for yourself. HiveServer2, if configured to authorize using Ranger, won't start. It has been confirmed and consistent on two different clusters.

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Sun, 07 Feb 2016 23:16:51 GMT

This bug prohibits us from using Ranger which is a great tool for permission governance.

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Sun, 07 Feb 2016 23:31:47 GMT

@Adi Jabkowsky I have experienced the same issue and to fix that I had to reinstall Ranger.

Do you have access to support?

Let me reproduce this

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Sun, 07 Feb 2016 23:40:17 GMT

I appreciate your offer but no support agreement yet unfortunately. We are still in (advanced) POC stages...

Thanks a lot anyways !

Highly appreciated.

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Mon, 19 Aug 2019 10:07:47 GMT

@Adi Jabkowsky

I changed admin user password for ranger in Ranger UI

Logged into the ambari and updated only following entry with new password

restarted Namenode ...No issues

log http://pastebin.com/vYDY8QtE

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Sun, 07 Feb 2016 23:58:08 GMT

@Adi Jabkowsky FYI I changed password only in 1 location

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Mon, 19 Aug 2019 10:07:39 GMT

I was referring to HiveServer2 - not Namenode. In my scenario the Ranger is managing Hive authorization. So unless Hive isn't set to use Ranger - you won't have this issue.

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Mon, 08 Feb 2016 00:07:38 GMT

@Adi Jabkowsky Did you update password in one location or 2 locations as described in official docs?

I changed the password entry only for admin setting as shared in my screenshot.

Restarted Hive process using ambari and no issues with Ranger auth

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Mon, 08 Feb 2016 00:08:02 GMT

I updated the 2 locations as described in official docs. Not just one.

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Mon, 08 Feb 2016 00:10:33 GMT

@Adi Jabkowsky That's the issue.

1st one is for Ranger UI admin password

2nd one is for Ambari admin user entry 😉 Rollback the 2nd password change (Put ambari admin password there)

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Mon, 08 Feb 2016 00:40:30 GMT

Please check if Hive is set for Ranger (Under Security just like in my screenshot). It might be set to NONE or SQLstdAuth

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Mon, 19 Aug 2019 10:07:31 GMT

@Adi Jabkowsky No, its Ranger

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Mon, 08 Feb 2016 00:47:24 GMT

@Adi Jabkowsky See this thread too https://community.hortonworks.com/questions/1929/changing-ranger-ui-password-causing-namenode-issue.html

Re: HiveServer2 won't start after changing Ranger's admin password

Adija1 — Mon, 08 Feb 2016 00:59:04 GMT

Tried every combination including reverting back to the default admin password in Ranger - still no luck. HiveServer2 won't start as long as it it's Auth are configured using Ranger. "No JSON object could be decoded"

😞

Thx a million for trying !

Re: HiveServer2 won't start after changing Ranger's admin password

nsabharwal — Mon, 08 Feb 2016 01:03:15 GMT

@Adi Jabkowsky Is this the error? http://pastebin.com/zc3iHYMu