Ranger Admin Not Starting After Ambari Upgrade

[#1]

I am getting below errors while trying to start Ranger Admin after the latest Ambari upgrade from 2.5.1 to 2.5.2:

2017-10-08 17:30:03,646 - Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
java.sql.SQLException: Access denied for user 'rangerdba'@'scregionm1.field.hortonworks.com' (using password: YES)

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 531, in <module>
    CheckHost().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/custom_actions/scripts/check_host.py", line 206, in actionexecute
    raise Fail(error_message)
resource_management.core.exceptions.Fail: Check db_connection_check was unsuccessful. Exit code: 1. Message: ERROR: Unable to connect to the DB. Please check DB connection properties.
java.sql.SQLException: Access denied for user 'rangerdba'@'scregionm1.field.hortonworks.com' (using password: YES)

I confirmed 'rangerdba' user has proper privileges on the MySQL node:

mysql> SELECT host,user,password,Grant_priv,Super_priv FROM mysql.user; +----------------------------------+-------------+-------------------------------------------+------------+------------+ | host | user | password | Grant_priv | Super_priv | +----------------------------------+-------------+-------------------------------------------+------------+------------+ | localhost | root | | Y | Y | | scregionm2.field.hortonworks.com | root | | Y | Y | | 127.0.0.1 | root | | Y | Y | | ::1 | root | | Y | Y | | % | rangerdba | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | Y | Y | | % | hive | *1C2AF9445ECD574DB1D3E4584424EDFB768C1E70 | N | Y | | localhost | rangerdba | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | Y | Y | | scregionm2.field.hortonworks.com | rangerdba | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | Y | Y | | % | rangerkms | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | N | N | | localhost | rangerkms | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | N | N | | scregionm2.field.hortonworks.com | rangerkms | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | N | N | | % | rangeradmin | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | N | N | | localhost | rangeradmin | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | N | N | | scregionm2.field.hortonworks.com | rangeradmin | *0DFC06DA054FDC6F974A6709CFCE366817B59894 | N | N | | % | streamline | *46ADB7217AF6CE7227CD73C4F4AD1C157D664E7F | N | N | | ambariservernode | rangeruser | | Y | Y | | scregionm1.field.hortonworks.com | rangerdba | | Y | Y | | scregionm0.field.hortonworks.com | rangerdba | | Y | Y | +----------------------------------+-------------+-------------------------------------------+------------+------------+ 18 rows in set (0.00 sec)

I also checked MySQL connection from the Ranger Admin node to the MySQL instance is fine:

sqlline> !connect jdbc:mysql://scregionm2.field.hortonworks.com/mysql rangerdba BadPass#1 Connecting to jdbc:mysql://scregionm2.field.hortonworks.com/mysql Connected to: MySQL (version 5.6.35) Driver: MySQL-AB JDBC Driver (version mysql-connector-java-5.1.17-SNAPSHOT ( Revision: ${bzr.revision-id} )) Autocommit status: true Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:mysql://scregionm2.field.hortonworks.> select version(); +------------+ | version() | +------------+ | 5.6.35 | +------------+ 1 row selected (0.031 seconds)

Any help is appreciated. Thanks.

[#2]

@dsun

It looks your ranger database is up and running correctly, can you validate that the in Ambari UI-->Ranger-->Configs--->Ranger Admin that the test connection works especially check the JDBC connect string for a Ranger database and the Ranger DB password.

Please let me know if you still encounter problems.

[#3]

@Geoffrey Shelton Okot Forgot to mention in my original post, 'test connection' in Ambari was the first task I checked, and it failed, that's why I started digging further with above testing.

[#4]

@dsun

Did you create a dump of the ranger batabase before the upgrade?

If not then create one,while looged in as mysql root user

mysqldump ranger > ranger_db.sql 

It seems the below statement will fail as your current password is wrong.

mysql -u rangerdba -p{password} 

Try the alternative to change the rangerdba password while logged on as mysql root user, below is an example

update user set password=PASSWORD('your_new_password') where User='user-name-here'; 
flush privileges;

Then use the new credentials in Ambari UI-->Ranger-->Configs--->Ranger Admin to test connection verify that it works

Hope that helps

[#5]

thanks @Geoffrey Shelton Okot, but that didn't help, still seeing the same error, I assume somewhere along the upgrade went wrong, and I am checking the logs:

2017-10-08 19:37:04,316  [JISQL] /usr/jdk64/jdk1.8.0_77/bin/java  -cp /usr/hdp/current/ranger-admin/ews/lib/mysql-jdbc-driver.jar:/usr/hdp/current/ranger-admin/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://scregionm2.field.hortonworks.com/mysql -u rangerdba -p '********' -noheader -trim -c \; -query "SELECT version();"
SQLException : SQL state: 28000 java.sql.SQLException: Access denied for user 'rangerdba'@'scregionm1.field.hortonworks.com' (using password: YES) ErrorCode: 1045
2017-10-08 19:37:04,893  [E] Can't establish db connection.. Exiting..

[#6 · Accepted solution]

@Geoffrey Shelton Okot I ran 'hdp-select' on the host where Ranger Admin is installed just to check the components are running and noticed it's showing the older version 2.6.1.0-129:

[root@scregionm1 ~]# hdp-select | grep ranger ranger-admin - 2.6.1.0-129 ranger-kms - None ranger-tagsync - 2.6.1.0-129 ranger-usersync - 2.6.1.0-129

I went ahead manually updated the components version by running 'hdp-select set all 2.6.2.0-205', then restarted Ranger service, not it's working. As I expected, something went wrong during the upgrade which is still a mystery.

Thanks.

[#7]

@dsun

I think nothing is wrong with the upgrade please run these 2 statements and it should resolve the problem

grant all privileges on *.* to 'rangerdba'@'scregionm1.field.hortonworks.com' identified by '{ranger_password}';
grant all privileges on *.* to 'rangerdba'@'localhost';

Please revert

[#8]

Thanks @Geoffrey Shelton Okot, all privileges were granted to rangerdba properly as you can see from my original post. It's related to the upgrade, as Ambari is using the mysql-jdbc-driver under /usr/hdp/2.6.2.0-205/ranger-admin/ews/lib/ while everything else was running the 2.6.1 bits for some reason. The upgrade is stuck on some hosts, and I need to figure out why which is another issue I'm trying to resolve.