rangerusersync user/password configuration issues

[#1]

HDP 2.3.4/Ambari 2.2.1.1

I am having some trouble understanding how the various Ranger users are used and where to configure the passwords. What I believe to be true (based on some other posts on Ranger in this forum), is:

1. Ranger admin user - I understand that the password for this has to be the same in Ranger Admin UI and under Ranger Advanced config for the field "admin_password" in the Advanced ranger-env configuration
2. Ranger amb_ranger_admin user - I understand that the password for this has to be the same in Ranger Admin UI and under Ranger Advanced config for the field "Ranger Admin username for Ambari" in the Advanced Ranger Admin configuration
3. Ranger rangerusersync user - This one seems to be my problem. It appears, based on the error trace below, that the password for this user needs to be configured somewhere but is not correct and I cannot figure out where I need to make changes.

Note that I have Ranger configured for Unix sync mode. Ranger admin and usersync processes are running on host api01 and my two HA name nodes are running on host nn01 and nn02.

So, what I am looking for is two things:

1. An explanation of how these three users are used to make Ranger operate
2. Should the usersync process be running on one of the name nodes instead of a different host? I believe that I only need to configure users and groups on the two name nodes and that other services, like Hive, rely on the equivalent of "hdfs groups <user>" to obtain group information and that that information comes from the Linux OS configured user/group information on the namenode. Since I am running usersync on a non-NN host, I am thinking this might be wrong.
3. Some help in solving the error related to rangerusersync, namely this one (full traces below):

03 May 2016 06:04:41  INFO PasswordValidator [Thread-8] - Response [FAILED: [rangerusersync] does not exists.] for user: rangerusersync

03 May 2016 06:04:09  INFO UnixAuthenticationService [main] - Starting User Sync Service!
03 May 2016 06:04:09  INFO UnixAuthenticationService [main] - Enabling Unix Auth Service!
03 May 2016 06:04:09  INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder
03 May 2016 06:04:10  WARN NativeCodeLoader [main] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
03 May 2016 06:04:10  INFO UnixAuthenticationService [main] - Enabling Protocol: [SSLv2Hello]
03 May 2016 06:04:10  INFO UnixAuthenticationService [main] - Enabling Protocol: [TLSv1]
03 May 2016 06:04:10  INFO UnixAuthenticationService [main] - Enabling Protocol: [TLSv1.1]
03 May 2016 06:04:10  INFO UnixAuthenticationService [main] - Enabling Protocol: [TLSv1.2]
03 May 2016 06:04:41  INFO PasswordValidator [Thread-8] - Response [FAILED: [rangerusersync] does not exists.] for user: rangerusersync
03 May 2016 06:04:41 ERROR UserGroupSync [UnixUserSyncThread] - Failed to initialize UserGroup source/sink. Will retry after 60000 milliseconds. Error details:
com.sun.jersey.api.client.UniformInterfaceException: GET http://api01.qa.quasar.local:6080/service/xusers/groups/?pageSize=1000&startIndex=0 returned a response status of 401 Unauthorized
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:686)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:507)
        at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildGroupList(PolicyMgrUserGroupBuilder.java:346)
        at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildUserGroupInfo(PolicyMgrUserGroupBuilder.java:156)
        at org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.init(PolicyMgrUserGroupBuilder.java:152)
        at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:51)
        at java.lang.Thread.run(Thread.java:745)

Thanks in advance...

[#2]

Hi @Mark Petronic :

What version of Ambari/HDP are you using?

To answer your questions about rangerusersync...

1] rangerusersync is an internal ranger user. See if you can login to Ranger UI using rangerusersync/rangerusersync (default). If this is not working, then you need to see if this password is changed. You can login as admin to verify and reset the password. Once password is changed for rangerusersync user, from usersync install dir (/usr/hdp/current/ranger-usersync) execute "python updatepolicymgrpassword.py" and provide the new password there.

2] It is not required to configure rangerusersync as it is an internal user and there is no requirement that it needs to be running on a particular host.

3] First error is harmless because ranger authentication goes through the normal flow and since you have configured unix authentication and rangerusersync does not exist as OS user, this error is shown and can be ignored. Second error means, usersync process is not able to communicate to ranger admin. See ranger admin logs also to get more clues. This could be because of wrong password (1 above) or any other misconfiguration. Is there an upgrade involved? In that case, make sure /etc/ranger/admin/conf/security-applicationContext.xml matches /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/classes/conf.dist/security-applicationContext.xml

Thanks,

[#3]

Thanks, I'm working through this but have a question still regarding my question #2. Can you please explain how usersync obtains the user/groups? Does it read /etc/passwd and /etc/group files from the node that the usersync process is running on or does it use some RPC call to the namenode to obtain that information from those same files that reside on the namenode? I believe that the ONLY place I need to create user and groups is on the namenode (or both namenodes in my case since I am running NN HA). I would just like to understand the process by which Ranger discovers the users and groups so that I can be sure where to create them in the first place. It's a bit of a mystery to me. That's why I asked if usersync should be running on the NNs or can it be running anywhere. You seem to state anywhere is ok.

[#4]

Oh, and I am running HDP 2.3.4/Ambari 2.2.1.1.

[#5]

For question#1: After the password is updated for rangerusersync user, restart ranger in order for the changes to take effect.

[#6]

Yes, if you are using unix sync, then usersync process needs to be running on the host where you have all the relevant users defined in /etc/passwd.

[#7]

Ok, I was thinking that must be the case. So, I decided to just remove Ranger and start over with a clean install. I used the Ambari REST APIs to remove the service. Then I dropped all the ranger users and databases from MySQL. However, when I try to reinstall Ranger via Ambari, it will not allow me to install on any node except one - the place where I first installed it - which is NOT the NN where I want to install it. Somehow, I believe Ambari has some left over config that is causing it to lock my choice to just one node. Both Ranger Admin and UserSync are forced to the same node. All the pick lists that you normally use to select a different node to install the service on are all disabled from selection. Any idea how to release Ranger to be installed where I want to?

[#8]

@yusaku - any thoughts?

[#9]

I spoke to two different HWX support engineers this week who are Ranger experts. I gained a bit more understanding on some points of confusion, so I thought I would pass this on in case anyone has the same questions:

• Usersync can be installed on any node, as @vperiasamy noted in his comment because, in Unix sync mode, it simply reads in the /etc/passwd and /etc/group information from the node where it is installed. In fact, per my discussion with the support guys, it is now clear that Ranger simply ONLY READS in the users and groups from the Unix settings to prepopulate the Ranger Admin DB more for "convenience" so that you have those names present in the UI to add to policies. If you add more users and groups, Usersync will pull them in. However, deleting users and groups are NOT reflected in the UI. I can understand why as that would require much more complex sychronization logic that is probably being avoided for the sake of simplicity in design. They said there are scripts you can run manually to remove users and groups. The same is true if configured for AD/LDAP. It ONLY READS in users that appear which your search specifications against your domain of users.
• The Linux users and groups should be defined on all cluster nodes so that the user/group names and UIDs and GIDs are all the same. However, I have some doubts that you really need to do that on the data nodes. I can see this helping on the master nodes where clients are running. The support guys said that, generally speaking, having this all defined ONLY on the name nodes "should" be sufficient for most Hadoop applications to work but indicated there are "corner cases" where some application process might look at the local nodes user/group definitions and, if they are out of sync with, other node, could result in strange behavior. So they recommend syncing up those settings on all nodes to be safe.

[#10]

When using Kerberos and running YARN with the LinuxContainerExecutor, it is required to have all users (who wish to run YARN applications) on all nodes!