Ranger User sync not finding files/permission denied

[#1]

Hello,

While setting up Ranger on our Kerberized cluster (HDP 2.5.0.0, Ranger 0.6.0), I am seeing the user sync is not working. When looking at the logs I am seeing the following error message:

21 Oct 2016 00:09:05 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder 21 Oct 2016 00:09:05 INFO PolicyMgrUserGroupBuilder [UnixUserSyncThread] - Using principal = rangerusersync/popul-vmmn01.inetuhosted.net@AD.POPULYTICS.COM and keytab = /etc/security/keytabs/rangerusersync.service.keytab 21 Oct 2016 00:09:05 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 21 Oct 2016 00:09:05 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 21 Oct 2016 00:09:05 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder created 21 Oct 2016 00:09:05 INFO UserGroupSync [UnixUserSyncThread] - initializing source: org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder 21 Oct 2016 00:09:05 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization started 21 Oct 2016 00:09:05 WARN FSInputChecker [UnixUserSyncThread] - Problem opening checksum file: file:/usr/hdp/current/ranger-usersync/conf/ugsync.jceks. Ignoring exception: java.io.FileNotFoundException: /usr/hdp/current/ranger-usersync/conf/.ugsync.jceks.crc (Permission denied) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.<init>(RawLocalFileSystem.java:111) at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:215) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:152) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:348) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:782) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.getInputStreamForFile(JavaKeyStoreProvider.java:70) at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.<init>(AbstractJavaKeyStoreProvider.java:107) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:49) at org.apache.hadoop.security.alias.JavaKeyStoreProvider.<init>(JavaKeyStoreProvider.java:41) at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:58) at org.apache.ranger.credentialapi.CredentialReader.getDecryptedString(CredentialReader.java:59) at org.apache.ranger.unixusersync.config.UserGroupSyncConfig.getLdapBindPassword(UserGroupSyncConfig.java:541) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.setConfig(LdapUserGroupBuilder.java:174) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.init(LdapUserGroupBuilder.java:135) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:55) at java.lang.Thread.run(Thread.java:745)

I am sure I missing some step in the install process, but I am not sure quite what it is. Any help would be greatly appreciated.

Thanks,

Nick

[#2]

@ Nick Pileggi, can you please check if ranger-usersync process is up and Ldap/AD users got synced on Ranger UI. The file not found is just a warning which will not block the process to stop.

If possible can you please attach log file of ugsync.log and xa_portal.log

[#3]

@ Nick Pileggi this warn you can ignore , it is known , can you please check if there is any other error , please let us know what is the source of user sync?

[#4]

Thanks for the quick reply. You both were right, there is another exception right below it:

21 Oct 2016 00:09:05 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldaps://popul-abead01.ad.populytics.com:636, ldapBindDn: CN=Hadoop Bind,OU=Service Accounts,OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, userSearchBase: [OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com], userSearchScope: 2, userObjectClass: user, userSearchFilter: , extendedUserSearchFilter: (objectclass=user), userNameAttribute: sAMAccountName, userSearchAttributes: [sAMAccountName], userGroupNameAttributeSet: null, pagedResultsEnab 21 Oct 2016 00:09:05 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 21 Oct 2016 00:09:05 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 21 Oct 2016 00:09:05 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Performing user search first 21 Oct 2016 00:09:05 ERROR CustomSSLSocketFactory [UnixUserSyncThread] - Unable to obtain keystore from file [/usr/hdp/current/ranger-usersync/conf/my$ javax.naming.CommunicationException: popul-abead01.ad.populytics.com:636 [Root exception is java.lang.NullPointerException] at com.sun.jndi.ldap.Connection.<init>(Connection.java:216) at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1614) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.createLdapContext(LdapUserGroupBuilder.java:147) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.getUsers(LdapUserGroupBuilder.java:377) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:302) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.ranger.ldapusersync.process.CustomSSLSocketFactory.createSocket(CustomSSLSocketFactory.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jndi.ldap.Connection.createSocket(Connection.java:328) at com.sun.jndi.ldap.Connection.<init>(Connection.java:203) ... 17 more

The source of the user sync is Active Directory.

Let me know if I can provide any other details.

Thanks,

Nick

[#5]

it seems userGroupNameAttributeSet is null(in line 7 of provided log : userGroupNameAttributeSet: null). Please review the value of property "ranger.usersync.ldap.user.groupnameattribute"

Also, does your LDAP server allows SSL connection without a certificate ?

[#6]

@PradeeP AgrawaL

I just double checked my property and it is there. I restarted the service and now have the log below:

25 Oct 2016 11:19:32 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldaps://popul-abead01.ad.populytics.com:636, ldapBindDn: CN=Hadoop Bind,OU=Service Accounts,OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, userSearchBase: [OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com], userSearchScope: 2, userObjectClass: user, userSearchFilter: , extendedUserSearchFilter: (objectclass=user), userNameAttribute: sAMAccountName, userSearchAttributes: [sAMAccountName, memberof, ismemberof], userGroupNameAttributeSet: [memberof, ismemberof], pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: false, groupSearchBase: [OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com], groupSearchScope: 2, groupObjectClass: group, groupSearchFilter: member=cn=(0),OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, extendedGroupSearchFilter: (&(objectclass=group)(member=cn=(0),OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com)(|(member={0})(member={1}))), extendedAllGroupsSearchFilter: (&(objectclass=group)(member=cn=(0),OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com)), groupMemberAttributeName: member, groupNameAttribute: cn, groupSearchAttributes: [member, cn], groupUserMapSyncEnabled: true, groupSearchFirstEnabled: false, userSearchEnabled: false, ldapReferral: ignore 25 Oct 2016 11:19:32 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 25 Oct 2016 11:19:32 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 25 Oct 2016 11:19:32 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Performing user search first 25 Oct 2016 11:19:32 ERROR CustomSSLSocketFactory [UnixUserSyncThread] - Unable to obtain keystore from file [/usr/hdp/current/ranger-usersync/conf/mytruststore.jks] 25 Oct 2016 11:19:32 ERROR UserGroupSync [UnixUserSyncThread] - Failed to initialize UserGroup source/sink. Will retry after 3600000 milliseconds. Error details: javax.naming.CommunicationException: popul-abead01.ad.populytics.com:636 [Root exception is java.lang.NullPointerException] at com.sun.jndi.ldap.Connection.<init>(Connection.java:216) at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137) at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1614) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.createLdapContext(LdapUserGroupBuilder.java:147) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.getUsers(LdapUserGroupBuilder.java:377) at org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder.updateSink(LdapUserGroupBuilder.java:302) at org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:58) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.ranger.ldapusersync.process.CustomSSLSocketFactory.createSocket(CustomSSLSocketFactory.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jndi.ldap.Connection.createSocket(Connection.java:328) at com.sun.jndi.ldap.Connection.<init>(Connection.java:203) ... 17 more

So that looks correct there. To your point, my AD does require a certificate. I have installed our CA certificate as a trust root certificate, but as I am thinking about this, do I also need to add my CA or AD cert into the ranger truststore?

Nick

[#7]

OK, I did screw that up a bit, by following Ancil's answer here: https://community.hortonworks.com/questions/1018/how-to-configure-ranger-usync-for-ldap-ssl.html

I have imported my CA and my AD servers' certificates into the java trust store and changed ranger to look at that trust store. At this point it feels so close to being correct. Here is the updated error message. It looks like a certificate issue, but I am not sure:

25 Oct 2016 15:36:43 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with -- ldapUrl: ldaps://popul-abead01.ad.populytics.com:636, ldapBindDn: CN=Hadoop Bind,OU=Service Accounts,OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, ldapBindPassword: ***** , ldapAuthenticationMechanism: simple, searchBase: OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, userSearchBase: [OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com], userSearchScope: 2, userObjectClass: user, userSearchFilter: , extendedUserSearchFilter: (objectclass=user), userNameAttribute: sAMAccountName, userSearchAttributes: [sAMAccountName, memberof, ismemberof], userGroupNameAttributeSet: [memberof, ismemberof], pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled: false, groupSearchBase: [OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com], groupSearchScope: 2, groupObjectClass: group, groupSearchFilter: member=cn=(0),OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com, extendedGroupSearchFilter: (&(objectclass=group)(member=cn=(0),OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com)(|(member={0})(member={1}))), extendedAllGroupsSearchFilter: (&(objectclass=group)(member=cn=(0),OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com)), groupMemberAttributeName: member, groupNameAttribute: cn, groupSearchAttributes: [member, cn], groupUserMapSyncEnabled: true, groupSearchFirstEnabled: false, userSearchEnabled: false, ldapReferral: ignore 25 Oct 2016 15:36:43 INFO UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 25 Oct 2016 15:36:43 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder updateSink started 25 Oct 2016 15:36:43 INFO LdapUserGroupBuilder [UnixUserSyncThread] - Performing user search first 25 Oct 2016 15:36:44 ERROR LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() failed with exception: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C0903D9, comment: AcceptSecurityContext error, data 775, v2580]; remaining name 'OU=Healthcare Analytics,DC=ad,DC=populytics,DC=com' 25 Oct 2016 15:36:44 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LDAPUserGroupBuilder.getUsers() user count: 0 25 Oct 2016 15:36:44 INFO UserGroupSync [UnixUserSyncThread] - End: initial load of user/group from source==>sink 25 Oct 2016 15:36:44 INFO UserGroupSync [UnixUserSyncThread] - Done initializing user/group source and sink

Thanks,

Nick

[#8]

Are you able to connect to your ldaps server from any ldap client tool ?

[#9 · Accepted solution]

Well I completely screwed this up. After that previous step it was working except my AD bind user had been locked out by my AD policy. So I completely started over by removing ranger and reinstalling it. After that I looked at Ancil's guide for setting up the trust store and when I restarted ranger it synced all of the users.

Thanks everyone for pointing me toward the solution.

Nick