Ranger AD users cannot log in to UI after HDP 2.5 upgrade

[#1]

So, with 2.4.x, I had it *all* working. Active directory users could log in tot he Ranger web UI, and I also had ranger usersync working against AD as well.

Today I updated to HDP 2.5 and while the usersync seems to work (woohoo!) , I cannot for the life of me successfully log into the ranger web UI with an active-directory user.

Making matters worse, while I have tried to enable ranger debugging, I get no additional information in the ranger xa log to help tell me what's going on.

From: /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/log4j.xml:

<category name="org.apache.ranger" additivity="false"> <priority value="debug" /> <appender-ref ref="xa_log_appender" /> </category>

yet, even with debugging enabled, all I see from the logs is not super helpful: (and yes, the id and pw are correct...)

2016-11-04 14:52:54,738 [http-bio-6080-exec-1] INFO org.apache.ranger.security.listener.SpringEventListener (SpringEventListener.java:87) - Login Unsuccessful:kbrodie | Ip Address:xxx.xxx.xxx.xxx | Bad Credentials

The last time I saw this type of error, it was when I was originally setting up ranger some time ago (under 2.4), and was having ssl issues with my imported certificate. Enabling debugging helped figure that out. But now... I'm not getting any more useful info when debugging is enabled.... I'm stuck.

any ideas? Thank you much in advance.

[#2]

Hello @Kent Brodie,

Please try to enable Ranger debug logging via Ambari. Go to Ranger > Configs > Advanced > Advanced admin-log4j section. And change the rootlogger level from warn to debug. Restart Ranger Admin and that should give debug log in xa_portal.log. Let us know what error / stack trace you see there.

Best of luck !

[#3]

OK, so after some effort I have the debugging working, and I'm confused. The error it's giving me is a 'trustanchors' error, which (after a wee bit of googling) usually indicates that it can't find or access the trustsstore (aka cacerts).

Normal LDAP works fine, secure (SSL) ldap is now broken. (worked with HDP2.4..).

In my case, the cacerts is the same one that was there before the upgrade..... further, permissions are all good, and I have verified the key we use in there is still present. I only have one java on the server.

Any hints would be appreciated. I'm missing something silly.

Caused by: org.springframework.ldap.CommunicationException: simple bind failed: mydomain.mycompany.net:636; nested exception is javax.naming.CommunicationException: simple bind failed: mydomain.mycompany.net:636 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty] at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:100) at org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:266) at org.springframework.ldap.core.support.AbstractContextSource.getContext(AbstractContextSource.java:106) at org.springframework.ldap.core.support.AbstractContextSource.getReadOnlyContext(AbstractContextSource.java:125) at org.springframework.ldap.core.LdapTemplate.executeReadOnly(LdapTemplate.java:792) at org.springframework.security.ldap.SpringSecurityLdapTemplate.searchForSingleEntry(SpringSecurityLdapTemplate.java:196) at org.springframework.security.ldap.search.FilterBasedLdapUserSearch.searchForUser(FilterBasedLdapUserSearch.java:116) at org.springframework.security.ldap.authentication.BindAuthenticator.authenticate(BindAuthenticator.java:90) at org.springframework.security.ldap.authentication.LdapAuthenticationProvider.doAuthentication(LdapAuthenticationProvider.java:178) ... 32 more Caused by: javax.naming.CommunicationException: simple bind failed: mydomain.mycompany.net:636 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty] at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:219) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2788) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) at org.springframework.ldap.core.support.LdapContextSource.getDirContextInstance(LdapContextSource.java:43) at org.springframework.ldap.core.support.AbstractContextSource.createContext(AbstractContextSource.java:254) ... 39 more Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1815) at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:128) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:426) at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:399) at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:359) at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:214)

[#4 · Accepted solution]

OK, so after pulling out my hair for a day it seems that the ranger ADMIN portion of ranger now either has its own truststore file, or the file vanished during the upgrade (I do not have an old 2.4 hadoop cluster to check). In any case, under the settings for "Advanced ranger-admin-site" , near the bottom is the "ranger.truststore.file", defaulted to the value: "

/etc/ranger/admin/conf/ranger-admin-keystore.jks".

On my ranger server. that file was not even there, which is what leads me to believe this MIGHT be a new ranger parameter for ranger-admin? To solve my error above, I simply created this file with the java "keytool" and imported the certificate I've always had from our AD folks.

In HDP 2.4, ranger admin needed the certificate in the default java keystore (usually something like /usr/java/latest/jre/lib/security/cacerts). It looks like ranger ADMIN has its own truststore now....? (why everything is split between ranger admin and ranger usersync, lord only knows..).

I know the ranger.usersync.truststore.file was always there (defaults to

/usr/hdp/current/ranger-usersync/conf/mytruststore.jks, but that's not the issue above above).

Is the ranger ADMIN truststore file something new? If not, I swear I'm losing my mind. In any case, anyone using ranger admin UI authenticated with LDAP/AD over SSL.... look here. 🙂

[#5]

I just worked through this same issue and only found this conversations after the fact.

I configured ranger.truststore.file and ranger.https.attrib.keystore.file to point to my $JAVA_HOME cacerts file (which had my AD cert previously imported), and I’m now able to log in to the Ranger-Admin UI with my AD credentials.

I’m not sure what other ramifications this may have, but it solved my issue.

It’s also raised another question, in that many HortonWorks components each have their own configurations for various keystores. (Ranger vs. Ranger-Admin as an apropos example). Are there any recommendations or best practices for consolidating all of these keystore locations into one place (like I did with the above solution)?

[#6]

Yeah, the decentralized nature of the keystores is... kind of a huge problem and like both you and I discovered, not inherently obvious. I still like the horton setup for ease of management... I *really* do. (Tried a hadoop cluster once built from scratch manually.. you have no ideas how bad that was).

I'm discovering the "ease" of managing courtesy of Horton's setup really applies only about 90% of the time. The other 10% is a bitch. heh. With the Ambari UI, just about everything is there. Great! ALMOST.