Ambari LDAP Sync to AD using SSL returns HTTP Error 503

[#1]

I'm trying to sync users and groups from an Active Directory LDAP which is using SSL. I'm using Ambari 2.1.0

I have used the following information on setup-ldap:

At the Primary URL* prompt, enter the server URL and port you collected above. Prompts marked with an asterisk are required values.
ldaps://<ADServerName>:636 (note, I have tried this without the ldaps prefix, but got the same error)
At the Secondary URL* prompt, enter the secondary server URL and port. This value is optional. 

At the Use SSL* prompt, enter your selection. If using LDAPS, enter true. 
true
At the User object class* prompt, enter the object class that is used for users. 
inetOrgPerson
At the User name attribute* prompt, enter your selection. The default value is uid. 
(default)
At the Group object class* prompt, enter the object class that is used for groups. 
group
At the Group name attribute* prompt, enter the attribute for group name. 
cn (default)
At the Group member attribute* prompt, enter the attribute for group membership. 
member
At the Distinguished name attribute* prompt, enter the attribute that is used for the distinguished name. 
dn (default)
At the Base DN* prompt, enter your selection. 
<BaseDN>
At the Referral method* prompt, enter to follow or ignore LDAP referrals. 
follow
At the Bind anonymously* prompt, enter your selection. 
false
At the Manager DN* prompt, enter your selection if you have set bind.Anonymously to false. 
<BindDN>
At the Enter the Manager Password* prompt, enter the password for your LDAP manager DN. 
<BindDNpw>
Do you want to provide custom TrustStore for Amabari
y
TrustStoretype [jks/jceks/pkcs12] (jks)
jks (default)
Path to TrustStore file:
/etc/ambari-server/keys/ldaps-keystore.jks
Password for TrustStore
Re-enter password:
====================
Review Settings
====================
authentication.ldap.managerDn: <BindDN>
authentication.ldap.managerPassword: <BindDNpw>
ssl.trustStore.type: jks
ssl.trustStore.path: /etc/ambari-server/keys/ldaps-keystore.jks
ssl.trustStore.password: 
Save settings [y/n] (y)? y
Saving...done
Ambari Server 'setup-ldap' completed successfully.

I have tried this originally without the TrustStore set up (originally), then I tried again with a self-signed certificate using the instructions in Section 3.3.2.3. Create and Import Self-Signed Certificate in Hadoop Security Guide (May 26, 2015)

When I try to do an LDAP Sync, I get this error

ambari-server sync-ldap --users users.txt --groups groups.txt
Using python  /usr/bin/python2.6
Syncing with LDAP...
Enter Ambari Admin login: admin
Enter Ambari Admin password:
Syncing specified users and groups.ERROR: Exiting with exit code 1.
REASON: Sync event creation failed. Error details: HTTP Error 503: Service Unavailable

I have successfully used the same BindDN info, etc using the ldapsearch command on the command line, and using that, I can list the users and groups that I am trying sync.

Any suggestions on how I can figure out why I'm getting this error?

[#2]

Are you able to issue a simple ldapsearch from the command line?

ldapsearch -H ldaps://192.168.1.1:636 -D "cn=xxxx,ou=xxxx,ou=xxxx,dc=xxxx,dc=pmh,dc=org" -x -W -b "cn=xxx,ou=xxx,ou=xxxx,dc=xxxx,dc=xxxx,dc=org"

[#3]

Yes, an ldapsearch command like the following works and returns the attributes of a group, for instance.

ldapsearch -x -H ldaps://<ADServerName>:636 -b "<BaseDN>" -D <BindDN> -w <BindDNpw> "CN=<ExampleGroup>" 

I found that I had to use the ldaps:// prefix on the hostname. If I used -h and the hostname and -p and the port. the ldapsearch did not work.

Note: The parameters to this command are the same as the ones used in the ldap setup.

I have also done some experimentation in a sandbox environment, using OpenLDAP without SSL, and in that environment, I was able to get users and groups to sync successfully.

I'm not sure what is different in the non-sandbox environment with the SSL LDAP.

[#4]

Has Ambari Server been restarted after you configured LDAP and is it up and running and accessible via the web UI?

[#5]

Yes, I have restarted ambari server (just restarted it and re-tried the LDAP sync just now to be absolutely sure). Still getting the HTTP Error 503: Service Unavailable.

Also: The Ambari web UI is up and running and accessible through a web browser.

[#6 · Accepted solution]

@ccook make sure that the Ambari Server trusts the certificate that the LDAP server is using. One quick way to get that certificate directly is to use openssl to retrieve that certificate from the LDAP server, and then explicitly add it to a new keystore:

$ openssl s_client -showcerts -connect ldapserver.domain.com:636

You'll see the certificate printed in STDOUT, just look for BEGIN CERTIFICATE.

You will need to grab the entire certificate including the ----BEGIN and END ---- text, and save it to a file. In this case we'll call it ldap.cert.

Once this has been done you can follow 1.2.(1-3) steps in the doc to create a new JKS keystore and import that certificate to ensure that it's trusted by Ambari: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.1/bk_Ambari_Security_Guide/content/_configure_...

Now you've got a JKS keystore with that certificate in it, you can tell Ambari to use that when connecting to your LDAP server using SSL by re-running the ambari-server setup-ldap. Just make sure you answer correctly for:

• Use SSL=true
• TrustStore type=jks
• Path to TrustStore file=/etc/ambari-server/keys/ldaps-keystore.jks
• Password for TrustStore={{ what you typed in step 1.2.3 }}

[#7]

Thanks for the suggestion.

I used the 'openssl s_client -showcerts command', and it showed 2 different certs. I saved each one to a separate file (ldap1.cert and ldap2.cert) and added them to my keystore with different aliases as follows:

After that, I did a new setup-ldap, pointing to the keystore, restarted ambari server, but still have the same 503 error.

keytool -import -trustcacerts -alias ad-cert1 -file ldap1.cert -keystore /etc/ambari-server/keys/ldaps-keystore.jks

keytool -import -trustcacerts -alias ad-cert2 –file ldap2.cert -keystore /etc/ambari-server/keys/ldaps-keystore.jks

[#8]

I decided to try something else. I set up a separate LDAP server on the ambari-server node. I populated it with a few users and groups, tested that I could dump them out with the ldapsearch command. Then I pointed my ldap-setup to this, non-SSL LDAP source. I got the same error:

REASON: Sync event creation failed. Error details: HTTP Error 503: Service Unavailable

I also get this same 503 error if I mis-type the credentials for the prompt for Ambari Admin login. It looks like the sync-ldap isn't connecting to ambari-server although the console works. Any suggestions how to confirm/test this?

[#9]

In this case we found that the http_proxy and https_proxy environmental variables were set causing the ambari-server sync-ldap command to route it's communication with the Ambari Server through a proxy. That proxy was returning a 503 as it didn't have access to the Ambari Server. Once that was resolved, we noticed that that a VIP was being used to front multiple DC's and the CA was not available. Once a CA cert is provided by the client we can re-create the JKS file from the commands in the document and sync successfully.

[#10]

The client provided a CA cert, which, once put into the keystore, worked consistently in allowing the LDAP Sync to work, including users, groups and memberships. Thanks so much, Paul, for your help in resolving this issue! It is worth mentioning that in the course of troubleshooting this, we also found that the keystore pointed to by ambari-server can only make use of a single cert, and its alias must be 'root'. As well, if your ambari server is running as a user other than root, it must be given permission to read the keystore file (eg if the keystore is created by root user)

Thanks again!