Created 01-08-2019 05:38 AM
@Jay SenSharma , @Geoffrey Shelton Okot
Hi All,
HDP-2.6.5
Ambari 2.6.2.2
openjdk version "1.8.0_181"
This is regarding the SSL configuration in all the server.
My Ambari server is working fine with HTTPS.
I am using only .key file and .cer file (i am using this as a cert file)
Hadoop components are not going into HTTPS. Namanode UI, Yarn Resource manager, mapreduce jobHistory UI, zeppelin UI.
[root@xxxxxxxx ~]# ambari-server setup-security
Using python /usr/bin/python Security setup options... ===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore. ===========================================================================
Enter choice, (1-5): 1
Do you want to configure HTTPS [y/n] (y)? y
SSL port [8443] ? 8443
Enter path to Certificate: /hadoop/certs/xxxxx.localhost.cer
Enter path to Private Key: /hadoop/certs/xxxxx.localhost.key
Please enter password for Private Key:
Importing and saving Certificate...done.
Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership...
NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start")
[root@lhdcsi02v ~]# ambari-server restart Using python /usr/bin/python Restarting ambari-server Waiting for server stop... Ambari Server stopped Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start............................................................ DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details. ERROR: Exiting with exit code 1.
REASON: Server not yet listening on http port 8443 after 50 seconds. Exiting.
-------------------------------------------------------------------------------------------------------------------------------------------------
keytool -import -noprompt -alias OwnCA -file xxxx.localhost.cer –storepass changeit -keystore /etc/pki/java/cacerts
------------------------------------------------------------------------------------------------------------------------------------------------------
Setup truststore
[root@xxxxxxxx ~]# ambari-server setup-security
Using python /usr/bin/python Security setup options... ===========================================================================
Choose one of the following options:
[1]Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore. ===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] ? y
The truststore is already configured. Do you want to re-configure the truststore [y/n] ? y
TrustStore type [jks/jceks/pkcs12] (jks):jks
Path to TrustStore file : /etc/pki/java/cacerts
Password for TrustStore: changeit
Re-enter password: changeit
Ambari Server 'setup-security' completed successfully. [root@xxxxx ~]#
[root@xxxxx conf]# keytool -import -noprompt -alias OwnCA -file /hadoop/certs/xxxx.localhost.cer -storepass changeit -keypass changeit -keystore /etc/hadoop/conf/hadoop-private-keystore.jks Certificate was added to keystore
/hadoop/certs/hadoop-private-keystore.jks i have copy in all the datanodes as well
ERROR:- NameNode Web UI Connection failed to https://xxxxxxx.localhost:50470 (<urlopen error EOF occurred in violation of protocol (_ssl.c:579)>)
Ambari server host is having the ceritifcates it is showing Datanode
[hdfs@xxxxx hdfs]$ openssl s_client -connect xxxx.localhost:50470 -tls1_2 CONNECTED(00000003) 140047696471952:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:659: ---
no peer certificate available ---
No client certificate CA names sent
--- SSL handshake has read 0 bytes and written 0 bytes
--- New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE Expansion:
NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1546675079 Timeout : 7200 (sec) Verify return code: 0 (ok)
Below are the things i have tried
/etc/amabri-agent/conf/ambari-agent.ini" in all the hosts in the cluster [security] (done)
force_https_protocol=PROTOCOL_TLSv1_2 ambari.properties (done) security.server.disabled.protocols=SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1
python /tmp/testPythonProtocols.py
PROTOCOL_SSLv2 PROTOCOL_SSLv23 PROTOCOL_SSLv3 PROTOCOL_TLSv1 PROTOCOL_TLSv1_1 PROTOCOL_TLSv1_2 ---
Namenode.
[root@xxxxx certs]# openssl s_client -connect xxx.production.local:50470 CONNECTED(00000003) 140713499117456:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1546923639 Timeout : 300 (sec) Verify return code: 0 (ok) ---
Created 01-08-2019 05:43 AM
In your "/var/lib/ambari-server/ambari-env.sh" file try adding the "-Djavax.net.debug=all" and then restart the ambari-server once.
Example:
# grep 'debug' /var/lib/ambari-server/ambari-env.sh export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djavax.net.debug=all'
After the above config change when you will restart ambari-server then you will see many SSL debugs inside the OUT file (not inside log).
# ambari-server restart # tail -f /var/log/ambari-server/ambari-server.out
Please share that "ambari-server.out" log file here.
Similarly if you want to findout the cause of failure of your HDFS components like NN/DN then please add the same JVM property "-Djavax.net.debug=all" to their JVM and restart them.
Based on the following error I am suspecting that your "java.security" might have some unwanted cipher.
So please check the following file inside your $JAVA_HOME/jre/lib/security/java.security and If you find that entry there '3DES_EDE_CBC' then please remove that string after taking a backup of the file.
# grep '3DES_EDE_CBC' $JAVA_HOME/jre/lib/security/java.security
then restart ambari-server (and other component which are using that JAVA_HOME)
For more details on '3DES_EDE_CBC' please refer to : https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...
Created 01-08-2019 08:31 AM
@Jay SenSharma File attched.
I have done the changes and restarted the ambari server.