HDFS, Yarn, Mapreduce URl going into HTTPS with unknown host connection error (ERR_CONNECTION_CLOSED)

vishal_bohra · ‎01-08-2019

Hi All,

HDP-2.6.5

Ambari 2.6.2.2

openjdk version "1.8.0_181"

This is regarding the SSL configuration in all the server.

My Ambari server is working fine with HTTPS.

I am using only .key file and .cer file (i am using this as a cert file)

Hadoop components are not going into HTTPS. Namanode UI, Yarn Resource manager, mapreduce jobHistory UI, zeppelin UI.

[root@xxxxxxxx ~]# ambari-server setup-security

Using python /usr/bin/python Security setup options... ===========================================================================

Choose one of the following options:

[1] Enable HTTPS for Ambari server.

[2] Encrypt passwords stored in ambari.properties file.

[3] Setup Ambari kerberos JAAS configuration.

[4] Setup truststore.

[5] Import certificate to truststore. ===========================================================================

Enter choice, (1-5): 1

Do you want to configure HTTPS [y/n] (y)? y

SSL port [8443] ? 8443

Enter path to Certificate: /hadoop/certs/xxxxx.localhost.cer

Enter path to Private Key: /hadoop/certs/xxxxx.localhost.key

Please enter password for Private Key:

Importing and saving Certificate...done.

Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership...

NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start")

[root@lhdcsi02v ~]# ambari-server restart Using python /usr/bin/python Restarting ambari-server Waiting for server stop... Ambari Server stopped Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start............................................................ DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details. ERROR: Exiting with exit code 1.

REASON: Server not yet listening on http port 8443 after 50 seconds. Exiting.

-------------------------------------------------------------------------------------------------------------------------------------------------

keytool -import -noprompt -alias OwnCA -file xxxx.localhost.cer –storepass changeit -keystore /etc/pki/java/cacerts

------------------------------------------------------------------------------------------------------------------------------------------------------

Setup truststore

[root@xxxxxxxx ~]# ambari-server setup-security

Using python /usr/bin/python Security setup options... ===========================================================================

Choose one of the following options:

[1]Enable HTTPS for Ambari server.

[2] Encrypt passwords stored in ambari.properties file.

[3] Setup Ambari kerberos JAAS configuration.

[4] Setup truststore.

[5] Import certificate to truststore. ===========================================================================

Enter choice, (1-5): 4

Do you want to configure a truststore [y/n] ? y

The truststore is already configured. Do you want to re-configure the truststore [y/n] ? y

TrustStore type [jks/jceks/pkcs12] (jks):jks

Path to TrustStore file : /etc/pki/java/cacerts

Password for TrustStore: changeit

Re-enter password: changeit

Ambari Server 'setup-security' completed successfully. [root@xxxxx ~]#

[root@xxxxx conf]# keytool -import -noprompt -alias OwnCA -file /hadoop/certs/xxxx.localhost.cer -storepass changeit -keypass changeit -keystore /etc/hadoop/conf/hadoop-private-keystore.jks Certificate was added to keystore

/hadoop/certs/hadoop-private-keystore.jks i have copy in all the datanodes as well

ERROR:- NameNode Web UI Connection failed to https://xxxxxxx.localhost:50470 (<urlopen error EOF occurred in violation of protocol (_ssl.c:579)>)

Ambari server host is having the ceritifcates it is showing Datanode

[hdfs@xxxxx hdfs]$ openssl s_client -connect xxxx.localhost:50470 -tls1_2 CONNECTED(00000003) 140047696471952:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:659: ---

no peer certificate available ---

No client certificate CA names sent

--- SSL handshake has read 0 bytes and written 0 bytes

--- New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE Expansion:

NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1546675079 Timeout : 7200 (sec) Verify return code: 0 (ok)

Below are the things i have tried

/etc/amabri-agent/conf/ambari-agent.ini" in all the hosts in the cluster [security] (done)

force_https_protocol=PROTOCOL_TLSv1_2 ambari.properties (done) security.server.disabled.protocols=SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1

python /tmp/testPythonProtocols.py

PROTOCOL_SSLv2 PROTOCOL_SSLv23 PROTOCOL_SSLv3 PROTOCOL_TLSv1 PROTOCOL_TLSv1_1 PROTOCOL_TLSv1_2 ---

Namenode.

[root@xxxxx certs]# openssl s_client -connect xxx.production.local:50470 CONNECTED(00000003) 140713499117456:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1546923639 Timeout : 300 (sec) Verify return code: 0 (ok) ---

jsensharma · ‎01-08-2019

@Vishal Bohra

In your "/var/lib/ambari-server/ambari-env.sh" file try adding the "-Djavax.net.debug=all" and then restart the ambari-server once.

Example:

# grep 'debug' /var/lib/ambari-server/ambari-env.sh

export 
AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djavax.net.debug=all'

After the above config change when you will restart ambari-server then you will see many SSL debugs inside the OUT file (not inside log).

# ambari-server restart
# tail -f  /var/log/ambari-server/ambari-server.out

Please share that "ambari-server.out" log file here.

Similarly if you want to findout the cause of failure of your HDFS components like NN/DN then please add the same JVM property "-Djavax.net.debug=all" to their JVM and restart them.

Based on the following error I am suspecting that your "java.security" might have some unwanted cipher.

So please check the following file inside your $JAVA_HOME/jre/lib/security/java.security and If you find that entry there '3DES_EDE_CBC' then please remove that string after taking a backup of the file.

# grep '3DES_EDE_CBC' $JAVA_HOME/jre/lib/security/java.security

then restart ambari-server (and other component which are using that JAVA_HOME)

For more details on '3DES_EDE_CBC' please refer to : https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

vishal_bohra · ‎01-08-2019

ambari-serverout.txt

@Jay SenSharma File attched.

I have done the changes and restarted the ambari server.