Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS, Yarn, Mapreduce URl going into HTTPS with unknown host connection error (ERR_CONNECTION_CLOSED)

Highlighted

HDFS, Yarn, Mapreduce URl going into HTTPS with unknown host connection error (ERR_CONNECTION_CLOSED)

New Contributor

ambari-serverlogs.txt

namenodelogstxt.txt

namnode-logs.txt

@Jay SenSharma , @Geoffrey Shelton Okot


Hi All,

HDP-2.6.5

Ambari 2.6.2.2

openjdk version "1.8.0_181"

This is regarding the SSL configuration in all the server.

My Ambari server is working fine with HTTPS.

I am using only .key file and .cer file (i am using this as a cert file)

Hadoop components are not going into HTTPS. Namanode UI, Yarn Resource manager, mapreduce jobHistory UI, zeppelin UI.

[root@xxxxxxxx ~]# ambari-server setup-security

Using python /usr/bin/python Security setup options... ===========================================================================

Choose one of the following options:

[1] Enable HTTPS for Ambari server.

[2] Encrypt passwords stored in ambari.properties file.

[3] Setup Ambari kerberos JAAS configuration.

[4] Setup truststore.

[5] Import certificate to truststore. ===========================================================================

Enter choice, (1-5): 1

Do you want to configure HTTPS [y/n] (y)? y

SSL port [8443] ? 8443

Enter path to Certificate: /hadoop/certs/xxxxx.localhost.cer

Enter path to Private Key: /hadoop/certs/xxxxx.localhost.key

Please enter password for Private Key:

Importing and saving Certificate...done.

Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership...

NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start")

[root@lhdcsi02v ~]# ambari-server restart Using python /usr/bin/python Restarting ambari-server Waiting for server stop... Ambari Server stopped Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start............................................................ DB configs consistency check found warnings. See /var/log/ambari-server/ambari-server-check-database.log for more details. ERROR: Exiting with exit code 1.

REASON: Server not yet listening on http port 8443 after 50 seconds. Exiting.

-------------------------------------------------------------------------------------------------------------------------------------------------

keytool -import -noprompt -alias OwnCA -file xxxx.localhost.cer –storepass changeit -keystore /etc/pki/java/cacerts

------------------------------------------------------------------------------------------------------------------------------------------------------

Setup truststore

[root@xxxxxxxx ~]# ambari-server setup-security

Using python /usr/bin/python Security setup options... ===========================================================================

Choose one of the following options:

[1]Enable HTTPS for Ambari server.

[2] Encrypt passwords stored in ambari.properties file.

[3] Setup Ambari kerberos JAAS configuration.

[4] Setup truststore.

[5] Import certificate to truststore. ===========================================================================

Enter choice, (1-5): 4

Do you want to configure a truststore [y/n] ? y

The truststore is already configured. Do you want to re-configure the truststore [y/n] ? y

TrustStore type [jks/jceks/pkcs12] (jks):jks

Path to TrustStore file : /etc/pki/java/cacerts

Password for TrustStore: changeit

Re-enter password: changeit

Ambari Server 'setup-security' completed successfully. [root@xxxxx ~]#

[root@xxxxx conf]# keytool -import -noprompt -alias OwnCA -file /hadoop/certs/xxxx.localhost.cer -storepass changeit -keypass changeit -keystore /etc/hadoop/conf/hadoop-private-keystore.jks Certificate was added to keystore

/hadoop/certs/hadoop-private-keystore.jks i have copy in all the datanodes as well

ERROR:- NameNode Web UI Connection failed to https://xxxxxxx.localhost:50470 (<urlopen error EOF occurred in violation of protocol (_ssl.c:579)>)

Ambari server host is having the ceritifcates it is showing Datanode

[hdfs@xxxxx hdfs]$ openssl s_client -connect xxxx.localhost:50470 -tls1_2 CONNECTED(00000003) 140047696471952:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:659: ---

no peer certificate available ---

No client certificate CA names sent

--- SSL handshake has read 0 bytes and written 0 bytes

--- New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE Expansion:

NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1546675079 Timeout : 7200 (sec) Verify return code: 0 (ok)

Below are the things i have tried

/etc/amabri-agent/conf/ambari-agent.ini" in all the hosts in the cluster [security] (done)

force_https_protocol=PROTOCOL_TLSv1_2 ambari.properties (done) security.server.disabled.protocols=SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1

python /tmp/testPythonProtocols.py

PROTOCOL_SSLv2 PROTOCOL_SSLv23 PROTOCOL_SSLv3 PROTOCOL_TLSv1 PROTOCOL_TLSv1_1 PROTOCOL_TLSv1_2 ---

Namenode.

[root@xxxxx certs]# openssl s_client -connect xxx.production.local:50470 CONNECTED(00000003) 140713499117456:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 289 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : 0000 Session-ID: Session-ID-ctx: Master-Key: Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1546923639 Timeout : 300 (sec) Verify return code: 0 (ok) ---

2 REPLIES 2

Re: HDFS, Yarn, Mapreduce URl going into HTTPS with unknown host connection error (ERR_CONNECTION_CLOSED)

Super Mentor

@Vishal Bohra

In your "/var/lib/ambari-server/ambari-env.sh" file try adding the "-Djavax.net.debug=all" and then restart the ambari-server once.

Example:

# grep 'debug' /var/lib/ambari-server/ambari-env.sh

export 
AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djavax.net.debug=all'


After the above config change when you will restart ambari-server then you will see many SSL debugs inside the OUT file (not inside log).

# ambari-server restart
# tail -f  /var/log/ambari-server/ambari-server.out



Please share that "ambari-server.out" log file here.

Similarly if you want to findout the cause of failure of your HDFS components like NN/DN then please add the same JVM property "-Djavax.net.debug=all" to their JVM and restart them.


Based on the following error I am suspecting that your "java.security" might have some unwanted cipher.

So please check the following file inside your $JAVA_HOME/jre/lib/security/java.security and If you find that entry there '3DES_EDE_CBC' then please remove that string after taking a backup of the file.

# grep '3DES_EDE_CBC' $JAVA_HOME/jre/lib/security/java.security 

then restart ambari-server (and other component which are using that JAVA_HOME)

For more details on '3DES_EDE_CBC' please refer to : https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

Re: HDFS, Yarn, Mapreduce URl going into HTTPS with unknown host connection error (ERR_CONNECTION_CLOSED)

New Contributor

ambari-serverout.txt

@Jay SenSharma File attched.

I have done the changes and restarted the ambari server.

Don't have an account?
Coming from Hortonworks? Activate your account here