Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HTTPS Hostname Wrong

HTTPS Hostname Wrong

New Contributor

I have enabled SSL encryption with Self Signed certificate. before that To put our nodes in DNS I have modified hostnames and made the necessary changes in the cluster. However I think I have missed something. Due to which I am getting following error. What all changes I have to do when I change the hostnames?

 

Exception in doCheckpoint
java.io.IOException: HTTPS hostname wrong:  should be <nn.mydomain.local>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:652)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:577)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
	at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:164)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:402)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:395)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadImageToStorage(TransferFsImage.java:114)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$2.run(SecondaryNameNode.java:456)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$2.run(SecondaryNameNode.java:444)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:443)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:540)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:395)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:356)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
	at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:409)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:357)
	at java.lang.Thread.run(Thread.java:745)
5 REPLIES 5

Re: HTTPS Hostname Wrong

Super Guru

Hossam,

 

When you create your key store and cert you need to make sure that the hostname you specified during creation matches the host on which the server using the certificate to listen on SSL is running.

 

The exception tells us that the certificate should have been issued for "nn.mydomain.local" as that is what the host resolves to.  Try using keytool -list -v -keystore <your_keystore> to check for what host the certificate was created.

 

If you are unsure about this, please post the steps you used to create your self-signed cert and we can target the root cause.

 

This error can hapen if your nn.mydomain.local but you have specified something else as the hostname when you created your keystore

 

Ben

Re: HTTPS Hostname Wrong

New Contributor

Thanks Ben for looking into it.

 

We have been generating certificate with openssl file. What I have done is I have hdfs.jks keystore. In this store I have added self signed certificates for all the nodes in the cluster and I have truststore cacerts which has all nodes certificates. I have deployed this truststore and keystore on all the nodes. Setup all required details in Cluster for HDFS, Yarn and MapReduce. As soon as I have restarted the server I got earlier error. Now as I see in the logs I have also got below two errors before earlier error.

 

 

1:33:16.389 AMINFOorg.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
Image has changed. Downloading updated image from NN.
1:33:16.389 AMINFOorg.apache.hadoop.hdfs.server.namenode.TransferFsImage
Opening connection to https://nn.mydomain.local:50470/imagetransfer?getimage=1&txid=1040569&storageInfo=-59:2055389451:0:cluster2
1:33:16.409 AMWARNorg.apache.hadoop.security.UserGroupInformation
PriviledgedActionException as:hdfs/nn.mydomain.local@MYDOMAIN.LOCAL (auth:KERBEROS) cause:java.io.IOException: HTTPS hostname wrong:  should be <nn.mydomain.local>

 

 

Note: This cluster is kerberized and its working fine without this SSL setup enablement.

 

My understanding is: Each node should have self signed certificate of all nodes in keystore and truststore should also have trust certificate from all the nodes. That is what I have done here.

 

 Let me know If we are missing somethig.

 

Thanks,

Bhavin

Re: HTTPS Hostname Wrong

New Contributor

Below is the configuration file which is used to generate certificate for all nodes. CN and DNS.1/2 where modified as per the node.

 

#
# OpenSSL example configuration file.
# This is mostly being used for generation of certificate requests.
#

# This definition stops the following lines choking if HOME isn't
# defined.
HOME = .
RANDFILE = $ENV::HOME/.rnd

# Extra OBJECT IDENTIFIER info:
#oid_file = $ENV::HOME/.oid
oid_section = new_oids

# To use this configuration file with the "-extfile" option of the
# "openssl x509" utility, name here the section containing the
# X.509v3 extensions to use:
# extensions =
# (Alternatively, use a configuration file that has only
# X.509v3 extensions in its main [= default] section.)

[ new_oids ]

# We can add new OIDs in here for use by 'ca', 'req' and 'ts'.
# Add a simple OID like this:
# testoid1=1.2.3.4
# Or use config file substitution like this:
# testoid2=${testoid1}.5.6

# Policies used by the TSA examples.
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7

####################################################################
[ ca ]
default_ca = CA_default # The default ca section

####################################################################
[ CA_default ]

dir = /etc/pki/CA # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
#unique_subject = no # Set to 'no' to allow creation of
# several ctificates with same subject.
new_certs_dir = $dir/newcerts # default place for new certs.

certificate = $dir/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number
# must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/cakey.pem# The private key
RANDFILE = $dir/private/.rand # private random number file

x509_extensions = usr_cert # The extentions to add to the cert

# Comment out the following two lines for the "traditional"
# (and highly broken) format.
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options

# Extension copying option: use with caution.
copy_extensions = copy

# Extensions to add to a CRL. Note: Netscape communicator chokes on V2 CRLs
# so this is commented out by default to leave a V1 CRL.
# crlnumber must also be commented out to leave a V1 CRL.
# crl_extensions = crl_ext

default_days = 365 # how long to certify for
default_crl_days= 30 # how long before next CRL
default_md = default # use public key default MD
preserve = no # keep passed DN ordering

# A few difference way of specifying how similar the request should look
# For type CA, the listed attributes must be the same, and the optional
# and supplied fields are just that :-)
policy = policy_match

# For the CA policy
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

# For the 'anything' policy
# At this point in time, you must list all acceptable 'object'
# types.
[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

####################################################################
[ req ]
default_bits = 2048
default_md = sha1
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_req # The extentions to add to the self signed cert
prompt = no

# Passwords for private keys if not present they will be prompted for
# input_password = secret
# output_password = secret

# This sets a mask for permitted string types. There are several options.
# default: PrintableString, T61String, BMPString.
# pkix : PrintableString, BMPString (PKIX recommendation before 2004)
# utf8only: only UTF8Strings (PKIX recommendation after 2004).
# nombstr : PrintableString, T61String (no BMPStrings or UTF8Strings).
# MASK:XXXX a literal mask value.
# WARNING: ancient versions of Netscape crash on BMPStrings or UTF8Strings.
string_mask = utf8only

# req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
C = US
ST = California
L = San Jose
O = Org
CN = nn.mydomain.local

# SET-ex3 = SET extension number 3

[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20

unstructuredName = An optional company name

[ usr_cert ]

# These extensions are added when 'ca' signs a request.

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# Here are some examples of the usage of nsCertType. If it is omitted
# the certificate can be used for anything *except* object signing.

# This is OK for an SSL server.
# nsCertType = server

# For an object signing certificate this would be used.
# nsCertType = objsign

# For normal client use this is typical
# nsCertType = client, email

# and for everything including object signing:
# nsCertType = client, email, objsign

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# This will be displayed in Netscape's comment listbox.
nsComment = "OpenSSL Generated Certificate"

# PKIX recommendations harmless if included in all certificates.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

# This stuff is for subjectAltName and issuerAltname.
# Import the email address.
# subjectAltName=email:copy
# An alternative to produce certificates that aren't
# deprecated according to PKIX.
# subjectAltName=email:move

# Copy subject details
# issuerAltName=issuer:copy

#nsCaRevocationUrl = http://www.domain.dom/ca-crl.pem
#nsBaseUrl
#nsRevocationUrl
#nsRenewalUrl
#nsCaPolicyUrl
#nsSslServerName

# This is required for TSA certificates.
# extendedKeyUsage = critical,timeStamping

[ v3_req ]

# Extensions to add to a certificate request

basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = nn.mydomain.local
DNS.2 = 1.1.1.1

[ v3_ca ]


# Extensions for a typical CA


# PKIX recommendation.

subjectKeyIdentifier=hash

authorityKeyIdentifier=keyid:always,issuer

# This is what PKIX recommends but some broken software chokes on critical
# extensions.
#basicConstraints = critical,CA:true
# So we do this instead.
basicConstraints = CA:true

# Key usage: this is typical for a CA certificate. However since it will
# prevent it being used as an test self-signed certificate it is best
# left out by default.
# keyUsage = cRLSign, keyCertSign
keyUsage = digitalSignature, keyEncipherment

# Some might want this also
# nsCertType = sslCA, emailCA

# Include email address in subject alt name: another PKIX recommendation
# subjectAltName=email:copy
# Copy issuer details
# issuerAltName=issuer:copy

# DER hex encoding of an extension: beware experts only!
# obj=DER:02:03
# Where 'obj' is a standard or added object
# You can even override a supported extension:
# basicConstraints= critical, DER:30:03:01:01:FF

[ crl_ext ]

# CRL extensions.
# Only issuerAltName and authorityKeyIdentifier make any sense in a CRL.

# issuerAltName=issuer:copy
authorityKeyIdentifier=keyid:always

[ proxy_cert_ext ]
# These extensions should be added when creating a proxy certificate

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# Here are some examples of the usage of nsCertType. If it is omitted
# the certificate can be used for anything *except* object signing.

# This is OK for an SSL server.
# nsCertType = server

# For an object signing certificate this would be used.
# nsCertType = objsign

# For normal client use this is typical
# nsCertType = client, email

# and for everything including object signing:
# nsCertType = client, email, objsign

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# This will be displayed in Netscape's comment listbox.
nsComment = "OpenSSL Generated Certificate"

# PKIX recommendations harmless if included in all certificates.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

# This stuff is for subjectAltName and issuerAltname.
# Import the email address.
# subjectAltName=email:copy
# An alternative to produce certificates that aren't
# deprecated according to PKIX.
# subjectAltName=email:move

# Copy subject details
# issuerAltName=issuer:copy

#nsCaRevocationUrl = http://www.domain.dom/ca-crl.pem
#nsBaseUrl
#nsRevocationUrl
#nsRenewalUrl
#nsCaPolicyUrl
#nsSslServerName

# This really needs to be in place for it to be a proxy certificate.
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

####################################################################
[ tsa ]

default_tsa = tsa_config1 # the default TSA section

[ tsa_config1 ]

# These are used by the TSA reply generation only.
dir = ./demoCA # TSA root directory
serial = $dir/tsaserial # The current serial number (mandatory)
crypto_device = builtin # OpenSSL engine to use for signing
signer_cert = $dir/tsacert.pem # The TSA signing certificate
# (optional)
certs = $dir/cacert.pem # Certificate chain to include in reply
# (optional)
signer_key = $dir/private/tsakey.pem # The TSA private key (optional)

default_policy = tsa_policy1 # Policy if request did not specify it
# (optional)
other_policies = tsa_policy2, tsa_policy3 # acceptable policies (optional)
digests = md5, sha1 # Acceptable message digests (mandatory)
accuracy = secs:1, millisecs:500, microsecs:100 # (optional)
clock_precision_digits = 0 # number of digits after dot. (optional)
ordering = yes # Is ordering defined for timestamps?
# (optional, default: no)
tsa_name = yes # Must the TSA name be included in the reply?
# (optional, default: no)
ess_cert_id_chain = no # Must the ESS cert id chain be included?
# (optional, default: no)

Re: HTTPS Hostname Wrong

Super Guru

Hello,

 

Can you please provide the following on your nn.mydomain.local host:

 

keytool -list -v -keystore <your_keystore>

 

What's happening here is that the secondary namenode is trying to get checkpoing information from the primary namenode.  It connects to nn.mydomain.local and gets the primary namenode's server certificate.  After that, it checks the "cn" of the certificate subject to make sure that it matches the name it connected to.

 

That's why your keytool output would be useful.

 

Check http://tools.ietf.org/html/rfc2818#section-3.1 for more information on this type of check.  The important bit for us here is:

In general, HTTP/TLS requests are generated by dereferencing a URI.
   As a consequence, the hostname for the server is known to the client.
   If the hostname is available, the client MUST check it against the
   server's identity as presented in the server's Certificate message,
   in order to prevent man-in-the-middle attacks.

 

There may be some other factors based on your environment, network, etc., but the place to start it making sure we know what the subject of your server certificate is.

 

Ben

 

Highlighted

Re: HTTPS Hostname Wrong

Super Collaborator

To check what the "Subject" value is for the certificate, as being presented by the server upon connection by a client, you can use the openssl command.

 

# openssl s_client -connect host.fqdn.name:port 

 

So verify the FQDN being presented in the "CN" value of the "Subject" field of the connection.

 

for example

 

# openssl s_client -connect www.google.com:443

 

CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEdjCCA16gAwIBAgIIO0tlu3YE0sYwDQYJKoZIhvcNAQEFBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTUwMjI3MjAwNTA0WhcNMTUwNTI4MDAwMDAw
WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCasr3L
0iT3eC6vigkdRrMgx31BdqigfH+YTpiizVQR8BZw4Y4g7VJ1WotuiPvV87f03e5p
uWq5adHjwrUqs4JEMpA3Ma6KgzxLU1CGrXdQswpX+wvB3R26CfodRw/6PH29o7Db
Au9azOy2b3k9GaPwhYa6OnMs41me9HIrCOAzuVbbtCBpc/IVWVY6NoXFxFDpbOTP
5Yqtj/RHkYTvO6Bb66kWfDbKfHHj0SmWujnWBIAH3BpgU1HEDVSrqH4W8MJKOQAS
Ux3tF4f3WWIdohJv16c3nQhLu5j9my4jCxhjvi0IW+0+MyjYYnId390mgitcG04M
MFiLQEs+3J3OTa15AgMBAAGjggFBMIIBPTAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwGQYDVR0RBBIwEIIOd3d3Lmdvb2dsZS5jb20waAYIKwYBBQUHAQEE
XDBaMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lBRzIuY3J0
MCsGCCsGAQUFBzABhh9odHRwOi8vY2xpZW50czEuZ29vZ2xlLmNvbS9vY3NwMB0G
A1UdDgQWBBS0zUHKAq1ttwclIcHPfjrTlD+3ZDAMBgNVHRMBAf8EAjAAMB8GA1Ud
IwQYMBaAFErdBhYbvPZotXb1gba7Yhq6WoEvMBcGA1UdIAQQMA4wDAYKKwYBBAHW
eQIFATAwBgNVHR8EKTAnMCWgI6Ahhh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lB
RzIuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQAGjetZT+3gdq4LQxUWdiRouw3TQgJO
XiglcsKU/UF1tbX7LhpZ+9/LNGXHdlIsoOIei7YDPKQ4NZ7HHwysJXjGofM8S0hF
nHHkIfmp5ARN4VKAXcevwm6JFZM45Cnzg0e1Kn89ShGrZSaIRQRKAIoVJyqmS6Rb
TtIOtGY5179XOWsSqsw4FEEBs4UbX/N3b6hyb1di+7Hr2EaAXlxrHLs6KuI/xJaT
lfTM+ou2N0NcmaJoppVym0Ss7Kr0873B7UPr2vcFeklM+NlUXkLsaJX9cMH0GpiF
b6FjzLTKeG0qUw4ei8t5jL7pzBB7+piqrTer5u93/sd/C2bIzgUs1tZk
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
---
SSL handshake has read 3218 bytes and written 444 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : RC4-SHA
Session-ID: 0A7405814F70E3E8B373BC934F69B15846B5DFBB14980845A718EF57BBAE75AD
Session-ID-ctx:
Master-Key: DCCDD92DF11F398475FC7A549864FD77FFE3FB8B0A339062050EE567FF2E1F2B6C3EB94050C5AF023130A746C6D9434A
Key-Arg : None
Start Time: 1425652804
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
^C

 

(note that you will see a "Verify return code: 19" I believe for having self signed certificates... it might be 21)