Issue starting namenodes after enabling SSL

[#1]

I am trying to enable SSL for HDP components ussing OpenSSL setting up my internal CA on one of the cluster nodes.

I made the changes for the enabling SSL on HDFS, YARN and mapreduce.

Below are some links I am following :

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-security/content/set_up_truststor...

https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html

https://community.hortonworks.com/articles/52876/enable-https-for-yarn-and-mapreduce2.html

The issue is, after making the required changes when i am restarting the HDFS service, the namenodes do not come up as Active/Standby namenode(both coming up as just namenodes)

I am runnning HDP 2.4 with Ambari version 2.1.

I am just a beginner at this SSL thing and do not have much idea.

Will really appreciate any help on this.

Thanks.

[#2]

@Sumit Sharma

You mentioned that

The issue is, after making the required changes when i am restarting the HDFS service, the namenodes do not come up as Active/Standby namenode(both coming up as just namenodes)

.

So can you please check if you are able to get the status of Active/StandBy NN using command line? (this is to isolate the issue with Ambari UI showing wrong status OR the NN itself is in inconsistent state). Please share the output of the following commands.

# su - hdfs 
$ hdfs haadmin -getServiceState nn1
active
$ hdfs haadmin -getServiceState nn2
standby

.

Also please check and share the NameNode logs (both) so that we can see if there are any issues ERROR/WARN messages.

.

Also please check if NameNode processes are UP and running or not?

# ps  -ef | grep NameNode

.

Have they opened the port properly? (You will need to change the ports here to point to your NN ssl port that you configured)

# netstat -tnlpa | grep 50070
# netstat -tnlpa | grep 8020

.

Your please check the NameNode Host names (FQDN `hostname -f`) are pointing to the correct resolvable FQDN and the certificate CN is also matching the FQDN ?

.

[#3]

@Jay SenSharma

The CLI shows the corret status of the name nodes.PFB the O/P of the commands

$hdfs haadmin -getServiceState nn1

active

$ hdfs haadmin -getServiceState nn2

standby

I checked the namenode logs also.. and found the below error when i restart the HDFS service

INFO hdfs.StateChange (FSNamesystem.java:reportStatus(5578)) - STATE* Safe mode ON, in safe mode extension. The reported blocks 27 has reached the threshold 0.9900 of total blocks 27. The number of live datanodes 3 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 9 seconds. 2017-08-05 12:29:07,774 INFO ipc.Server (Server.java:run(2172)) - IPC Server handler 32 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.67.169.48:50022 Call#2335 Retry#10 org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot renew lease for DFSClient_NONMAPREDUCE_-1379506884_1. Name node is in safe mode. The reported blocks 27 has reached the threshold 0.9900 of total blocks 27. The number of live datanodes 3 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 6 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1331) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewLease(FSNamesystem.java:4471) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.renewLease(NameNodeRpcServer.java:1056) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.renewLease(ClientNamenodeProtocolServerSideTranslatorPB.java:652) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145) Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot renew lease for DFSClient_NONMAPREDUCE_-1379506884_1. Name node is in safe mode. The reported blocks 27 has reached the threshold 0.9900 of total blocks 27. The number of live datanodes 3 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 6 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327) ... 12 more

[#4]

@Sumit Sharma

Can you please check if your Journal Nodes are having the right ports (HTTPS) defined in the following properties inside your hdfs-site.xml file? If following is not added then please add that.
Example:

dfs.journalnode.https-address 0.0.0.0:8481

. Also please check the "dfs.namenode.https-address.xxxxx.mn1" and "dfs.namenode.https-address.xxxxx.mn2" properties are having correct Hostname & port defined and are accessible from ambari server.

.

- Journal Node Ports are correct and opened?

netstat -tnlpa | grep $PID_JOURNALNODE

.

[#5]

@Jay SenSharma

the https address is there and the netstat command is also showing the below result

tcp 0 0 0.0.0.0:8480 0.0.0.0:* LISTEN 22936/java

tcp 0 0 0.0.0.0:8481 0.0.0.0:* LISTEN 22936/java

tcp 0 0 0.0.0.0:8485 0.0.0.0:* LISTEN 22936/java

tcp 0 0 10.67.169.48:8485 10.67.169.49:49628 ESTABLISHED 22936/java

When i change the dfs.http.policy to HTTP_AND_HTTPS, the namenodes come up as active an standby and i am able to open

http://<nnHOSTNAME>:50070

but in any case, the secured service is not working.

I have created the certificates on each host of the cluster giving the FQDN as the common name.Still i feel I made some mistake with keystore and truststore configuration.

[#6]

@Jay SenSharma

i checked the ports, they are open. But after enabling SSL, shouldn't the port change to 50470?

Namenodes are also up.