Support Questions
Find answers, ask questions, and share your expertise

Ambari problems with PORT type alerts

Ambari problems with PORT type alerts

Explorer

Hi,

I am currently testing the wire encryption of a HDP cluster. Basically most of the stuff is working as it should. But there is Ambari Alerts.

Since I activated SSL and encrypted RPC for HDFS, HBase... etc I have some warnings or errors in the logs file of theses services. For example HDFS (Datanode, RPC) you can see the following message in the data node's log file :

 


>[2018-08-01 14:49:21,951][HDFS][ERROR][org.apache.hadoop.hdfs.server.datanode.DataNode][org.apache.hadoop.hdfs.server.datanode.DataXceiver@3ccb86f1][DataXceiver:run] zei332sd.distrb.example.com:1019:DataXceivererror processing unknown operation src: /10.122.139.77:36924 dst: /10.122.139.77:1019java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:361) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:180) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:112) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:194) at java.lang.Thread.run(Thread.java:748) >

 

Which is quite understandable when checking the code. This is caused by the alert "data node process". It seems this can only be handled by fixing the data node's code and checking if the stream has data to read. Any other hints?

Another problem is with the HTTP services (HBase for example):

 


>[2018-08-01 14:52:21,963][HBASE][WARN][org.mortbay.log][1373191457@qtp-1199833406-2] EXCEPTION javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.InputRecord.read(InputRecord.java:505) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ... 5 more >

 

This is caused by the "hbase region server process" alert. This alert can be disabled since there's another one checking the same port but using the WEB type.

There is no problem with the RPC port of HBase since HBase does check if the stream isn't closed before doing the handshake.

I also found this error (HTTP / HTTPS) in the history server logs. I'll post them as well.

Yarn seems to be OK.

Has someone seen theses warnings / errors?

Since ambari is checking quite often the log files are filled with unreadable lines (making the rest less understandable)

I am currently working on custom alerts to replace the port type once that cause the warnings/errors. I am searching for a solution with the data nodes.

Kind regards

Manfred