Created on
08-19-2019
08:55 AM
- last edited on
08-19-2019
09:48 AM
by
cjervis
INFO webapp.ApiServer (ApiServer.java:getService(197)) - GET: getService for appName = ats-hbase user = yarn (auth:SIMPLE)
2019-08-1ROR webapp.ApiServer (ApiServer.java:getService(213)) - Get service failed: {}necting to ResourceManager at <hostname>/IP:8050
Created 08-19-2019 09:51 AM
Thank you for posting to the community @kwabstian53. Could you provide some further context around your question so other community members may be better able to assist?
Created 08-19-2019 10:00 AM
resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/var/lib/ambari-agent/tmp/hdfs-service-check -H 'Content-Type: application/octet-stream' 'http://localhost:50070/webhdfs/v1/tmp/id940a6b28_date521919?op=CREATE&user.name=hdfs&overwrite=True' 1>/tmp/tmpbc0udk 2>/tmp/tmpOdlga7' returned 7. curl: (7) Failed connect to thdfc-slave-03:50075; Connection timed out 307
Created on 08-19-2019 07:35 PM - edited 08-19-2019 08:48 PM
I am not sure how your recently reported error is linked to the originally reported error in this thread.
However i will try to respond to your later error (as earlier error does not have much context)
We see this error:
resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/var/lib/ambari-agent/tmp/hdfs-service-check -H 'Content-Type: application/octet-stream' 'http://localhost:50070/webhdfs/v1/tmp/id940a6b28_date521919?op=CREATE&user.name=hdfs&overwrite=True' 1>/tmp/tmpbc0udk 2>/tmp/tmpOdlga7' returned 7. curl: (7) Failed connect to thdfc-slave-03:50075; Connection timed out 307
Which indicates that a simple webhdfs call made to NameNode "localhost:50070" is failing because during the webhdfs call when it is trying to connect to a DataNode in order to actually create a data that time it is not able to get a response from the DataNode running/listening to "thdfc-slave-03:50075".
NameNode host is not able to connect to DataNode on port 50075 due to "Connection timed out 307". Which indicates some communication/Network issue.
1. So please check if that DataNode is actually running or do we see any error in the DataNode log on that host?
2. On the DataNode host please check if the port "50075" is actually opened and listening? Or if the Firewall/IPTables is disabled ?
# ps -ef | grep -i DataNode
# netstat -tnlpa | greop $DATANODE_PID
# service iptables status
# systemctl status firewalld
3. From the Active NameNode host where you see this error please try to run the following telnet command in order to verify the port access (just to isolate any firewall issue)
# telnet thdfc-slave-03 50075
(OR)
# nc -v thdfc-slave-03 50075
.
.
Created 08-20-2019 07:40 AM
When I run hdfs service check , resource_management.core.exceptions.ExecutionFailed: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' -k 'https://thdfc-master-01:50470/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmp3KhXGG 2>/tmp/tmpu8iQ0r' returned 35. curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s). 000 INFO ipc.Server (Server.java:logException(2726)) - IPC Server handler 0 on 8020, call Call#132 Retry#0 org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.getTransactionId from 10.115.40.47:58433 org.apache.hadoop.ipc.RetriableException: NameNode still not started at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.checkNNStartup(NameNodeRpcServer.java:2210) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getTransactionID(NameNodeRpcServer.java:1274) at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.getTransactionId(NamenodeProtocolServerSideTranslatorPB.java:120) at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:12970) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
Created 08-20-2019 07:43 AM
etstat -tnlpa |grep 22682 tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 22682/java tcp 0 0 105.0.0.1:24041 0.0.0.0:* LISTEN 22682/java tcp 0 0 0.0.0.0:8010 0.0.0.0:* LISTEN 22682/java tcp 0 0 0.0.0.0:50475 0.0.0.0:* LISTEN 22682/java tcp 0 0 10.115.40.20:63590 10.115.40.107:8020 ESTABLISHED 22682/java