Created on 04-14-2018 09:01 AM - edited 08-17-2019 08:30 PM
I installed Ambari, but cannot start ResourceManager. I followed different indications that I found in this forum, but nothing helped so far. In particular, UI shows the following error :
"Connection failed to http://eureambarislave1.local.eurecat.org:8088 (<urlopen error [Errno 111] Connection refused>)"
But the port 8088 is opened and proxy is setup.
The output of "jps" in ResourceManager host:
# jps
9953 QuorumPeerMain
15683 RMAdminCLI
17508 NodeManager
15813 RMAdminCLI
19881 RunJar
11435 RMAdminCLI
4428 RunJar
16428 RMAdminCLI
23535 SecondaryNameNode
11760 RMAdminCLI
19313 Jps
5811 RunJar
15923 RMAdminCLI
16053 RMAdminCLI
11319 RMAdminCLI
17241 ApplicationHistoryServer
27546 JobHistoryServer
11675 RMAdminCLI
22687 DataNode
11519 RMAdminCLI
Is it fine that I have so many processes of "RMAdminCLI"?
Also, I cannot open http://xxx.xx.xx.xx:8088 in the browser.
Created 04-16-2018 10:13 AM
I solved this problem by opening the ports 2888 and 3888 that are used by ZooKeeper nodes for communicating between each other.
Created 04-14-2018 09:06 AM
I found ResourceManager logs in "/var/log/hadoop-yarn"
2018-04-12 22:41:29,700 INFO resourcemanager.ResourceManager (LogAdapter.java:info(45)) -
STARTUP_MSG: /************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG: user = yarn
STARTUP_MSG: host = eureambarislave1.local.eurecat.org/192.168.0.10
STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3.2.6.4.0-91
Created 04-15-2018 03:07 PM
@Liana Napalkova The jps output doesn't show ResourceManager process. RMAdminCli is the process which runs when "yarn rmadmin" command is executed. Not exactly sure who is executing it, but that shouldn't be a problem. Is ResourceManager service showing started in ambari?
Created 04-15-2018 09:19 PM
The "jps" command shows that ResourceManager is running. However, there is an alert next to ResourceManager that says "Connection failed to http://eureambarislave2.local.eurecat.org:8088 (<urlopen error [Errno 111] Connection refused>). I restarted everything several times and checked that the port 8088 is opened. But the error still exists. I cannot access ResourceManager in a browser.
Created 04-15-2018 09:32 PM
By the way, this is how I opened the port 8088:
iptables -I INPUT 1 -p tcp --dport 8088 -j ACCEPT
Also, I created a TCP security rule for 8088.
What do I miss?
Created 04-16-2018 03:01 AM
Can you check if ResourceManager is listening on port 8088 as follows.
ps -ef | grep `lsof -i :8088 | grep -i listen | awk '{print $2}'`
If the command returns no output, then we can check for errors in resourcemanager logs. By default the location is /var/log/hadoop-yarn/yarn/
Created 04-16-2018 08:54 AM
I get the following output after running this command:
root 811 31804 0 08:51 pts/0 00:00:00 grep --color=auto lsof -i :8088 | grep -i listen | awk '{print }'
But the command "netstat -tunalp | grep LISTEN" does not show the port 8088.
Created 04-16-2018 09:18 AM
I think you typed in ' quote instead of ` . Can you give the full output of the following commands instead.
jps -l | grep -i resourcemanager netstat -anp | grep 8088
Created 04-16-2018 09:27 AM
This is what I get:
# jps -l | grep -i resourcemanager
4274 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
# netstat -anp | grep 8088 [root@eureambarislave2 sbin]
#
The netstat returns nothing.
Created 04-16-2018 09:27 AM
Created 04-16-2018 09:41 AM
No, I do not have ResourceManager HA enabled.
This is the content of "yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.log":
2018-04-15 23:13:33,362 INFO resourcemanager.ResourceManager (LogAdapter.java:info(45)) -
STARTUP_MSG: /************************************************************
STARTUP_MSG: Starting ResourceManager
STARTUP_MSG: user = yarn
STARTUP_MSG: host = eureambarislave2.local.eurecat.org/192.168.0.15 STARTUP_MSG: args = []
STARTUP_MSG: version = 2.7.3.2.6.4.0-91
And this is the content of "yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.out":
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. (-i) 64019
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 65536
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Created 04-16-2018 09:41 AM
Was that the entire content? Can you upload the file yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.log?
Created 04-16-2018 09:54 AM
Sorry, yes, there is more content in LOG file. Please see the error messages below. There seem to be a problem with ZooKeeper, but I do not have any alerts for ZooKeeper.
Zookeeper seems to run well:
# jps -l | grep -i zookeeper
5043 org.apache.zookeeper.server.quorum.QuorumPeerMain
# netstat -anp | grep 2181
tcp6 0 0 :::2181 :::* LISTEN 5043/java
Errors log:
2018-04-16 09:16:23,821 ERROR resourcemanager.ResourceManager (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-04-16 09:16:25,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server eureambarislave1.local.eurecat.org/192.168.0.10:2181. Will not attempt to authenticate using SASL (unknown error)
2018-04-16 09:16:25,316 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established, initiating session, client: /192.168.0.15:53808, server: eureambarislave1.local.eurecat.org/192.168.0.10:2181
2018-04-16 09:16:25,316 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-04-16 09:16:25,417 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZK operation. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:326)
Created 04-16-2018 09:13 AM
By the way, I opened the port 8088 as follows:
iptables -I INPUT 1 -p tcp --dport 8088 -j ACCEPT
Is it correct?
Created 04-16-2018 10:13 AM
I solved this problem by opening the ports 2888 and 3888 that are used by ZooKeeper nodes for communicating between each other.