ResourceManager is started with alerts and ip:8088 is not accessible in the browser.

[#1]

I installed Ambari, but cannot start ResourceManager. I followed different indications that I found in this forum, but nothing helped so far. In particular, UI shows the following error :

"Connection failed to http://eureambarislave1.local.eurecat.org:8088 (<urlopen error [Errno 111] Connection refused>)"

But the port 8088 is opened and proxy is setup.

The output of "jps" in ResourceManager host:

# jps

9953 QuorumPeerMain

15683 RMAdminCLI

17508 NodeManager

15813 RMAdminCLI

19881 RunJar

11435 RMAdminCLI

4428 RunJar

16428 RMAdminCLI

23535 SecondaryNameNode

11760 RMAdminCLI

19313 Jps

5811 RunJar

15923 RMAdminCLI

16053 RMAdminCLI

11319 RMAdminCLI

17241 ApplicationHistoryServer

27546 JobHistoryServer

11675 RMAdminCLI

22687 DataNode

11519 RMAdminCLI

Is it fine that I have so many processes of "RMAdminCLI"?

Also, I cannot open http://xxx.xx.xx.xx:8088 in the browser.

[#2]

I found ResourceManager logs in "/var/log/hadoop-yarn"

2018-04-12 22:41:29,700 INFO resourcemanager.ResourceManager (LogAdapter.java:info(45)) -

STARTUP_MSG: /************************************************************

STARTUP_MSG: Starting ResourceManager

STARTUP_MSG: user = yarn

STARTUP_MSG: host = eureambarislave1.local.eurecat.org/192.168.0.10

STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.3.2.6.4.0-91

[#3]

@Liana Napalkova The jps output doesn't show ResourceManager process. RMAdminCli is the process which runs when "yarn rmadmin" command is executed. Not exactly sure who is executing it, but that shouldn't be a problem. Is ResourceManager service showing started in ambari?

[#4]

The "jps" command shows that ResourceManager is running. However, there is an alert next to ResourceManager that says "Connection failed to http://eureambarislave2.local.eurecat.org:8088 (<urlopen error [Errno 111] Connection refused>). I restarted everything several times and checked that the port 8088 is opened. But the error still exists. I cannot access ResourceManager in a browser.

[#5]

By the way, this is how I opened the port 8088:

iptables -I INPUT 1 -p tcp --dport 8088 -j ACCEPT

Also, I created a TCP security rule for 8088.

What do I miss?

[#6]

Can you check if ResourceManager is listening on port 8088 as follows.

ps -ef | grep `lsof -i :8088 | grep -i listen | awk '{print $2}'`

If the command returns no output, then we can check for errors in resourcemanager logs. By default the location is /var/log/hadoop-yarn/yarn/

[#7]

I get the following output after running this command:

root 811 31804 0 08:51 pts/0 00:00:00 grep --color=auto lsof -i :8088 | grep -i listen | awk '{print }'

But the command "netstat -tunalp | grep LISTEN" does not show the port 8088.

[#8]

I think you typed in ' quote instead of ` . Can you give the full output of the following commands instead.

jps -l | grep -i resourcemanager

netstat -anp | grep 8088

[#9]

This is what I get:

# jps -l | grep -i resourcemanager

4274 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

# netstat -anp | grep 8088 [root@eureambarislave2 sbin]

#

The netstat returns nothing.

[#10]

Do you see any errors in Resourcemanager logs? They are usually at /var/log/hadoop-yarn/yarn/ . Do you have ResourceManager HA enabled?

[#11]

No, I do not have ResourceManager HA enabled.
This is the content of "yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.log":

2018-04-15 23:13:33,362 INFO resourcemanager.ResourceManager (LogAdapter.java:info(45)) -

STARTUP_MSG: /************************************************************

STARTUP_MSG: Starting ResourceManager

STARTUP_MSG: user = yarn

STARTUP_MSG: host = eureambarislave2.local.eurecat.org/192.168.0.15 STARTUP_MSG: args = []

STARTUP_MSG: version = 2.7.3.2.6.4.0-91

And this is the content of "yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.out":

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. (-i) 64019

max locked memory (kbytes, -l) 64

max memory size (kbytes, -m) unlimited

open files (-n) 32768

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 8192

cpu time (seconds, -t) unlimited

max user processes (-u) 65536

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

[#12]

Was that the entire content? Can you upload the file yarn-yarn-resourcemanager-eureambarislave2.local.eurecat.org.log?

[#13]

Sorry, yes, there is more content in LOG file. Please see the error messages below. There seem to be a problem with ZooKeeper, but I do not have any alerts for ZooKeeper.

Zookeeper seems to run well:

# jps -l | grep -i zookeeper

5043 org.apache.zookeeper.server.quorum.QuorumPeerMain

# netstat -anp | grep 2181

tcp6 0 0 :::2181 :::* LISTEN 5043/java

Errors log:
2018-04-16 09:16:23,821 ERROR resourcemanager.ResourceManager (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM

2018-04-16 09:16:25,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server eureambarislave1.local.eurecat.org/192.168.0.10:2181. Will not attempt to authenticate using SASL (unknown error)

2018-04-16 09:16:25,316 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established, initiating session, client: /192.168.0.15:53808, server: eureambarislave1.local.eurecat.org/192.168.0.10:2181

2018-04-16 09:16:25,316 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1142)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect

2018-04-16 09:16:25,417 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZK operation. org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:326)

[#14]

By the way, I opened the port 8088 as follows:

iptables -I INPUT 1 -p tcp --dport 8088 -j ACCEPT

Is it correct?

[#15 · Accepted solution]

I solved this problem by opening the ports 2888 and 3888 that are used by ZooKeeper nodes for communicating between each other.