Reply
Contributor
Posts: 96
Registered: ‎05-19-2016

All node managers down

My resource managers are active and so is Job history server. All my worker nodes had been exiting randomly for some time but used to restart automatically. today, all my node managers are down. what could be the reason? My worker nodes are typical with Hdfs And yarn on them. hdfs is running fine. what does it indicate when all node managers are down? There was no unusual load on servers. also, if i restart them, it still goes down. please suggest what could cause this?

New Contributor
Posts: 2
Registered: ‎03-20-2017

Re: All node managers down

Check network connection between node managers and cloudera manager, this could a network issue. try to do a 100 MB file transfer between trouble hosts and healthy hosts, compare time between them.

if file transfer between nodes (with node manager down) is taking longer than expected, you have to contact your network team to check network switch connecting those nodes.

Contributor
Posts: 96
Registered: ‎05-19-2016

Re: All node managers down

But in that case, HDFS would be down as well, No? HDFS is installed on same servers as Node managers are and HDFS is working fine without any warnings or errors

Contributor
Posts: 96
Registered: ‎05-19-2016

Re: All node managers down

Also, Node manager continue to exit even if it is on same node as CM

New Contributor
Posts: 2
Registered: ‎03-20-2017

Re: All node managers down

i faced similar issue with 'impalad'. where there was issue with network switch issue.

i suggest its worth trying.

Announcements