Answering my own question here, as it might help others Turns out there was a solution: On the server that was being shutdown - I needed to shutdown all the services via the Ambari UI first and then issue the shutdown command after that. Once I did this - hdfs is no longer hanging after the server was shutdown. Strange that not shutting down the services correctly on the server that was shutting down was what was causing the remaining server to hang afterwards - but that was the case. So I'm down to just one issue now: The data is written correctly to hbase but the insert to Elasticsearch is still encountering the no route to host exceptions: This line of code: JavaEsSpark.saveToEs Is it possible to update spark.es.nodes at run-time to remove the ip of the failed node? Or is there away to get "saveToEs" work even when 1 IP in the list is no longer valid? ... View more

Thanks again for replying. Its a spark streaming job, so it runs 24 7. Also there is no issue with data loss - before the shutdown the replication is working correctly between the 2 servers and this has been verified. The 4 VMS of each of the G10 servers are running different services - not the same! So for example there are only 2 HBase VMs, one on each G10 server - so if one server is shutdown you will only loose one HBase VM and the data is fully replicated across both HBase VMs. Again every service on each G10 server is also replicated on the other - its a deliberate design that ALL services are fully replicated across both G10 servers for high availability. It hangs on a hbase insert in the spark streaming job and just doesn’t move any further - unless I turn back on the shutdown VMs and everything returns to normal and the job actually "unhangs" & continues from where it was (which is great). There will obviously be a back log of spark batches to process. When the VMs are shutdown and the spark streaming job is hanging - there are no errors being logged in either the spark logs or the hbase logs. But I tracked the NoRouteToHostExceptions down to the HDFS logs - it is not just logged once, but it keeps logging them at certain intervals until I turn the shutdown VM back on. I was trying to keep my description concise, to avoid making it sound overly complicated, but you don’t have to shutdown the entire G10 server to replicate the issue, if you just shutdown one of the HBase VMs the same thing happens. I'm not actually encountering any loss of data, only issue is that when a node in the hadoop cluster is shutdown and its IP no longer exists in the network - the remaining nodes are hanging indefinitely with NoRouteToHostExceptions So my issue can be summarised down to this: When a node in a Hadoop(HDFS) cluster is shutdown (& its IP address no longer exists) and HDFS on the remaining node is hanging indefinitely with NoRouteToHostExceptions - is there a way to fix this? (Or maybe not and that is ok too, as at least then I can explain what the issue is and why it cant be fixed) Thanks ... View more

Thanks for your very quick reply When the 2 servers/8 VMS are running - everything is fine. The system needs to be fully fault tolerant if one servers fails - the testers simulate this by shutting down one of the G10 servers. The system needs to continue as normal for an extended period of time while the server is shutdown (It will not be rebooted straight away - this is to simulate a failure in prod, where the server might be offline for a few hours for what ever reason) When the G10 server is shutdown, the IPs of the 4 VMs that were shutdown no longer exist on the network and that is what I mean by invalid IP - the HDFS services on the remaining 4 VMs on the remaining server hang with NoRouteToHost exceptions as they can no longer route TCP traffic to the VMs that were shutdown. I left it for a few hours, so at the moment the system does not recover from these NoRouteToHost exceptions. When I edited the IPs in /etc/hosts file and changed them to valid ips (other test server running on the network) it varied between 3 and 15 minutes for the system to return to normal. ... View more