yet another errno 1111 - cant connect to data node

[#1]

this morning one of my datanodes was not sending heartbeats and the whole server seemed to have been locked up. now when it came back i did the usual start up of things through ambari no issues everything came back green. however 2 alerts popped up. one was the Datanode process and the other was the Datanode webui giving me an errno 1111 connect refused can not connect to 0.0.0.0:1022 connection refused

in the past i could connect and see the datanode was working by putting <ip-address>:1022 in my webbrowser and it would take me to a screen showing its running. however its not allowing that anymore. ambari says its started but HDFS is showing a dead node and the webui isnt working. i checked the logs and i cant for the life of me see any type of error. and there is no error in the start up process as well.

so any help on troubleshooting and or fixing this issue would be great.

thanks in advance.

[#2]

Hi @steve coyle, have you upgraded your Linux kernel recently? If so it is likely this issue: https://issues.apache.org/jira/browse/HDFS-12029

You can either rollback the kernel version, or upgrade to a newer kernel that fixes that this regression (assuming your vendor has one), or apply the workaround mentioned in the description of that Jira.