DiskBlockManager throwing exception "Failed to create local dir" when trying to broadcast variable

[#1]

Hello,

I am getting a java.io.IOException: Failed to create local dir in /mnt/tmp/blockmgr ...

This exception is thrown when I am trying to create a broadcast variable(This is on a 6 node cluster). 

This error is not happening consistently. Sometimes it runs fine and some time it fails. I checked the cluster and there is enough disk space left(more than 500GB) and I have permissions to write to /tmp folder. [The exception is not thrown every run].

Any sugestions on this?

Thanks in advance

[#2]

Hello arundh93, Even I am facing the same issue.
Did you get any solution yet ?

[#3]

Hi, 

Please check your Network Configuration. You might be seeing mutiple random jobs hitting your cluster and failing. So while confuguring your network permissions, allow only that IP from which you are doing SSH. It has resolved the same problem for me and you may do the same.

Thanks!!