@mabilgen Cause Root cause is due to condition where JournalNode identifies that promised epoch from the NameNode is not newest, on the contrary it sees that the RPC request from NameNode has lesser epoch value than the locally stored promised epoch . Therefore is throws the warning - "IPC's epoch 155 is less than the last promised epoch 156" . JournalNode therefore reject this RPC request from NameNode to avoid the split-brain. It will accept the RPC request from the NameNode which is send the RPC request with the newest epoch. This can be caused due to various reasons in the environment. 1.) any big job. 2.) any network issue. 3.) not enough resources on node Instructions To resolve the issue we need to identify if network is stable in the cluster. Also it is good to see if this issue is happening regularly within any specific NameNode. If yes, then there could some network hardware issue or resource issue with this NameNode. Few tuneables could be: a.) We can try to raise the number of dfs.datanode.max.transfer.threads from 4K to 16K and observe the performance on the cluster. b.) Also, we can raise the heapsize of NameNode to a higher value if there is too many GC activities or too long Full GC.
... View more
