When you try to connect to one datanode from another, the connection has to be going through 25GbE network. This is most likely a DNS setup issue. Try with ping and nc to see which network is used when connecting between 2 data nodes.
There is more to multi-home configuration than is in the HDFS document, and the more complete discussion may help you resolve your problem, especially wrt DNS and naming. Biggest question being, do the cluster hosts have the same name on all networks? Hope this helps.
Each server should have one consistent hostname on all interfaces. Theoretically, DNS allows a server to have a different hostname per network interface. But it is required for multi-homed Hadoop clusters that each server have only one hostname, consistent among all network interfaces used by Hadoop. (Network interfaces excluded from use by Hadoop may allow other hostnames.)
This is not the case on my environment, but that's OK. We will just register all nodes via 25GbE high speed network for now then.