"Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 upgrade

[#1]

After a upgrade using Cloudera Manager and pracels I get these events in the active NameNode log. 

WARN BlockStateChange: BLOCK* processReport: Report from the DataNode (<datanodeUuid>) is unsorted. This will cause overhead on the NameNode which needs to sort the Full BR. Please update the DataNode to the same version of Hadoop HDFS as the NameNode (2.6.0-cdh5.7.1).

When checking the version of HDFS on the datanode in the event I can see that it reports the correct version. 

# hdfs version
Hadoop 2.6.0-cdh5.7.1
Subversion http://github.com/cloudera/hadoop -r ae44a8970a3f0da58d82e0fc65275fff8deabffd
Compiled by jenkins on 2016-06-01T23:26Z
Compiled with protoc 2.5.0
From source with checksum 298b68dc3b308983f04cb37e8416f13
This command was run using /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/jars/hadoop-common-2.6.0-cdh5.7.1.jar

Has anybody seen this warning before?

You can reproduce the error with "hdfs dfsadmin -triggerBlockReport".

We upgraded from CDH5.5.1 -> CDH5.7.1 using parcels. 

[#2]

Are these new logs that repeat over the day or is it an older entry? If you did a rolling upgrade, then it may be normal to see it during the period of upgrade, cause the sorting of BRs in the DN-side was a feature added only in 5.7.0+. Once the upgrade is complete and all DNs are restarted to use the newer code, you shouldn't observe the log entry anymore.

Also check up your Live Nodes page on the NameNode Web UI to ensure all reporting DNs have their version story right.

[#3]

Hi. 

These logs repeat over the day, upgraded to "5.7.1" 4 days ago and the Warning still show when the any of the nodes do a blockreport. 

Didn't do a rolling upgrade, followed the steps in this article. 

http://www.cloudera.com/documentation/enterprise/latest/topics/install_upgrade_to_cdh57_parcels.html

Looking at the NameNode Web UI / live nodes all of them report version "2.6.0-cdh5.7.1". 

I'm running NameNode HA. 

[#4]

Another thing, looking at the "VERSION" file in the DatanNode directories they all have "layoutVersion=-56".

And the NameNode/JournalNode directory "VERSION" file report layoutVersion=-60.

Is this normal, thought they should report the same version?

[#5 · Accepted solution]

Thank you for double checking!

The "-56" is the right layout version for DNs, so that's not the issue.
There is instead a bug in that we'd left the sorted field out during
conversion of the BlockReportContext objects:
https://github.com/cloudera/hadoop-common/blob/cdh5.7.1-release/hadoop-hdfs-project/hadoop-hdfs/src/...

Internal report for this identified bug is CDH-40629, and the fix will be
in the next update releases of 5.7.2 onwards and also in the 5.8.x series
from 5.8.0 onwards.

At present this shouldn't harm you, except the feature of HDFS-9260 won't
really be in use and the perf. of NN would be the same as 5.6.x and lower.

[#6]

Hi.

Thank you for your support on this issue!
We will proceed to upgrade our production cluster to 5.7.1 now that we know the details for this log entry.