question Re: "Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 u in Archives of Support Questions (Read Only)

"Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 upgrade

JesperCelander — Mon, 13 Jun 2016 14:56:29 GMT

After a upgrade using Cloudera Manager and pracels I get these events in the active NameNode log.

WARN BlockStateChange: BLOCK* processReport: Report from the DataNode (<datanodeUuid>) is unsorted. This will cause overhead on the NameNode which needs to sort the Full BR. Please update the DataNode to the same version of Hadoop HDFS as the NameNode (2.6.0-cdh5.7.1).

When checking the version of HDFS on the datanode in the event I can see that it reports the correct version.

# hdfs version
Hadoop 2.6.0-cdh5.7.1
Subversion http://github.com/cloudera/hadoop -r ae44a8970a3f0da58d82e0fc65275fff8deabffd
Compiled by jenkins on 2016-06-01T23:26Z
Compiled with protoc 2.5.0
From source with checksum 298b68dc3b308983f04cb37e8416f13
This command was run using /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/jars/hadoop-common-2.6.0-cdh5.7.1.jar

Has anybody seen this warning before?

You can reproduce the error with "hdfs dfsadmin -triggerBlockReport".

We upgraded from CDH5.5.1 -> CDH5.7.1 using parcels.

Re: "Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 u

Harsh J — Tue, 14 Jun 2016 05:03:38 GMT

Are these new logs that repeat over the day or is it an older entry? If you did a rolling upgrade, then it may be normal to see it during the period of upgrade, cause the sorting of BRs in the DN-side was a feature added only in 5.7.0+. Once the upgrade is complete and all DNs are restarted to use the newer code, you shouldn't observe the log entry anymore.

Also check up your Live Nodes page on the NameNode Web UI to ensure all reporting DNs have their version story right.

Re: "Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 u

JesperCelander — Tue, 14 Jun 2016 07:49:04 GMT

Hi.

These logs repeat over the day, upgraded to "5.7.1" 4 days ago and the Warning still show when the any of the nodes do a blockreport.

Didn't do a rolling upgrade, followed the steps in this article.

http://www.cloudera.com/documentation/enterprise/latest/topics/install_upgrade_to_cdh57_parcels.html

Looking at the NameNode Web UI / live nodes all of them report version "2.6.0-cdh5.7.1".

I'm running NameNode HA.

Re: "Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 u

JesperCelander — Tue, 14 Jun 2016 08:35:26 GMT

Another thing, looking at the "VERSION" file in the DatanNode directories they all have "layoutVersion=-56".

And the NameNode/JournalNode directory "VERSION" file report layoutVersion=-60.

Is this normal, thought they should report the same version?

Re: "Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 u

Harsh J — Tue, 14 Jun 2016 10:42:42 GMT

Thank you for double checking!

The "-56" is the right layout version for DNs, so that's not the issue.
There is instead a bug in that we'd left the sorted field out during
conversion of the BlockReportContext objects:
https://github.com/cloudera/hadoop-common/blob/cdh5.7.1-release/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java#L2981-L2988

Internal report for this identified bug is CDH-40629, and the fix will be
in the next update releases of 5.7.2 onwards and also in the 5.8.x series
from 5.8.0 onwards.

At present this shouldn't harm you, except the feature of HDFS-9260 won't
really be in use and the perf. of NN would be the same as 5.6.x and lower.

Re: "Report from the DataNode (datanodeUuid) is unsorted" events after 5.5.1 -> 5.7.1 u

JesperCelander — Tue, 14 Jun 2016 13:53:06 GMT

Hi.

Thank you for your support on this issue!
We will proceed to upgrade our production cluster to 5.7.1 now that we know the details for this log entry.