Support Questions
Find answers, ask questions, and share your expertise

Upgrade from HDP 2.6.2 to 3.0.0 HDFS error Failed to upgrade storage directory

Upgrade from HDP 2.6.2 to 3.0.0 HDFS error Failed to upgrade storage directory

New Contributor

I am trying to upgrade my HDP 2.6.2 to 3.0.0 on a single node standalone cluster. HDP was installed manually, this is not the sandbox. I first upgraded Ambari to, registered HDP 3.0.0 and started the upgrade process. At the RESTART HDFS/DATANODE step, it stalls and eventually times out after 30 retries. The datanode log has the following:

	Caused by: java.nio.file.FileSystemException: /hadoop/hdfs/data/current/BP-398974976- -> /hadoop/hdfs/data/current/BP-398974976- Operation not permitted
        at sun.nio.fs.UnixException.translateToIOException(
        at sun.nio.fs.UnixException.rethrowAsIOException(
        at sun.nio.fs.UnixFileSystemProvider.createLink(
        at java.nio.file.Files.createLink(
        at org.apache.hadoop.fs.HardLink.createHardLink(
        at org.apache.hadoop.hdfs.server.datanode.DataStorage$
        at org.apache.hadoop.hdfs.server.datanode.DataStorage$
        ... 4 more
2019-02-14 14:58:51,152
	ERROR datanode.DataNode ( - Initialization failed for Block pool <registering> (Datanode Uuid 4d5fe213-7fcd-404e-8c50-ffe58b9fbc30) service to Exiting. All specified directories have failed to load.
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(

I created a checkpoint right before starting the upgrade. /hadoop/hdfs/data/current/BP-398974976- does not exists, at least not at this step. How do I correct this or roll back the upgrade? The latter I don't see any option and I cannot restart HDFS from Ambari any longer.