sqoop2 always results in

[#1]

With CDH 5.x configured ar dfs.replication = 3 Sqoop2 as well as Hive file writing alwys results in the following exception, both at command line and inside HUE.


Exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): file /user/sqoop2/.staging/job_1411486780629_0017/libjar

Requested replication 1 is less than the required minimum 3

at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.verifyReplication(BlockManager.java:945)

[#2]

Also, on a CDH 6 node cluster with namednodes on different servers and sqoop located on edge servers

All else seams highly functional and ballanced

[#3 · Accepted solution]

This error is because you have configured a minimum required replication rather than a default level of replication.

Some systems, like Sqoop 2, purposefully set a low replication level for temporary files that they aren't worried about losing. With a required minimum replication the namenode will reject these requests as invalid.

The fix is to update the minimum required replication back to 1. Do this by resetting the property dfs.namenode.replication.min.

[#4]

This was right on the money.....thanks