question Re: sqoop2 always results in in Archives of Support Questions (Read Only)

sqoop2 always results in

thebadeye — Tue, 21 Apr 2026 14:00:34 GMT

With CDH 5.x configured ar dfs.replication = 3 Sqoop2 as well as Hive file writing alwys results in the following exception, both at command line and inside HUE.


Exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): file /user/sqoop2/.staging/job_1411486780629_0017/libjar

Requested replication 1 is less than the required minimum 3

at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.verifyReplication(BlockManager.java:945)

Re: sqoop2 always results in

thebadeye — Tue, 30 Sep 2014 17:10:55 GMT

Also, on a CDH 6 node cluster with namednodes on different servers and sqoop located on edge servers

All else seams highly functional and ballanced

Re: sqoop2 always results in

busbey — Tue, 30 Sep 2014 18:26:47 GMT

This error is because you have configured a minimum required replication rather than a default level of replication.

Some systems, like Sqoop 2, purposefully set a low replication level for temporary files that they aren't worried about losing. With a required minimum replication the namenode will reject these requests as invalid.

The fix is to update the minimum required replication back to 1. Do this by resetting the property dfs.namenode.replication.min.

Re: sqoop2 always results in

thebadeye — Tue, 30 Sep 2014 19:29:33 GMT

This was right on the money.....thanks