Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

sqoop2 always results in

avatar
Explorer
With CDH 5.x configured ar dfs.replication = 3 Sqoop2 as well as Hive file writing alwys results in the following exception, both at command line and inside HUE.


Exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): file /user/sqoop2/.staging/job_1411486780629_0017/libjar


Requested replication 1 is less than the required minimum 3

at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.verifyReplication(BlockManager.java:945)

1 ACCEPTED SOLUTION

avatar
Expert Contributor

This error is because you have configured a minimum required replication rather than a default level of replication.

 

Some systems, like Sqoop 2, purposefully set a low replication level for temporary files that they aren't worried about losing. With a required minimum replication the namenode will reject these requests as invalid.

 

The fix is to update the minimum required replication back to 1. Do this by resetting the property dfs.namenode.replication.min.

View solution in original post

3 REPLIES 3

avatar
Explorer

Also, on a CDH 6 node cluster with namednodes on different servers and sqoop located on edge servers

 

All else seams highly functional and ballanced

avatar
Expert Contributor

This error is because you have configured a minimum required replication rather than a default level of replication.

 

Some systems, like Sqoop 2, purposefully set a low replication level for temporary files that they aren't worried about losing. With a required minimum replication the namenode will reject these requests as invalid.

 

The fix is to update the minimum required replication back to 1. Do this by resetting the property dfs.namenode.replication.min.

avatar
Explorer

This was right on the money.....thanks