Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

sqoop2 always results in

SOLVED Go to solution
Highlighted

sqoop2 always results in

New Contributor
With CDH 5.x configured ar dfs.replication = 3 Sqoop2 as well as Hive file writing alwys results in the following exception, both at command line and inside HUE.


Exception: org.apache.hadoop.ipc.RemoteException(java.io.IOException): file /user/sqoop2/.staging/job_1411486780629_0017/libjar


Requested replication 1 is less than the required minimum 3

at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.verifyReplication(BlockManager.java:945)

1 ACCEPTED SOLUTION

Accepted Solutions

Re: sqoop2 always results in

Expert Contributor

This error is because you have configured a minimum required replication rather than a default level of replication.

 

Some systems, like Sqoop 2, purposefully set a low replication level for temporary files that they aren't worried about losing. With a required minimum replication the namenode will reject these requests as invalid.

 

The fix is to update the minimum required replication back to 1. Do this by resetting the property dfs.namenode.replication.min.

3 REPLIES 3

Re: sqoop2 always results in

New Contributor

Also, on a CDH 6 node cluster with namednodes on different servers and sqoop located on edge servers

 

All else seams highly functional and ballanced

Re: sqoop2 always results in

Expert Contributor

This error is because you have configured a minimum required replication rather than a default level of replication.

 

Some systems, like Sqoop 2, purposefully set a low replication level for temporary files that they aren't worried about losing. With a required minimum replication the namenode will reject these requests as invalid.

 

The fix is to update the minimum required replication back to 1. Do this by resetting the property dfs.namenode.replication.min.

Re: sqoop2 always results in

New Contributor

This was right on the money.....thanks