Reply
Explorer
Posts: 116
Registered: ‎09-14-2016
Accepted Solution

sqoop stuck at the end

Hi,

 

I have this very weird issue, using sqoop to pulling data from oracle to hdfs/hive, i have three jobs running, total records is 312,163,901, but it stops at 312,160,000, the jobs show as running, but it hangs and nohting in the log and no update.

 

A different table works fine.

 

To me somehow the last 3901 got stuck, can someone help or give some hints?

 

Thanks

Shannon

 

 

Posts: 376
Topics: 11
Kudos: 58
Solutions: 32
Registered: ‎09-02-2016

Re: sqoop stuck at the end

@ponypony

 

This is due to java heap space issue. 

 

Pls try the below steps

1. Check the current mapreduce.map.memory.mb and mapreduce.reduce.memory.mb (there are different ways to check - you can use either Cloudera manager -> Yarn -> configuration (or) from hive/beeline CLI (or) mapred-site.xml or yarn-site.xml )

2. Increase (1 or 2 GB) java heap space temporarly,  i've already shared the details in the below link. NOTE: The below link refers to different issue but you can use this soluation for your issue too as sqoop uses  MR

 

http://community.cloudera.com/t5/Hadoop-101-Training-Quickstart/Map-and-Reduce-Error-Java-heap-space...

 

3. Try the sqoop now, if the issue fixed then work with your admin and increase it permanently 

 

Highlighted
Explorer
Posts: 116
Registered: ‎09-14-2016

Re: sqoop stuck at the end

Thanks!

Announcements