question Re: sqoop stuck at the end in Archives of Support Questions (Read Only)

sqoop stuck at the end

ponypony — Fri, 16 Sep 2022 11:57:46 GMT

Hi,

I have this very weird issue, using sqoop to pulling data from oracle to hdfs/hive, i have three jobs running, total records is 312,163,901, but it stops at 312,160,000, the jobs show as running, but it hangs and nohting in the log and no update.

A different table works fine.

To me somehow the last 3901 got stuck, can someone help or give some hints?

Thanks

Shannon

Re: sqoop stuck at the end

saranvisa — Fri, 21 Jul 2017 02:34:11 GMT

@ponypony

This is due to java heap space issue.

Pls try the below steps

1. Check the current mapreduce.map.memory.mb and mapreduce.reduce.memory.mb (there are different ways to check - you can use either Cloudera manager -> Yarn -> configuration (or) from hive/beeline CLI (or) mapred-site.xml or yarn-site.xml )

2. Increase (1 or 2 GB) java heap space temporarly, i've already shared the details in the below link. NOTE: The below link refers to different issue but you can use this soluation for your issue too as sqoop uses MR

http://community.cloudera.com/t5/Hadoop-101-Training-Quickstart/Map-and-Reduce-Error-Java-heap-space/m-p/47023#M4622

3. Try the sqoop now, if the issue fixed then work with your admin and increase it permanently

Re: sqoop stuck at the end

ponypony — Tue, 25 Jul 2017 19:43:46 GMT

Thanks!