question Sqoop GC overhead limit exceeded after CDH5.2 update in Archives of Support Questions (Read Only)

Sqoop GC overhead limit exceeded after CDH5.2 update

juergen_thomann — Fri, 16 Sep 2022 09:10:20 GMT

Hi,

we updated sqoop from cdh5.0.1 to cdh5.2 and now it fails everytime with a GC overhead limit exceeded error.

The old version was able to import over 14GB of data over one mapper and the import fails now when a mapper gets too many rows. I checked a heap dump and the memory was completely used by over 3.5 million rows of data (-Xmx 1700M).

The connector is mysql-jdbc version 5.1.33 and the job imports the data as text file in a have table.

Can I avoid this with a setting or is this a bug that should go to jira?

Thank you,

Jürgen

Re: Sqoop GC overhead limit exceeded after CDH5.2 update

sveiss — Thu, 23 Oct 2014 03:26:48 GMT

This appears to be a regression caused by the fix in SQOOP-1400. Instead of fetching results from MySQL row-by-row, sqoop is instead attempting to load the entire result set in memory.

We worked around it by upgrading to MySQL/J Connector 5.1.33 (which you're already on), and then including "--fetch-size -2147483648" in our sqoop command line options list. This restores the old row-by-row behaviour (the weird fetch size is a sentinel value recognised by the MySQL JDBC driver.)

Re: Sqoop GC overhead limit exceeded after CDH5.2 update

juergen_thomann — Thu, 23 Oct 2014 07:33:03 GMT

Thanks for the answer!

I also found the workaround after some time, but you were faster to post it. I'll open a Jira for it, that it will be fixed in new versions.

Re: Sqoop GC overhead limit exceeded after CDH5.2 update

Snd — Sun, 22 Jul 2018 16:59:32 GMT

Use ?dontTrackOpenResources=true&defaultFetchSize=1000&useCursorFetch=true property in Mysql Connection string. It work without changing JVM parameter.

Re: Sqoop GC overhead limit exceeded after CDH5.2 update

Radhe — Sun, 22 Jul 2018 17:05:50 GMT

Thanks a lot! It worked forme also

Re: Sqoop GC overhead limit exceeded after CDH5.2 update

wing — Sat, 09 Mar 2019 09:43:55 GMT

@Snd wrote:
Use ?dontTrackOpenResources=true&defaultFetchSize=1000&useCursorFetch=true property in Mysql Connection string. It work without changing JVM parameter.

Thank you! it's worked!!!!