question Re: issue in importing bulk data from oracle in Archives of Support Questions (Read Only)

issue in importing bulk data from oracle

alizadeh_uut1 — Sat, 07 Jul 2018 20:17:27 GMT

hi,

I enter this command to import some data from oracle. It is ok and the result has 1.3 million records.

sqoop-import --connect jdbc:oracle:thin:@//serverIP:Port/xxxx   --query "SELECT col1,col2,col3 FROM table where condition AND \$CONDITIONS "  --target-dir /user/root/myresult --split-by col1 -m 10 --username xxx --password xxx

but when I delete the condition to import whole table which has 12million records, It fails.

always the first maps are loged as succeeded and the last one just hangs. but when I check mapreduce logs for succeeded maps, I see that they have been fail with this message:

container killed by the applicationmaster. container killed on request. exit code is 143 container exited with a non-zero exit code 143.

I googled and I found

https://stackoverflow.com/questions/42306865/sqoop-job-get-stuck-when-import-data-from-oracle-to-hive

as the same issue I have. but this post hasn't been answered yet. It'd be helpful if you take a look.

Re: issue in importing bulk data from oracle

sandyy006 — Sun, 08 Jul 2018 01:56:31 GMT

@Sara Alizadeh, The first command was executed with 10 mappers (-m 10) for 1.3 millions and your mappers might be going out of memory if you are using the same number of mappers for 12 million records.

Do increase the number of mappers (i'd say 100) and re-run the job.

Re: issue in importing bulk data from oracle

alizadeh_uut1 — Sun, 08 Jul 2018 15:53:51 GMT

thanks for your quick reply.

Is there any other solution to accomplish the import with less memory but slower?? My memory resources are limited. about 55G is assigned to yarn.

an other question, what is the proper size of memory for mappers? I googled a lot for this question and I came with that I need to reduce my mappers memory , and increase them in number. as you said, e.g. 100 mappers. take a look at my ref.

does this sound OK to you?

p.s. my mapper mem is 3G and my reducers have 2G.

Re: issue in importing bulk data from oracle

alizadeh_uut1 — Sat, 21 Jul 2018 15:06:18 GMT

I solved this issue

Just posting this to those who may have the same problem.

I strengthen the link between my database and my big data servers. The link was slow so sqoop transmission rate got very low.