issue in importing bulk data from oracle

[#1]

hi,

I enter this command to import some data from oracle. It is ok and the result has 1.3 million records.

sqoop-import --connect jdbc:oracle:thin:@//serverIP:Port/xxxx   --query "SELECT col1,col2,col3 FROM table where condition AND \$CONDITIONS "  --target-dir /user/root/myresult --split-by col1 -m 10 --username xxx --password xxx

but when I delete the condition to import whole table which has 12million records, It fails.

always the first maps are loged as succeeded and the last one just hangs. but when I check mapreduce logs for succeeded maps, I see that they have been fail with this message:

container killed by the applicationmaster. container killed on request. exit code is 143 container exited with a non-zero exit code 143.

I googled and I found

https://stackoverflow.com/questions/42306865/sqoop-job-get-stuck-when-import-data-from-oracle-to-hiv...

as the same issue I have. but this post hasn't been answered yet. It'd be helpful if you take a look.

[#2]

@Sara Alizadeh, The first command was executed with 10 mappers (-m 10) for 1.3 millions and your mappers might be going out of memory if you are using the same number of mappers for 12 million records.

Do increase the number of mappers (i'd say 100) and re-run the job.

[#3]

thanks for your quick reply.

Is there any other solution to accomplish the import with less memory but slower?? My memory resources are limited. about 55G is assigned to yarn.

an other question, what is the proper size of memory for mappers? I googled a lot for this question and I came with that I need to reduce my mappers memory , and increase them in number. as you said, e.g. 100 mappers. take a look at my ref.

does this sound OK to you?

p.s. my mapper mem is 3G and my reducers have 2G.

[#4 · Accepted solution]

I solved this issue

Just posting this to those who may have the same problem.

I strengthen the link between my database and my big data servers. The link was slow so sqoop transmission rate got very low.