We want to troubleshoot Sqoop export action that is failing randomly (like once in 10 days) because mapper task is exceeding its memory limit. The job runs successfully 99% of times with same config. Interestingly if failed job is rerun without any changes it always runs successfully.
What we looked at so far:
We understand that fix is to increase mapper memory limit but:
1) Are you doing any ETL process before loading data into target DB?
2) Is it failing in Map stage or reducer stage?
4) Ideally required mappers are 10 to process your 900 K to 1 M records. Is there any specific reason why you are setting mappers explicitly? YARN will assign the mappers appropriately.
4) Can you attach your sqoop log?
5) attach your sqoop export script?