Support Questions
Find answers, ask questions, and share your expertise

Does incremental.last.value update when Sqoop job fails?

Highlighted

Does incremental.last.value update when Sqoop job fails?

Explorer

I need to confirm whether the value for incremental.last.value is updated when my Sqoop job fails to import to HDFS.

2 REPLIES 2
Highlighted

Re: Does incremental.last.value update when Sqoop job fails?

Hi @Kausha Simpson, No, the incremental import state is updated in the Sqoop metastore only after all MR job(s) started by Sqoop finish successfully. So, if the Sqoop doesn't rich that point, the job data won't be updated. See below the last several lines of output of an "--incremental lastmodified" Sqoop import job I did back in January.

... Preceded by the MR job output...
File Output Format Counters
Bytes Written=153893
16/01/24 00:28:31 INFO mapreduce.ImportJobBase: Transferred 150.2861 KB in 22.651 seconds (6.6349 KB/sec)
16/01/24 00:28:31 INFO mapreduce.ImportJobBase: Retrieved 5000 records.
16/01/24 00:28:31 INFO tool.ImportTool: Saving incremental import state to the metastore
16/01/24 00:28:31 INFO tool.ImportTool: Updated data for job: incjob
Highlighted

Re: Does incremental.last.value update when Sqoop job fails?

Hi @Predrag Minovic

I am using Hortonworks 2.5.3. In my sqoop job, the incremental.last.value in sqoop metastore is updating even if the sqoop job fails.

18/01/26 14:26:50 ERROR tool.ImportTool: Merge MapReduce job failed!

18/01/26 14:26:50 INFO tool.ImportTool: Saving incremental import state to the metastore

18/01/26 14:26:51 INFO tool.ImportTool: Updated data for job: job1

When I run the same job again, it is taking the last value as updated value.