I need to confirm whether the value for incremental.last.value is updated when my Sqoop job fails to import to HDFS.
Hi @Kausha Simpson, No, the incremental import state is updated in the Sqoop metastore only after all MR job(s) started by Sqoop finish successfully. So, if the Sqoop doesn't rich that point, the job data won't be updated. See below the last several lines of output of an "--incremental lastmodified" Sqoop import job I did back in January.
... Preceded by the MR job output... File Output Format Counters Bytes Written=153893 16/01/24 00:28:31 INFO mapreduce.ImportJobBase: Transferred 150.2861 KB in 22.651 seconds (6.6349 KB/sec) 16/01/24 00:28:31 INFO mapreduce.ImportJobBase: Retrieved 5000 records. 16/01/24 00:28:31 INFO tool.ImportTool: Saving incremental import state to the metastore 16/01/24 00:28:31 INFO tool.ImportTool: Updated data for job: incjob
Hi @Predrag Minovic
18/01/26 14:26:50 ERROR tool.ImportTool: Merge MapReduce job failed!
18/01/26 14:26:50 INFO tool.ImportTool: Saving incremental import state to the metastore
18/01/26 14:26:51 INFO tool.ImportTool: Updated data for job: job1
When I run the same job again, it is taking the last value as updated value.