Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop import suddenly fails after few years

Sqoop import suddenly fails after few years

New Contributor

Hi,

we are importing data to hdfs with sqoop now for a few years.

Yesterday it started to fail with the following error:

19/02/20 05:14:00 INFO mapreduce.Job: Job job_1549278403763_0242 failed with state FAILED due to: Job commit failed: org.apache.hive.hcatalog.common.HCatException : 2012 : Moving of data failed during commit : Could not find a unique destination path for move: file = hdfs://nas.ad.com:8020/apps/hive/warehouse/dbname.db/tablename_SCRATCH0.7991305468219227/part-m-00000 , src = hdfs://nas.ad.com:8020/apps/hive/warehouse/dbname.db/tablename_SCRATCH0.7991305468219227, dest = hdfs://nas.ad.com:8020/apps/hive/warehouse/dbname.db/tablename

at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.getFinalPath(FileOutputCommitterContainer.java:664)

at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.moveTaskOutputs(FileOutputCommitterContainer.java:517)

at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.moveTaskOutputs(FileOutputCommitterContainer.java:606)

at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:781)

at org.apache.hive.hcatalog.mapreduce.FileOutputCommitterContainer.commitJob(FileOutputCommitterContainer.java:252)

at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)

at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

The command we are using is:

/usr/bin/sqoop import --connect "jdbc:sap://IP:30015/?databaseName=TH1&instanceNumber=00&currentschema=schemaname" --username HADOOP --password Init1234 --driver com.sap.db.jdbc.Driver --table BUFFER_INVOICE_HEADER --split-by datetimecreated --hcatalog-database dbname --hcatalog-table tablename --hcatalog-storage-stanza "stored as orc" --null-string null --null-non-string null --map-column-java ORIGINALXML=String --map-column-hive ORIGINALXML=String -m 28

The last file name in hdfs folder is "part-m-00027_a_998"

There are 27968 files in that hdfs directory.

part-m-00000

part-m-00000_a_1

.

.

.

part-m-00027_a_998

--append does not work with HCatalog (Append mode for imports is not compatible with HCatalog. Please remove the parameter--append-mod)

Is there a way to overcome this limitation? According to this post limitation is 999 files, which is also our case.

Don't have an account?
Coming from Hortonworks? Activate your account here