Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Unable to override the existing sqoop job in CDH quickstart VM

Unable to override the existing sqoop job in CDH quickstart VM

Explorer

Hello,


I have created a sqoop job as


[cloudera@quickstart sqoop]$ sqoop job --create orders_import -- import --connect "jdbc:mysql://${hostname}:3306/retail_db" --table orders --target-dir /user/cloudera/cca175/review/sqoop/orders --username root --password cloudera -m 5

 

I have triggered the above sqoop job and successfully completed. But when I tried to override to import the data into avro format with a new file name it throws an exception.

 

[cloudera@quickstart sqoop]$ sqoop job --exec orders_import -- --delete-target-dir --target-dir /user/cloudera/cca175/review/sqoop/orders_as_avro --as-avrodatafile


Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/01/23 14:36:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.0
Enter password:
17/01/23 14:36:38 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/01/23 14:36:38 INFO tool.CodeGenTool: Beginning code generation
17/01/23 14:36:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM orders AS t LIMIT 1
17/01/23 14:36:38 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM orders AS t LIMIT 1
17/01/23 14:36:38 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-cloudera/compile/da508ace8c9dcaa1eae5a2005fb413db/orders.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/01/23 14:36:41 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloudera/compile/da508ace8c9dcaa1eae5a2005fb413db/orders.jar
17/01/23 14:36:41 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/01/23 14:36:41 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/01/23 14:36:41 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/01/23 14:36:41 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/01/23 14:36:41 INFO mapreduce.ImportJobBase: Beginning import of orders
17/01/23 14:36:41 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/01/23 14:36:42 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/01/23 14:36:42 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
17/01/23 14:36:43 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://quickstart.cloudera:8020/user/cloudera/cca175/review/sqoop/orders already exists
17/01/23 14:36:43 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://quickstart.cloudera:8020/user/cloudera/cca175/review/sqoop/orders already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:203)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:176)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:273)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:507)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:213)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:268)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

[cloudera@quickstart sqoop]$

 

 

Is there any limitation in the parameters overriding during the Sqoop JOB?

 

Note: I am using CDH quickstart vm

3 REPLIES 3

Re: Unable to override the existing sqoop job in CDH quickstart VM

Champion
sqoop job --exec jd  -- --delete-target-dir --target-dir /user/matt/stud/studs --as-avrodatafile

My Enviroment  - sqoop-1.4.4-cdh5.0.0

I tried the same as you did but i did get the file already existing error under DEBUG which is ignored .

 

if you dont mind could you share your directory structure. 

also  could please try using -- verbose in your sqoop --exec it will give us some idea and share with us if you can.

what version of sqoop and CDH are you runining in your machine.

 

-rw-r--r--   1 matt supergroup          0 2017-01-24 04:09 /user/matt/stud/_SUCCESS
-rw-r--r--   1 matt supergroup         14 2017-01-24 04:09 /user/matt/stud/part-m-00000
-rw-r--r--   1 matt supergroup         20 2017-01-24 04:09 /user/matt/stud/part-m-00001
-rw-r--r--   1 matt supergroup         24 2017-01-24 04:09 /user/matt/stud/part-m-00002
-rw-r--r--   1 matt supergroup          6 2017-01-24 04:09 /user/matt/stud/part-m-00003
drwxr-xr-x   - matt supergroup          0 2017-01-24 04:23 /user/matt/stud/studs
[matt@localhost ~]$ hadoop fs -ls /user/matt/stud/studs/
Found 5 items
-rw-r--r--   1 matt supergroup          0 2017-01-24 04:23 /user/matt/stud/studs/_SUCCESS
-rw-r--r--   1 matt supergroup        301 2017-01-24 04:23 /user/matt/stud/studs/part-m-00000.avro
-rw-r--r--   1 matt supergroup        307 2017-01-24 04:23 /user/matt/stud/studs/part-m-00001.avro
-rw-r--r--   1 matt supergroup        311 2017-01-24 04:23 /user/matt/stud/studs/part-m-00002.avro
-rw-r--r--   1 matt supergroup        293 2017-01-24 04:23 /user/matt/stud/studs/part-m-00003.avro

 

Re: Unable to override the existing sqoop job in CDH quickstart VM

Explorer

sqoop Version : Sqoop 1.4.5-cdh5.4.2

CDH quickstart VM: cdh5.4.2

 

Directory structure: /home/cloudera/review/sqoop

Re: Unable to override the existing sqoop job in CDH quickstart VM

Champion
Could you add --verbose to your sqoop command
paste the full log ?
Don't have an account?
Coming from Hortonworks? Activate your account here