Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop ingesting many txt file in one shoot to mysql

Sqoop ingesting many txt file in one shoot to mysql

Rising Star

Hi guys, I would like to know if in Sqoop is possible load too many files at the same time as we load whole included files in a folder into a pig relation just indicating in the load something like:

test = LOAD '/user/me/datasets/*' USING PigStorage(',');

I have tried to apply the same logic in a SQOOP stement to load or update in one shoot many files with the same structure into a mysql table with the code:

sqoop export --connect jdbc:mysql://nn01.itversity.com/retail_export --username retail_dba --password itversity \ --table roles --update-key id_emp --update-mode allowinsert --export-dir /user/ingenieroandresangel/datasets/export/* \ -m 1

The weird thing in my scenario is that even if at the end of my statement the systems shows up a message:

17/08/14 15:05:02 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=1 Launched map tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5548 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=2774 Total vcore-milliseconds taken by all map tasks=2774 Total megabyte-milliseconds taken by all map tasks=4260864 17/08/14 15:05:02 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 17/08/14 15:05:02 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 19.3098 seconds (0 bytes/sec) 17/08/14 15:05:02 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 17/08/14 15:05:02 INFO mapreduce.ExportJobBase: Exported 0 records. 17/08/14 15:05:02 ERROR mapreduce.ExportJobBase: Export job failed! 17/08/14 15:05:02 ERROR tool.ExportTool: Error during export: Export job failed!

the table in my sql has load all records in my two proposed txt files so, in theory that worked even with this error but i'm not sure so please if someone can explain that behavior in technical sens i appreciate.

thanks.

1 REPLY 1
Highlighted

Re: Sqoop ingesting many txt file in one shoot to mysql

Contributor

@Andres Urrego Can you please use --verbose option and share the full logs of the failed job?

Don't have an account?
Coming from Hortonworks? Activate your account here