Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SQOOP Retry

Solved Go to solution

SQOOP Retry

New Contributor

Hi Team,

 

I am ingestingnthe data from Oracle 11.2.0  DB to Datalake using SQOOP. Is there any way , if database is not avialable for some time so sqoop can retry itself.

Currently, I am using below 

 

SQOOP Command

==============

sqoop import -Dmap.retry.exponentialBackOff=TRUE -Dmap.retry.numRetries=10 -D mapred.child.java.opts=-Djava.security.egd=file:/dev/../dev/urandom -m 1 --connect "jdbc:oracle:thin:@(description = (address_list = (address = (protocol = tcp)(host = *******)(Port = 1521))) (connect_data = (sid = ******)))" --username ****** --password **** --query " select count(*) from table_test where 1=1 and \$CONDITIONS " --target-dir /user/checkdb_$$

 

It is retrying for number of time , but it failed when databases come up. Is there any way , so sqoop can retry and get success ? Please help.

 

ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.net.ns.NSProtocol.connect(NSProtocol.java:395)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)
... 18 more

15/07/16 16:42:13 INFO mapreduce.Job: map 100% reduce 0%
15/07/16 16:42:13 INFO mapreduce.Job: Job job_1436788113235_2368 failed with state FAILED due to: Task failed task_1436788113235_2368_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/07/16 16:42:14 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=51367
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=51367
Total vcore-seconds taken by all map tasks=51367
Total megabyte-seconds taken by all map tasks=52599808
15/07/16 16:42:14 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
15/07/16 16:42:14 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 158.0142 seconds (0 bytes/sec)
15/07/16 16:42:14 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
15/07/16 16:42:14 INFO mapreduce.ImportJobBase: Retrieved 0 records.
15/07/16 16:42:14 ERROR tool.ImportTool: Error during import: Import job failed!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: SQOOP Retry

Master Guru
Thank you for following up!

Please consider marking the thread resolved so others with similar questions may be able to locate topics with solutions faster.
5 REPLIES 5

Re: SQOOP Retry

Expert Contributor

I'd think you could use an orchestrator service like Oozie to do this for you. 

 

http://oozie.apache.org/

 

Re: SQOOP Retry

New Contributor

Thanks,

 

But , is there any parameter within SQOOP which can retry ?

 

Regards

-Niraj

Re: SQOOP Retry

Master Guru
Sqoop CLI currently does not have auto-retry capabilities for the situation
you've mentioned.

Oozie actions, as suggested by Ben, do have a retry specifier that you can
leverage:
http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a18_User-Retry_for_Workflow...

Highlighted

Re: SQOOP Retry

New Contributor

Thanks Harsh for confirming about SQOOP CLI.

 

I am using retry mechanism with Azkaban orchestration tool and it is working fine.

 

But was just wondering if any hidden parameter for SQOOP CLI which can do retry .

 

 

Re: SQOOP Retry

Master Guru
Thank you for following up!

Please consider marking the thread resolved so others with similar questions may be able to locate topics with solutions faster.
Don't have an account?
Coming from Hortonworks? Activate your account here