SQOOP Retry

[#1]

Hi Team,

I am ingestingnthe data from Oracle 11.2.0  DB to Datalake using SQOOP. Is there any way , if database is not avialable for some time so sqoop can retry itself.

Currently, I am using below 

SQOOP Command

==============

sqoop import -Dmap.retry.exponentialBackOff=TRUE -Dmap.retry.numRetries=10 -D mapred.child.java.opts=-Djava.security.egd=file:/dev/../dev/urandom -m 1 --connect "jdbc:oracle:thin:@(description = (address_list = (address = (protocol = tcp)(host = *******)(Port = 1521))) (connect_data = (sid = ******)))" --username ****** --password **** --query " select count(*) from table_test where 1=1 and \$CONDITIONS " --target-dir /user/checkdb_$$

It is retrying for number of time , but it failed when databases come up. Is there any way , so sqoop can retry and get success ? Please help.

ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.net.ns.NSProtocol.connect(NSProtocol.java:395)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)
... 18 more

15/07/16 16:42:13 INFO mapreduce.Job: map 100% reduce 0%
15/07/16 16:42:13 INFO mapreduce.Job: Job job_1436788113235_2368 failed with state FAILED due to: Task failed task_1436788113235_2368_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/07/16 16:42:14 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=51367
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=51367
Total vcore-seconds taken by all map tasks=51367
Total megabyte-seconds taken by all map tasks=52599808
15/07/16 16:42:14 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
15/07/16 16:42:14 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 158.0142 seconds (0 bytes/sec)
15/07/16 16:42:14 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
15/07/16 16:42:14 INFO mapreduce.ImportJobBase: Retrieved 0 records.
15/07/16 16:42:14 ERROR tool.ImportTool: Error during import: Import job failed!

[#2]

I'd think you could use an orchestrator service like Oozie to do this for you. 

http://oozie.apache.org/

[#3]

Thanks,

But , is there any parameter within SQOOP which can retry ?

Regards

-Niraj

[#4]

Sqoop CLI currently does not have auto-retry capabilities for the situation
you've mentioned.

Oozie actions, as suggested by Ben, do have a retry specifier that you can
leverage:
http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a18_User-Retry_for_Workflow...

[#5]

Thanks Harsh for confirming about SQOOP CLI.

I am using retry mechanism with Azkaban orchestration tool and it is working fine.

But was just wondering if any hidden parameter for SQOOP CLI which can do retry .

[#6 · Accepted solution]

Thank you for following up!

Please consider marking the thread resolved so others with similar questions may be able to locate topics with solutions faster.