Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SQOOP Retry

avatar
Explorer

Hi Team,

 

I am ingestingnthe data from Oracle 11.2.0  DB to Datalake using SQOOP. Is there any way , if database is not avialable for some time so sqoop can retry itself.

Currently, I am using below 

 

SQOOP Command

==============

sqoop import -Dmap.retry.exponentialBackOff=TRUE -Dmap.retry.numRetries=10 -D mapred.child.java.opts=-Djava.security.egd=file:/dev/../dev/urandom -m 1 --connect "jdbc:oracle:thin:@(description = (address_list = (address = (protocol = tcp)(host = *******)(Port = 1521))) (connect_data = (sid = ******)))" --username ****** --password **** --query " select count(*) from table_test where 1=1 and \$CONDITIONS " --target-dir /user/checkdb_$$

 

It is retrying for number of time , but it failed when databases come up. Is there any way , so sqoop can retry and get success ? Please help.

 

ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor

at oracle.net.ns.NSProtocol.connect(NSProtocol.java:395)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)
... 18 more

15/07/16 16:42:13 INFO mapreduce.Job: map 100% reduce 0%
15/07/16 16:42:13 INFO mapreduce.Job: Job job_1436788113235_2368 failed with state FAILED due to: Task failed task_1436788113235_2368_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/07/16 16:42:14 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=51367
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=51367
Total vcore-seconds taken by all map tasks=51367
Total megabyte-seconds taken by all map tasks=52599808
15/07/16 16:42:14 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
15/07/16 16:42:14 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 158.0142 seconds (0 bytes/sec)
15/07/16 16:42:14 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
15/07/16 16:42:14 INFO mapreduce.ImportJobBase: Retrieved 0 records.
15/07/16 16:42:14 ERROR tool.ImportTool: Error during import: Import job failed!

1 ACCEPTED SOLUTION

avatar
Mentor
Thank you for following up!

Please consider marking the thread resolved so others with similar questions may be able to locate topics with solutions faster.

View solution in original post

5 REPLIES 5

avatar
Master Collaborator

I'd think you could use an orchestrator service like Oozie to do this for you. 

 

http://oozie.apache.org/

 

avatar
Explorer

Thanks,

 

But , is there any parameter within SQOOP which can retry ?

 

Regards

-Niraj

avatar
Mentor
Sqoop CLI currently does not have auto-retry capabilities for the situation
you've mentioned.

Oozie actions, as suggested by Ben, do have a retry specifier that you can
leverage:
http://archive.cloudera.com/cdh5/cdh/5/oozie/WorkflowFunctionalSpec.html#a18_User-Retry_for_Workflow...

avatar
Explorer

Thanks Harsh for confirming about SQOOP CLI.

 

I am using retry mechanism with Azkaban orchestration tool and it is working fine.

 

But was just wondering if any hidden parameter for SQOOP CLI which can do retry .

 

 

avatar
Mentor
Thank you for following up!

Please consider marking the thread resolved so others with similar questions may be able to locate topics with solutions faster.