Support Questions
Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Sqoop Performance issue

Labels:
Shivhar
New Contributor

Created on ‎03-28-2019 01:44 PM - last edited on ‎03-29-2019 05:08 AM by Community Manager Community Manager

I am trying to sqoop the data from teradata to hive database. I am giving number of mapper as 10. Upon running the script, 9 mappers are getting completed within a min and last mapper is taking 2 hours of time. i have provided the log where my script stalls.

 

My source table has 1.3 billion of records out of which i am running sqoop in loop (~120 chucks and each chuck has 30 million records).  For one chuck, its taking 2-3 hours of time. Could you pleas help out how to improve the performance for the sqoop.

 

2019-03-28 13:20:46,538 INFO [IPC Server handler 3 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:20:52,571 INFO [IPC Server handler 15 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:20:58,601 INFO [IPC Server handler 7 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:04,630 INFO [IPC Server handler 26 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:10,661 INFO [IPC Server handler 29 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:16,691 INFO [IPC Server handler 12 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:22,722 INFO [IPC Server handler 25 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:28,752 INFO [IPC Server handler 9 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:34,782 INFO [IPC Server handler 24 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:40,813 INFO [IPC Server handler 27 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:46,845 INFO [IPC Server handler 13 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:52,875 INFO [IPC Server handler 17 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:21:58,907 INFO [IPC Server handler 10 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:04,939 INFO [IPC Server handler 16 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:10,972 INFO [IPC Server handler 23 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:17,003 INFO [IPC Server handler 28 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:23,035 INFO [IPC Server handler 2 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:29,066 INFO [IPC Server handler 22 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:35,095 INFO [IPC Server handler 6 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:41,126 INFO [IPC Server handler 1 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:47,155 INFO [IPC Server handler 18 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:53,188 INFO [IPC Server handler 0 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:22:59,222 INFO [IPC Server handler 21 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:05,252 INFO [IPC Server handler 11 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:11,282 INFO [IPC Server handler 20 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:17,316 INFO [IPC Server handler 8 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:23,347 INFO [IPC Server handler 4 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:29,377 INFO [IPC Server handler 5 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:35,410 INFO [IPC Server handler 19 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:41,444 INFO [IPC Server handler 14 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:47,478 INFO [IPC Server handler 3 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:53,510 INFO [IPC Server handler 15 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:23:59,543 INFO [IPC Server handler 7 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:05,574 INFO [IPC Server handler 26 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:11,604 INFO [IPC Server handler 29 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:17,634 INFO [IPC Server handler 12 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:23,668 INFO [IPC Server handler 25 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:29,700 INFO [IPC Server handler 9 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:35,731 INFO [IPC Server handler 24 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:41,762 INFO [IPC Server handler 27 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:47,792 INFO [IPC Server handler 13 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:53,822 INFO [IPC Server handler 17 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:24:59,853 INFO [IPC Server handler 10 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:05,884 INFO [IPC Server handler 16 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:11,917 INFO [IPC Server handler 23 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:17,946 INFO [IPC Server handler 28 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:23,978 INFO [IPC Server handler 2 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:30,011 INFO [IPC Server handler 22 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:36,041 INFO [IPC Server handler 6 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:42,071 INFO [IPC Server handler 1 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:48,101 INFO [IPC Server handler 18 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:25:54,131 INFO [IPC Server handler 0 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:00,161 INFO [IPC Server handler 21 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:06,194 INFO [IPC Server handler 11 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:12,223 INFO [IPC Server handler 20 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:18,256 INFO [IPC Server handler 8 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:24,288 INFO [IPC Server handler 4 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:30,318 INFO [IPC Server handler 5 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:36,355 INFO [IPC Server handler 19 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:42,385 INFO [IPC Server handler 14 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:48,415 INFO [IPC Server handler 3 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:26:54,445 INFO [IPC Server handler 15 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:00,475 INFO [IPC Server handler 7 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:06,503 INFO [IPC Server handler 26 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:12,536 INFO [IPC Server handler 29 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:18,566 INFO [IPC Server handler 12 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:24,597 INFO [IPC Server handler 25 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:30,628 INFO [IPC Server handler 9 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:36,658 INFO [IPC Server handler 24 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:42,689 INFO [IPC Server handler 27 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:48,722 INFO [IPC Server handler 13 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:27:54,751 INFO [IPC Server handler 17 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:00,781 INFO [IPC Server handler 10 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:06,814 INFO [IPC Server handler 16 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:12,846 INFO [IPC Server handler 23 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:18,876 INFO [IPC Server handler 28 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:24,906 INFO [IPC Server handler 2 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:30,937 INFO [IPC Server handler 22 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:36,969 INFO [IPC Server handler 6 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:42,998 INFO [IPC Server handler 1 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:49,027 INFO [IPC Server handler 18 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:28:55,057 INFO [IPC Server handler 0 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:01,088 INFO [IPC Server handler 21 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:07,118 INFO [IPC Server handler 11 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:13,147 INFO [IPC Server handler 20 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:19,180 INFO [IPC Server handler 8 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:25,212 INFO [IPC Server handler 4 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:31,242 INFO [IPC Server handler 5 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:37,273 INFO [IPC Server handler 19 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:43,303 INFO [IPC Server handler 14 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:49,337 INFO [IPC Server handler 3 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:29:55,365 INFO [IPC Server handler 15 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:01,396 INFO [IPC Server handler 7 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:07,427 INFO [IPC Server handler 26 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:13,457 INFO [IPC Server handler 29 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:19,488 INFO [IPC Server handler 12 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:25,517 INFO [IPC Server handler 25 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:31,547 INFO [IPC Server handler 9 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:37,577 INFO [IPC Server handler 24 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:43,607 INFO [IPC Server handler 27 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:49,639 INFO [IPC Server handler 13 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:30:55,668 INFO [IPC Server handler 17 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:01,701 INFO [IPC Server handler 10 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:07,731 INFO [IPC Server handler 16 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:13,761 INFO [IPC Server handler 23 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:19,793 INFO [IPC Server handler 28 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:25,824 INFO [IPC Server handler 2 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:31,855 INFO [IPC Server handler 22 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:37,884 INFO [IPC Server handler 1 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:43,916 INFO [IPC Server handler 6 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:49,947 INFO [IPC Server handler 18 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:31:55,977 INFO [IPC Server handler 0 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:02,007 INFO [IPC Server handler 21 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:08,037 INFO [IPC Server handler 11 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:14,067 INFO [IPC Server handler 20 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:20,100 INFO [IPC Server handler 8 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:26,130 INFO [IPC Server handler 4 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:32,163 INFO [IPC Server handler 5 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:38,195 INFO [IPC Server handler 19 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:44,226 INFO [IPC Server handler 14 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:50,255 INFO [IPC Server handler 3 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:32:56,285 INFO [IPC Server handler 15 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:02,315 INFO [IPC Server handler 7 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:08,345 INFO [IPC Server handler 26 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:14,378 INFO [IPC Server handler 29 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:20,407 INFO [IPC Server handler 12 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:26,435 INFO [IPC Server handler 25 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:32,466 INFO [IPC Server handler 9 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:38,496 INFO [IPC Server handler 24 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:44,526 INFO [IPC Server handler 27 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:50,558 INFO [IPC Server handler 13 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:33:56,588 INFO [IPC Server handler 17 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:02,621 INFO [IPC Server handler 10 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:08,651 INFO [IPC Server handler 16 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:14,681 INFO [IPC Server handler 23 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:20,713 INFO [IPC Server handler 28 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:26,742 INFO [IPC Server handler 2 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:32,772 INFO [IPC Server handler 22 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:38,802 INFO [IPC Server handler 1 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:44,832 INFO [IPC Server handler 6 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:50,865 INFO [IPC Server handler 18 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:34:56,895 INFO [IPC Server handler 0 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:02,925 INFO [IPC Server handler 21 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:08,958 INFO [IPC Server handler 11 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:14,987 INFO [IPC Server handler 20 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:21,017 INFO [IPC Server handler 8 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:27,048 INFO [IPC Server handler 4 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:33,081 INFO [IPC Server handler 5 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:39,115 INFO [IPC Server handler 19 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:45,149 INFO [IPC Server handler 14 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:51,180 INFO [IPC Server handler 3 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:35:57,214 INFO [IPC Server handler 15 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:36:03,245 INFO [IPC Server handler 7 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:36:09,277 INFO [IPC Server handler 26 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:36:15,309 INFO [IPC Server handler 29 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:36:21,342 INFO [IPC Server handler 12 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:36:27,374 INFO [IPC Server handler 25 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0
2019-03-28 13:36:32,069 INFO [IPC Server handler 5 on 33298] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1553288215659_3531_m_000019_0 is : 0.0

882 Views
0 Kudos
2 REPLIES 2

Tomas79
Master Collaborator

Created ‎03-29-2019 12:39 AM

I dont think that anybody can help you based on this. It is not the performance of the sqoop - sqoop is just an application which creates and runs on your behalf a mapreduce job, where the MR job is using Teradata connector to fetch the splits. To investigate the issue try to monitor the Teradata what is happening there (scans, connections). And go to the YARN logs of the MR job and try to identify what is taking so long.
866 Views
0 Kudos

Harsh J
Master Guru

Created ‎04-03-2019 06:35 PM

This may not help you directly as there isn't enough evidence in the post
to investigate, but when you have a straggler task that takes several times
longer than others in its phase, it is more likely to be due to a key based
skew.

If your source table's primary key set, or the selected split-by key is not
high in cardinality, the division of tasks will be skewed. It is worth
inspecting this (a COUNT over GROUP BY of any column will help tell) and
adjusting the import query parameters accordingly to gain maximum
parallelism.

One way to confirm if this is the case is to check actual task record
counters when the tasks are running, between those that complete in short
durations and the straggler.
836 Views
0 Kudos
Take a Tour of the Community
Don't have an account?
Register
Your experience may be limited. Sign in to explore more.
Announcements
Product Announcements
CDP Public Cloud Release Summary: June 2022
What's New @ Cloudera
Cloudera DataFlow for the Public Cloud 2.1 introduces in-place upgrades as a Technical Preview feature
What's New @ Cloudera
CDP Operational Database (COD) supports CDP Control Planes for multiple regions.
What's New @ Cloudera
CDP Operational Database (COD) supports Multiple Availability Zones (Multi-AZ) on AWS
What's New @ Cloudera
COD now supports the “Store File Tracking” (SFT) as a general availability feature.
View More Announcements
; ;