Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Spark job always stuck at state : ACCEPTED

Labels:
avatar
author-rank cmy
New Contributor

Created on ‎09-19-2020 09:31 PM - last edited on ‎09-24-2020 12:32 PM by Community Manager Community Manager

Hi all, 

I am a hadoop admin for my cluster and there is an application that will stuck at state : ACCEPTED intermittently and turn to state : FAILED . The issue is intermittent, it happens once/twice for every 3-400 attempts. It means out of 300 attempts, 297 will run normally but 3 will fail intermittently. 

This is how the error happen normally : 

20/09/19 21:13:45 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:46 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:47 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:48 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:49 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:50 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:51 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:52 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:53 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:54 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:55 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:56 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:57 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:58 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:13:59 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:00 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:01 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:02 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:03 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:04 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:05 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:06 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:07 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:08 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:09 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:10 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:11 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:12 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:13 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:14 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:15 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:16 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:17 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:18 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:19 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:20 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:21 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:22 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:23 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:24 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:25 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:26 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:27 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:28 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:29 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:30 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:31 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:32 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:33 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:34 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:35 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:36 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:37 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:38 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:39 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:40 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:41 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:42 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:43 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:44 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:45 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:46 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:47 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:48 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:49 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:50 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:51 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:52 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:53 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:54 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:55 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:56 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:57 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:58 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:14:59 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:00 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:01 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:02 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:03 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:04 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:05 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:06 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:07 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:08 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:09 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:10 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:11 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:12 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:13 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:14 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:15 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:16 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:17 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:18 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:19 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:20 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:21 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:22 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:23 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:24 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:25 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:26 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:27 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:28 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:29 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:30 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:31 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:32 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:33 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:34 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:35 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:36 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:37 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:38 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:39 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:40 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:41 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:42 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:43 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:44 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:45 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:46 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:47 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:48 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:49 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:50 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:51 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:52 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:53 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:54 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:55 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:56 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:57 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:58 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:15:59 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:00 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:01 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:02 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:03 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:04 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:05 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:06 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:07 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:08 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:09 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:10 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:11 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:12 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:13 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:14 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:15 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:16 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:17 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:18 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:19 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:20 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:21 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:22 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:23 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:24 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:25 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:26 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:27 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:28 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:29 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:30 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:31 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:32 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:33 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:34 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:35 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:36 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:37 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:38 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:39 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:40 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:41 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:42 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:43 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:44 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:45 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:46 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:47 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:48 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:49 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:50 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:51 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:52 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:53 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:54 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:55 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:56 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:57 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:58 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:16:59 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:00 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:01 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:02 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:03 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:04 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:05 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:06 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:07 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:08 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:09 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:10 INFO Client: Application report for application_1599812013064_109877 (state: ACCEPTED)
20/09/19 21:17:11 INFO Client: Application report for application_1599812013064_109877 (state: FAILED)


Whereas normal behavior will be like this 

20/09/20 03:47:14 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:15 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:16 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:17 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:18 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:19 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:20 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:21 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:22 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:23 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:24 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:25 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:26 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:27 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:28 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:29 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:30 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:31 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:32 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:33 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:34 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:35 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:36 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:37 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:38 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:39 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:40 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:41 INFO Client: Application report for application_1599812013064_113242 (state: ACCEPTED)
20/09/20 03:47:42 INFO Client: Application report for application_1599812013064_113242 (state: RUNNING)

 

This is the error I will get after it's stuck in ACCEPTED state for very long

20/09/19 21:17:10 ERROR ApplicationMaster: Uncaught exception: 
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
20/09/19 21:17:10 INFO ApplicationMaster: Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds])
20/09/19 21:17:10 INFO ApplicationMaster: Deleting staging directory hdfs://TSPSTREAM01/user/streamf10w/.sparkStaging/application_1599812013064_109876
20/09/19 21:17:10 INFO ShutdownHookManager: Shutdown hook called

 

NodeManger log at the data node side 

 

2020-09-19 21:15:27,144 WARN  nodemanager.LinuxContainerExecutor (LinuxContainerExecutor.java:handleExitCode(591)) - Exception from container-launch with container ID: container_e62_1599812013064_109877_01_000001 and exit code: 13
org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:125)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:141)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.handleLaunchForLaunchType(LinuxContainerExecutor.java:564)
        at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:479)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:500)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:310)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:105)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

 




This is how the pyspark job is submitted

 

spark-submit \ 
--master yarn \ 
--deploy-mode cluster \ 
--queue xxx \ 
--driver-cores 1 \ 
--driver-memory 1g \ 
--num-executors 1 \ 
--executor-memory 1g \ 
--executor-cores 1 \ 
--conf spark.pyspark.python=/anaconda_env/projects/xxx/bin/python \ 
--conf spark.pyspark.driver.python=/anaconda_env/projects/xxx/bin/python \ 
--conf spark.executor.memoryOverhead=5120 \ 
--py-files xxx.py

 


Does anyone what would be the possible root cause? From what I see here is that intermittently, when the jobs are submitted to the cluster, it will stuck in ACCEPTED state for very long and turn it into FAILED. My cluster has plenty of resources 4800GB with 864 cores. This job will take normally average 9GB and 2 cores for a duration of 30s. 

Do I need to tune the parameter of spark-submit job to prevent this from happening?

Thank you in advance!

3,232 Views
0 Kudos
1 REPLY 1

avatar
author-rank EvgenyWeb
New Contributor

Created ‎09-09-2021 06:24 AM

Hi, did you solve the problem? I have exactly the same issue and the same exception, python Spark doesn't start, it is stuck in ACCEPTED state, and then fails with

TimeoutException: Futures timed out after [100000 milliseconds]

and the stack trace is the same. I would really appreciate any advice.

2,704 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements