Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

oozie workflow job stucks in PREP stage

Highlighted

oozie workflow job stucks in PREP stage

New Contributor

Hi guys

I tried to run sqoop import job from oozie workflow to test if it working, but there is some problem:

job stucks in PREP stage

here is the configuration:

<sqoop xmlns="uri:oozie:sqoop-action:0.2">
  <job-tracker>${jobTracker}</job-tracker>
  <name-node>${nameNode}</name-node>
  <command>import \
--driver com.microsoft.sqlserver.jdbc.SQLServerDriver \
--connect 'jdbc:sqlserver://IP.ADDRESS.CHANGED;database=DATABASETEST123' \
--username=USERNAME \
--password=******** \
--table dbo.Testtable \
--compress \
--as-parquetfile \
--split-by id \
--hive-import \
--hive-overwrite \
--hive-table table1 \
--m 30</command>
  <configuration />
</sqoop>

here are oozie.logs generated exactly after start this job:

2018-04-04 17:15:24,061  WARN ParameterVerifier:523 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-04-04 17:15:24,464  INFO ActionStartXCommand:520 - SERVER[ooziehost] USER[username] GROUP[-] TOKEN[] APP[Batch job for sqoop test] JOB[0000000-180404170027904-oozie-oozi-W] ACTION[0000000-180404170027904-oozie-oozi-W@:start:] Start action [0000000-180404170027904-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-04-04 17:15:24,465  INFO ActionStartXCommand:520 - SERVER[ooziehost] USER[username] GROUP[-] TOKEN[] APP[Batch job for sqoop test] JOB[0000000-180404170027904-oozie-oozi-W] ACTION[0000000-180404170027904-oozie-oozi-W@:start:] [***0000000-180404170027904-oozie-oozi-W@:start:***]Action status=DONE
2018-04-04 17:15:24,465  INFO ActionStartXCommand:520 - SERVER[ooziehost] USER[username] GROUP[-] TOKEN[] APP[Batch job for sqoop test] JOB[0000000-180404170027904-oozie-oozi-W] ACTION[0000000-180404170027904-oozie-oozi-W@:start:] [***0000000-180404170027904-oozie-oozi-W@:start:***]Action updated in DB!
2018-04-04 17:15:24,722  INFO WorkflowNotificationXCommand:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180404170027904-oozie-oozi-W] ACTION[0000000-180404170027904-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000000-180404170027904-oozie-oozi-W@:start:
2018-04-04 17:15:24,723  INFO WorkflowNotificationXCommand:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-180404170027904-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000000-180404170027904-oozie-oozi-W
2018-04-04 17:15:24,770  INFO ActionStartXCommand:520 - SERVER[ooziehost] USER[username] GROUP[-] TOKEN[] APP[Batch job for sqoop test] JOB[0000000-180404170027904-oozie-oozi-W] ACTION[0000000-180404170027904-oozie-oozi-W@sqoop-6bbd] Start action [0000000-180404170027904-oozie-oozi-W@sqoop-6bbd] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-04-04 17:15:43,705  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] CoordMaterializeTriggerService - Curr Date= 2018-04-04T17:20+0400, Num jobs to materialize = 0
2018-04-04 17:15:43,706  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.CoordMaterializeTriggerService]
2018-04-04 17:15:43,942  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2018-04-04 17:15:43,943  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running coordinator status service from last instance time =  2018-04-04T17:14+0400
2018-04-04 17:15:43,949  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running bundle status service from last instance time =  2018-04-04T17:14+0400
2018-04-04 17:15:43,952  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2018-04-04 17:15:44,047  INFO PauseTransitService:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2018-04-04 17:15:44,061  INFO PauseTransitService:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2018-04-04 17:16:43,953  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2018-04-04 17:16:43,954  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running coordinator status service from last instance time =  2018-04-04T17:15+0400
2018-04-04 17:16:43,959  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running bundle status service from last instance time =  2018-04-04T17:15+0400
2018-04-04 17:16:43,962  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2018-04-04 17:16:44,062  INFO PauseTransitService:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]
2018-04-04 17:16:44,075  INFO PauseTransitService:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.PauseTransitService]
2018-04-04 17:17:43,963  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.StatusTransitService]
2018-04-04 17:17:43,963  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running coordinator status service from last instance time =  2018-04-04T17:16+0400
2018-04-04 17:17:43,969  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Running bundle status service from last instance time =  2018-04-04T17:16+0400
2018-04-04 17:17:43,972  INFO StatusTransitService$StatusTransitRunnable:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Released lock for [org.apache.oozie.service.StatusTransitService]
2018-04-04 17:17:44,075  INFO PauseTransitService:520 - SERVER[ooziehost] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Acquired lock for [org.apache.oozie.service.PauseTransitService]

I cant even kill the job it stays in running mode in the workflow

Any idea?

Thank you

2 REPLIES 2

Re: oozie workflow job stucks in PREP stage

Expert Contributor

Hi @Shota Akhalaia,

Can you try restarting oozie service once? and try?

Also collect queue dump before restarting service

oozie admin -oozie http://localhost:11000/oozie -queuedump

-Shubham

Re: oozie workflow job stucks in PREP stage

New Contributor

Hi

I tried queue dump:

oozie admin -oozie http://localhost:11000/oozie -queuedump
[Server Queue Dump]:
[action.start_0000000-180404170027904-oozie-oozi-W@sqoop-6bbd] priority=0 delay=419
******************************************
[Server Uniqueness Map Dump]:
action.start_0000000-180404170027904-oozie-oozi-W@sqoop-6bbd=Thu Apr 05 16:59:23 GET 2018

then restarted oozie and tried to kill the job again but it is still remaining in running state in PREP stage :/

do you have any idea what am I doing wrong?

Thank you

Don't have an account?
Coming from Hortonworks? Activate your account here