Support Questions
Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Oozie hive-add-partition keeps on running state forever

Labels:
hsfelix
Contributor

Created ‎09-01-2017 04:52 PM

I've installed a fresh HDP 2.6.1.0 and lost my past configuration files.

Tried to configure the Twitter pipeline but I'm stuck at Oozie Workflow.

When I launch the coordinator job to run every 60minutes, it hangs on running state forever. It simply doesn't do anything.

This also happened in the past when I changed some memory configurations, so I suspect it must be it...

I've tunned my memory configurations with the python script so I guess everything is ok...

Can someone give me some hints to resolve this issue?

Many thanks in advance.

Best regards

804 Views
0 Kudos
2 REPLIES 2

ssubhas
Guru

Created ‎09-04-2017 06:00 AM

@Hugo Felix

Add_partition if ACID is enabled, requires updating of Hive Metastore tables. There are known issues related to the transactions. Check hiveserver2 log to see if there are any transaction related exceptions.

583 Views
0 Kudos

hsfelix
Contributor

Created on ‎09-04-2017 09:37 AM - edited ‎08-17-2019 05:32 PM

@sindhu many thanks for your answer... hiveserver2logs are clean...
Which logs can I check to tackle the problem?
I'll post here the yarn log which I think is clean.

yarn.txt

Also here it goes the oozie log:

2017-09-04 09:07:01,107  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] Start action [0000065-170901161649746-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-09-04 09:07:01,107  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] [***0000065-170901161649746-oozie-oozi-W@:start:***]Action status=DONE
2017-09-04 09:07:01,107  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] [***0000065-170901161649746-oozie-oozi-W@:start:***]Action updated in DB!
2017-09-04 09:07:01,139  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000065-170901161649746-oozie-oozi-W
2017-09-04 09:07:01,139  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000065-170901161649746-oozie-oozi-W@:start:
2017-09-04 09:07:01,154  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Start action [0000065-170901161649746-oozie-oozi-W@hive-add-partition] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-09-04 09:07:03,202  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Trying to get job [job_1504271777639_0005], attempt [1]
2017-09-04 09:07:03,216  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1504271777639_0005] status [RUNNING]
2017-09-04 09:07:03,217  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] [***0000065-170901161649746-oozie-oozi-W@hive-add-partition***]Action status=RUNNING
2017-09-04 09:07:03,217  INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] [***0000065-170901161649746-oozie-oozi-W@hive-add-partition***]Action updated in DB!
2017-09-04 09:07:03,221  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] No Notification URL is defined. Therefore nothing to notify for job 0000065-170901161649746-oozie-oozi-W@hive-add-partition
2017-09-04 09:17:12,662  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Trying to get job [job_1504271777639_0005], attempt [1]
2017-09-04 09:17:12,701  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1504271777639_0005] status [RUNNING]
2017-09-04 09:27:24,642  WARN ResumeXCommand:523 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[] E1100: Command precondition does not hold before execution, [workflow's status is RUNNING is not SUSPENDED], Error Code: E1100
2017-09-04 09:28:12,685  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Trying to get job [job_1504271777639_0005], attempt [1]
2017-09-04 09:28:12,704  INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1504271777639_0005] status [RUNNING]
2017-09-04 09:38:24,684  WARN ResumeXCommand:523 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[] E1100: Command precondition does not hold before execution, [workflow's status is RUNNING is not SUSPENDED], Error Code: E1100
<br>

34644-1o.jpg

34646-3a.jpg

34645-2a.jpg

Best regards

Preview file
122 KB
583 Views
0 Kudos
Take a Tour of the Community
Don't have an account?
Register
Announcements
What's New @ Cloudera
CDP Operational Database (COD) supports CDP Control Planes for multiple regions.
What's New @ Cloudera
CDP Operational Database (COD) supports Multiple Availability Zones (Multi-AZ) on AWS
What's New @ Cloudera
COD now supports the “Store File Tracking” (SFT) as a general availability feature.
Community Announcements
Introducing Community Tours
What's New @ Cloudera
Cloudera Machine Learning now supports adding or replacing GPUs in CML Workspaces
View More Announcements
; ;