Created 09-01-2017 04:52 PM
I've installed a fresh HDP 2.6.1.0 and lost my past configuration files.
Tried to configure the Twitter pipeline but I'm stuck at Oozie Workflow.
When I launch the coordinator job to run every 60minutes, it hangs on running state forever. It simply doesn't do anything.
This also happened in the past when I changed some memory configurations, so I suspect it must be it...
I've tunned my memory configurations with the python script so I guess everything is ok...
Can someone give me some hints to resolve this issue?
Many thanks in advance.
Best regards
Created 09-04-2017 06:00 AM
Add_partition if ACID is enabled, requires updating of Hive Metastore tables. There are known issues related to the transactions. Check hiveserver2 log to see if there are any transaction related exceptions.
Created on 09-04-2017 09:37 AM - edited 08-17-2019 05:32 PM
@sindhu many thanks for your answer... hiveserver2logs are clean...
Which logs can I check to tackle the problem?
I'll post here the yarn log which I think is clean.
Also here it goes the oozie log:
2017-09-04 09:07:01,107 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] Start action [0000065-170901161649746-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2017-09-04 09:07:01,107 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] [***0000065-170901161649746-oozie-oozi-W@:start:***]Action status=DONE 2017-09-04 09:07:01,107 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] [***0000065-170901161649746-oozie-oozi-W@:start:***]Action updated in DB! 2017-09-04 09:07:01,139 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000065-170901161649746-oozie-oozi-W 2017-09-04 09:07:01,139 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000065-170901161649746-oozie-oozi-W@:start: 2017-09-04 09:07:01,154 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Start action [0000065-170901161649746-oozie-oozi-W@hive-add-partition] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 2017-09-04 09:07:03,202 INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Trying to get job [job_1504271777639_0005], attempt [1] 2017-09-04 09:07:03,216 INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1504271777639_0005] status [RUNNING] 2017-09-04 09:07:03,217 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] [***0000065-170901161649746-oozie-oozi-W@hive-add-partition***]Action status=RUNNING 2017-09-04 09:07:03,217 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] [***0000065-170901161649746-oozie-oozi-W@hive-add-partition***]Action updated in DB! 2017-09-04 09:07:03,221 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] No Notification URL is defined. Therefore nothing to notify for job 0000065-170901161649746-oozie-oozi-W@hive-add-partition 2017-09-04 09:17:12,662 INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Trying to get job [job_1504271777639_0005], attempt [1] 2017-09-04 09:17:12,701 INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1504271777639_0005] status [RUNNING] 2017-09-04 09:27:24,642 WARN ResumeXCommand:523 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[] E1100: Command precondition does not hold before execution, [workflow's status is RUNNING is not SUSPENDED], Error Code: E1100 2017-09-04 09:28:12,685 INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] Trying to get job [job_1504271777639_0005], attempt [1] 2017-09-04 09:28:12,704 INFO HiveActionExecutor:520 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[0000065-170901161649746-oozie-oozi-W@hive-add-partition] checking action, hadoop job ID [job_1504271777639_0005] status [RUNNING] 2017-09-04 09:38:24,684 WARN ResumeXCommand:523 - SERVER[sandbox.hortonworks.com] USER[root] GROUP[-] TOKEN[] APP[hive-add-partition-wf] JOB[0000065-170901161649746-oozie-oozi-W] ACTION[] E1100: Command precondition does not hold before execution, [workflow's status is RUNNING is not SUSPENDED], Error Code: E1100 <br>
Best regards