Hello,
When I start an oozie workflow, then regardless of action type(sqoop, spark or ssh) it always fails with the same error from syslog:
2019-04-08 14:54:33,393 ERROR [pool-10-thread-1] org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl: Response from the timeline server is not successful, HTTP error code: 500, Server response: {"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 280 actions: IOException: 280 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"} 2019-04-08 14:54:33,394 ERROR [Job ATS Event Dispatcher] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Exception while publishing configs on JOB_SUBMITTED Event for the job : job_1554726387894_0011 org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.publishConfigsOnJobSubmittedEvent(JobHistoryEventHandler.java:1254) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1414) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1795) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1791) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Response from the timeline server is not successful, HTTP error code: 500, Server response: {"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 280 actions: IOException: 280 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"} at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putObjects(TimelineV2ClientImpl.java:322) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putObjects(TimelineV2ClientImpl.java:251) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$EntitiesHolder$1.call(TimelineV2ClientImpl.java:374) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$EntitiesHolder$1.call(TimelineV2ClientImpl.java:367) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher$1.publishWithoutBlockingOnQueue(TimelineV2ClientImpl.java:478) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher$1.run(TimelineV2ClientImpl.java:433) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)What is causing this error?
example workflow.xml
<workflow-app xmlns = "uri:oozie:workflow:0.4" name="hadoop_main_workflow">
<!-- start -->
<start to = "spark_job"/>
<action name="spark_job" retry-max="5" retry-interval="5">
<spark xmlns="uri:oozie:spark-action:0.2">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<master>yarn</master>
<mode>client</mode>
<name>spark_job</name>
<jar>spark_job.py</jar>
<spark-opts>
--master yarn
--deploy-mode client
--driver-memory 11288m
--executor-memory 24GB
--num-executors 8
--conf spark.dynamicAllocation.enabled=true
--conf spark.executor.cores=2
--conf spark.shuffle.service.enabled=true
--conf spark.yarn.driver.memoryOverhead=1024
--conf spark.yarn.executor.memoryOverhead=1024
--jars /usr/hdp/3.1.0.0-78/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar
--conf spark.security.credentials.hiveserver2.enabled=false
--py-files /usr/hdp/3.1.0.0-78/hive_warehouse_connector/pyspark_hwc-1.0.0.3.1.0.0-78.zip
</spark-opts>
<file>spark_job.py</file>
</spark>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name = "kill_job">
<message>Job failed</message>
</kill>
<end name = "end" />
</workflow-app>
job.properties:
nameNode=hdfs://namenodehost:8020
resourceManager=namenodehost:8050
queueName=${nameNode}/user/oozie/workflows/hadoop_main_workflow
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/oozie/workflows/hadoop_main_workflow
oozie.action.sharelib.for.sqoop=sqoop
oozie.action.sharelib.for.spark=spark2
Stack:
HDP 3.1.0
oozie 4.3.1.3.1.0.0-78
