https://community.cloudera.com/t5/Support-Questions/Oozie-workflow-actions-fail-with-quot-response-from-timeline/m-p/237953#M199764 <P>Hello,</P><P>When I start an oozie workflow, then regardless of action type(sqoop, spark or ssh) it always fails with the same error from syslog:</P><PRE>2019-04-08 14:54:33,393 <STRONG>ERROR [pool-10-thread-1] org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl: Response from the timeline server is not successful</STRONG>, <STRONG>HTTP error code: 500, Server response: {"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 280 actions: IOException: 280 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"} </STRONG>2019-04-08 14:54:33,394 E<STRONG>RROR [Job ATS Event Dispatcher] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:</STRONG> <STRONG>Exception while publishing configs on JOB_SUBMITTED Event &nbsp;for the job</STRONG> : job_1554726387894_0011 org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.publishConfigsOnJobSubmittedEvent(JobHistoryEventHandler.java:1254) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1414) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1795) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1791) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) &nbsp;&nbsp;&nbsp;&nbsp;at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Response from the timeline server is not successful, HTTP error code: 500, Server response: {"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 280 actions: IOException: 280 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"} &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putObjects(TimelineV2ClientImpl.java:322) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putObjects(TimelineV2ClientImpl.java:251) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$EntitiesHolder$1.call(TimelineV2ClientImpl.java:374) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$EntitiesHolder$1.call(TimelineV2ClientImpl.java:367) &nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.FutureTask.run(FutureTask.java:266) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher$1.publishWithoutBlockingOnQueue(TimelineV2ClientImpl.java:478) &nbsp;&nbsp;&nbsp;&nbsp;at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher$1.run(TimelineV2ClientImpl.java:433) &nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) &nbsp;&nbsp;&nbsp;&nbsp;at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)</PRE><P>What is causing this error?</P><P>example workflow.xml</P><PRE>&lt;workflow-app xmlns = "uri:oozie:workflow:0.4" name="hadoop_main_workflow"&gt; &nbsp; &nbsp; &lt;!-- start --&gt; &nbsp; &nbsp; &lt;start to = "spark_job"/&gt; &nbsp; &nbsp; &lt;action name="spark_job" retry-max="5" retry-interval="5"&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;spark xmlns="uri:oozie:spark-action:0.2"&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;job-tracker&gt;${resourceManager}&lt;/job-tracker&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name-node&gt;${nameNode}&lt;/name-node&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;master&gt;yarn&lt;/master&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;mode&gt;client&lt;/mode&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;spark_job&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;jar&gt;spark_job.py&lt;/jar&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;spark-opts&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --master yarn &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --deploy-mode client &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --driver-memory 11288m &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --executor-memory 24GB &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --num-executors 8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --conf spark.dynamicAllocation.enabled=true &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --conf spark.executor.cores=2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --conf spark.shuffle.service.enabled=true &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --conf spark.yarn.driver.memoryOverhead=1024 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --conf spark.yarn.executor.memoryOverhead=1024 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --jars /usr/hdp/3.1.0.0-78/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.1.0.0-78.jar &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --conf spark.security.credentials.hiveserver2.enabled=false &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; --py-files /usr/hdp/3.1.0.0-78/hive_warehouse_connector/pyspark_hwc-1.0.0.3.1.0.0-78.zip &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;/spark-opts&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;file&gt;spark_job.py&lt;/file&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;/spark&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;ok to="end"/&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;error to="kill"/&gt; &nbsp; &nbsp; &lt;/action&gt; &nbsp; &nbsp; &lt;kill name = "kill_job"&gt; &nbsp; &nbsp; &nbsp; &nbsp; &lt;message&gt;Job failed&lt;/message&gt; &nbsp; &nbsp; &lt;/kill&gt; &nbsp; &nbsp; &lt;end name = "end" /&gt; &lt;/workflow-app&gt;</PRE><P><BR /></P><P>job.properties:<BR /><BR /></P><PRE>nameNode=hdfs://namenodehost:8020 resourceManager=namenodehost:8050 queueName=${nameNode}/user/oozie/workflows/hadoop_main_workflow oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/oozie/workflows/hadoop_main_workflow oozie.action.sharelib.for.sqoop=sqoop oozie.action.sharelib.for.spark=spark2</PRE><P><BR />Stack:<BR />HDP 3.1.0</P><P>oozie 4.3.1.3.1.0.0-78</P><P><BR /></P> Tue, 09 Apr 2019 21:44:48 GMT gpjalocha 2019-04-09T21:44:48Z