Member since
08-22-2017
3
Posts
0
Kudos Received
0
Solutions
08-22-2017
11:37 PM
We just upgraded fully working HDF + HDP clusters to HDF 3.0.1.1 and HDP 2.6.1 respectively and are now seeing a nearly identical error in our logs. I have tried to track this down in RM UI and various log files, but cannot seem to figure it out. On the HDF side, we're using HiveStreaming, which appears to be causing the errors. 2017-08-22 16:32:05,169 WARN [ATS Logger 0] org.apache.hadoop.hive.ql.hooks.ATSHook Failed to create ATS domain hive_bf609617-b443-4e32-a8af-3527b33dcb52
java.lang.RuntimeException: Failed to connect to timeline server. Connection retries limit exceeded. The posted timeline event may be missing
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:209)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineJerseyRetryFilter.handle(TimelineClientImpl.java:250)
at com.sun.jersey.api.client.Client.handle(Client.java:652)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:539)
at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPostingObject(TimelineWriter.java:161)
at org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:115)
at org.apache.hadoop.yarn.client.api.impl.TimelineWriter$1.run(TimelineWriter.java:112)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112)
at org.apache.hadoop.yarn.client.api.impl.TimelineWriter.putDomain(TimelineWriter.java:98)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putDomain(TimelineClientImpl.java:355)
at org.apache.hadoop.hive.ql.hooks.ATSHook.createTimelineDomain(ATSHook.java:122)
at org.apache.hadoop.hive.ql.hooks.ATSHook.access$200(ATSHook.java:62)
at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:179)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
... View more