Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Timeline Service 2.0 reader and HBase modes

Yarn Timeline Service 2.0 reader and HBase modes

Explorer

Hi, 

 

I've been running HDP 3.1 for months in several clusters but I never quite understood how this Yarn Timeline Service (and it's reader) works and it has been always a cause for frustration. I don't think there is enough documentation.

 

What is the relation between Timeline Service V1.5 and Timeline Service V2.0 reader? how does it work exactly?

 

Where can I read more about this, specifically what are the differences in the behavior of the files, services, config files, etc. at the OS / filesystem level, etc. between checking "is_hbase_system_service_launch" , unchecking it, checking "use_external_hbase", and having (or not) a dedicated hbase standalone service installed?!

 

Most of the clusters run without a seperate Hbase service in which case, from my experience, it's better to just leave "is_hbase_system_service_launch" unchecked which from what I can understand runs it in some kind of "embedded mode" (in what host does it run? what process starts it? where are the logs? how can I check if it's running? what user launches it? where does it write the db?")

 

For some reason some clusters came with this option checked and almost always is a source of issues. The same questions arise. It's not quite clear where things go in either case and searching through the file system can be quite confusing because of all the yarn/hbase related files required for all the combinations / modes.

 

Right now I'm trying to troubleshoot:

 

WARN  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(183)) - Got failure attempting to read from timeline storage, assuming HBase down
java.io.UncheckedIOException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions:
Sat Jul 20 05:10:15 UTC 2019, RpcRetryingCaller{globalStartTime=1563599415627, pause=1000, maxAttempts=4}, org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server host-192-168-0-72.dhub-fp.com,17020,1563552530920 is not running yet
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1487)
        at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3269)
        at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42002)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

 

 

Both Timeline Service V1.5, RM's, NM's, Yarn clients and Timeline Service v2.0 Reader are running (as far as ambari is concerned). There is only 1 alert in Ambari "ATSv2 HBase Application" The HBase application reported a 'STOPPED' state. Check took 2.226s".

 

Could you please help me understand this questions (eg: point me to a book or document) so I can troubleshoot this and avoid/solve future issues!?

 

Thank you in advance,

David

Don't have an account?
Coming from Hortonworks? Activate your account here