Okay, I see that the ATS puts the logs in /ats/done directory in HDFS and this is configured by property yarn.timeline-service.entity-group-fs-store.done-dir 1. What is the difference between  org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore and  org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore ? Does the ATS generate logs in path /ats/done only if  yarn.timeline-service.entity-group-fs-store.summary-store is set to the EntityGroupFSTimelineStore ?  2. Do all three versions of the Application Timeline Server (1.0, 1.5, 2.0) generate logs in /ats/done or there is a difference in versions? 3. For some executed queries, I see that only single entity, summary logs get generated whereas for other executed queries I see multiple entity, summary logs getting generated. Is there any structure to this? Like when do we expect a single entity or summary logs or multiple? TIA ... View more

I can see the information in the Tez UI about past executed queries. Where is the Tez UI fetching this information from? HDFS? If yes? Then the question is where in hdfs? and how can I parse it programmatically say through webhdfs api's?   ... View more

My main objective is to extract query text, start_time, user,time_taken etc. of executed queries on the Tez engine.  HiveServer2 logs are in log4j format which is difficult to extract information from. Are there any logs that get generated and stored on HDFS? Is there any structure to these log files?  ... View more