question Re: Workaround for log aggregation bug in Support Questions

Workaround for log aggregation bug

zack_riesland — Wed, 10 Feb 2016 23:29:31 GMT

We have log aggregation enabled in the Yarn configuration for our cluster (yarn.log-aggregation-enable).

But it doesn't seem to work.

When I try to drill into the history of a job in the resource manager GUI, the link for "logs" always takes me to a page that says: "aggregation is not enabled".

I've opened a ticket asking for help on this, and they told us we need to upgrade, so we did, but it didn't help.

I opened another ticket and am currently waiting for a response.

In the meantime, has anyone seen this?

Is there is a known hack to fix it?

Any advice about where to look for the solution?

We're currently on 2.2.8

Re: Workaround for log aggregation bug

aervits — Wed, 10 Feb 2016 23:49:55 GMT

@Zack Riesland

did you follow this doc? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/ch_log_aggregation.html

this should also give you some info http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/

Re: Workaround for log aggregation bug

zack_riesland — Wed, 10 Feb 2016 23:57:06 GMT

Yes. It basically just says to set two config properties, which I did.

Re: Workaround for log aggregation bug

nsabharwal — Thu, 11 Feb 2016 00:02:28 GMT

@Zack Riesland

Please see this thread https://community.hortonworks.com/questions/4719/yarn-logs-seems-to-be-truncated.html

I am afraid that you may not find something really useful. Please followup with the support suggestion.

Re: Workaround for log aggregation bug

TerryP — Thu, 11 Feb 2016 00:09:09 GMT

Have you tried to retrieve the logs using the YARN CLI?

yarn logs -applicationId <id of the application>

This will stream back the aggregated log to the screen ... if you have access in HDFS to see the log files. You will see a message about aggregation not being enabled if you lack permissions to see the log files. In that case, modify the command to use the application owner.

yarn logs -appOwner <user id> -applicationId <id of the application>

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:14:15 GMT

I remember having this issue until I configured TimeLine Server. @Zack Riesland

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_yarn_resource_mgt/content/ch_timeline_server.html

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:16:28 GMT

Latest doc @Zack Riesland

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_yarn_resource_mgt/content/ref-ca0dc265-903c-418a-9145-2f39fb3f6dd4.1.html

Re: Workaround for log aggregation bug

zack_riesland — Thu, 11 Feb 2016 00:18:51 GMT

Thanks Terry,

Both of these approaches work - I get back the relevant logs for a given application ID.

But I'm interested in (and tasked with) getting the UI links to work for simplicity of all the folks on our team.

Re: Workaround for log aggregation bug

zack_riesland — Thu, 11 Feb 2016 00:21:39 GMT

Thanks Artem,

Yes, this all seems to be setup correctly. I believe we had to tackle this a while back to get the Tez View working.

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:22:41 GMT

And you migrated to 1.5 of TS? @Zack Riesland

Re: Workaround for log aggregation bug

zack_riesland — Thu, 11 Feb 2016 00:26:24 GMT

I'm not explicitly specifying yarn.timeline-service.version, so I guess I'm running whatever the 2.2.8 default is.

Do you think I'll have better luck if I upgrade?

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:33:10 GMT

I would either wait for support or try in dev cluster. @Zack Riesland

Re: Workaround for log aggregation bug

zack_riesland — Thu, 11 Feb 2016 00:36:42 GMT

I just got a reply from my support ticket and it was literally a link to this thread.

So I guess we better figure it out here!

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:39:19 GMT

@Zack Riesland if you have a dev cluster, try going step by step and enabling TS. Also look for any deprecated properties.

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:43:25 GMT

@Mark Herring FYI

Re: Workaround for log aggregation bug

zack_riesland — Thu, 11 Feb 2016 00:44:34 GMT

We don't have a dev cluster. I am the only one who needs the cluster today, so I can break stuff as long as it is put back together by tonight's ingest.

It sounds like that's the only way I'm going to get this to work...

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 00:50:05 GMT

@Zack Riesland you can roll back to old configs when you are done and it doesn't work. Take some standard precautions like backup yarn-site.xml, etc. Definitely post your results here.

Re: Workaround for log aggregation bug

zack_riesland — Thu, 11 Feb 2016 00:53:59 GMT

Thanks,

Another followup:

The instructions say:

yarn.timeline-service.entity-group-fs-store.active-dir and yarn.timeline-service.entity-group-fs-store.done-dir must exist on the cluster on HDFS. Active-dir should have permission 01777, owned by YARN, group admin-group. Done-dir should have permission 0700, owned by yarn, group admin-group.

2 things:

1) 01777 isn't a valid permission set

2) when it says 'admin-group', does it literally mean that the group should be set to 'admin-group', or just a group with admin privileges? Almost everything in HDFS seems to be in either 'hadoop' or 'hdfs'.

Re: Workaround for log aggregation bug

aervits — Thu, 11 Feb 2016 01:06:03 GMT

@Zack Riesland 1. I believe they mean 1777 which is a sticky bit. https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html

2. yes the group with admin privs not literally admin-group.

Re: Workaround for log aggregation bug

iroberts — Fri, 12 Feb 2016 02:33:41 GMT

@Zack Riesland , thanks for looking at this with me over webex. It turns out it was the ownership of the mr-history directory that was causing aggregation to not work on the web side, this needed to be owned by mapred and hdfs. The initial failure to start was due to that incorrect class which we fixed.