question Re: yarn logs location on disk in Support Questions

yarn logs location on disk

sunilosunil — Fri, 16 Sep 2022 12:13:44 GMT

Is there a YARN API or command to know path to yarn logs location on disk for given container and application id ? Also want to add; we don't have log aggregation working and I'm perticularly looking for direct physical link to the file not the web interface.

Thanks,

Sunil

Re: yarn logs location on disk

mathieu.d — Tue, 12 Sep 2017 12:51:23 GMT

Not sure this information is available.

You could go with the "yarn logs" command

or go with the basic way using command line :

- pdsh to distribute the same command on every data-node

- launch a find on the container id

regards,

mathieu

Re: yarn logs location on disk

sunilosunil — Fri, 15 Sep 2017 16:43:58 GMT

Finding logs manually in machine sound very brute force; I was thinking more of an API or CLI option to find logs

Anyway the main issue we're trying to solve is access to logs to all developers in prod environment. Our node managers are behind the bars and not accessible ( any port or web ) to develoeprs and it's unlikely to happen. So we're trying to find a way to proxy the logs.

I discovered that there is a jobhistory proxy to look at completed jobs / yarn apps but I coudln't get it working for running app. Is there any trick / way to access running app's logs like above ?

http://resourcemanager.xyz.com:19888/jobhistory/logs//dataNode.com:8041/container_id_000001/container_id_000001/root