Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

yarn logs location on disk

avatar
Contributor

Is there a YARN API or command to know path to yarn logs location on disk for given container and application id ? Also want to add; we don't have log aggregation working and I'm perticularly looking for direct physical link to the file not the web interface.

 

Thanks,

Sunil

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Not sure this information is available.

 

You could go with the "yarn logs" command

or go with the basic way using command line :

- pdsh to distribute the same command on every data-node

- launch a find on the container id

 

regards,

mathieu

 

 

 

 

View solution in original post

2 REPLIES 2

avatar
Super Collaborator

Not sure this information is available.

 

You could go with the "yarn logs" command

or go with the basic way using command line :

- pdsh to distribute the same command on every data-node

- launch a find on the container id

 

regards,

mathieu

 

 

 

 

avatar
Contributor

Finding logs manually in machine sound very brute force; I was thinking more of an API or CLI option to find logs

 

Anyway the main issue we're trying to solve is access to logs to all developers in prod environment. Our node managers are behind the bars and not accessible ( any port or web ) to develoeprs and it's unlikely to happen. So we're trying to find a way to proxy the logs.

 

I discovered that there is a jobhistory proxy to look at completed jobs / yarn apps but I coudln't get it working for running app. Is there any trick / way to access running app's logs like above ?

 

http://resourcemanager.xyz.com:19888/jobhistory/logs//dataNode.com:8041/container_id_000001/containe...