question Re: Using yarn logs command in Support Questions

Using yarn logs command

Daggers — Mon, 16 Dec 2019 03:56:20 GMT

Hi,

If I try and get the logs for an application like this:

yarn logs -applicationId application_1575531060741_10424

The command fails because I am not running it as the application owner. I need to run it like this:

yarn logs -applicationId application_1575531060741_10424 -appOwner hive

The problem is I want to write out all the yarn logs to the os so I can ingest them into splunk. If I try and figure out the appOwner for each application then this is awkward and time consuming even in a script.

Is there a better way to dump all the yarn logs to the os ?

Thanks

Re: Using yarn logs command

sagarshimpi — Mon, 16 Dec 2019 03:40:46 GMT

Hi @Daggers

I think you can try this -

1. Below properties decides the path for storing yarn logs in hdfs -

Belos is sample example from my cluster - yarn.nodemanager.remote-app-log-dir = /app-logs yarn.nodemanager.remote-app-log-dir-suffix = logs-ifile

2. You can do "hadoop dfs -copyToLocal" for above path which will copy all applications to local and then you can pass to splunk ?

Do you think that can work for you?

Let me know if you have more questions on above.

Re: Using yarn logs command

sagarshimpi — Mon, 16 Dec 2019 06:55:11 GMT

@Daggers

You can also check for HDFS NFS gateway which will allow hdfs filesystem to mount on local OS exposed via NFS.

https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html

Re: Using yarn logs command

Daggers — Thu, 19 Dec 2019 02:55:09 GMT

Hey great suggestion ! I think this might work.

Re: Using yarn logs command

Daggers — Thu, 19 Dec 2019 02:59:14 GMT

I wonder if there is a way to ensure that all the files have finished being written to /tmp/log (The location at my site of yarn.nodemanager.remote-app-log-dir) before I copy them ?

Re: Using yarn logs command

sagarshimpi — Thu, 19 Dec 2019 03:44:00 GMT

Hi @Daggers

You can write simple script using yarn rest api to fetch only completed applications [month/daywise] and copy only those applications from hdfs to local. Please check below link -

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html

Re: Using yarn logs command

sagarshimpi — Thu, 19 Dec 2019 06:37:27 GMT

Hi @Daggers

Please feel free to select best answer if your questions are answered to close the thread.

Thanks