Using yarn logs command

Question

Hi,

If I try and get the logs for an application like this:

yarn logs -applicationId application_1575531060741_10424

The command fails because I am not running it as the application owner. I need to run it like this:

yarn logs -applicationId application_1575531060741_10424 -appOwner hive

The problem is I want to write out all the yarn logs to the os so I can ingest them into splunk. If I try and figure out the appOwner for each application then this is awkward and time consuming even in a script.

Is there a better way to dump all the yarn logs to the os ?

Thanks

sagarshimpi · Accepted Answer

Hi @Daggers

I think you can try this -

1. Below properties decides the path for storing yarn logs in hdfs -

Belos is sample example from my cluster -

yarn.nodemanager.remote-app-log-dir = /app-logs
yarn.nodemanager.remote-app-log-dir-suffix = logs-ifile

2. You can do "hadoop dfs -copyToLocal" for above path which will copy all applications to local and then you can pass to splunk ?

Do you think that can work for you?

Let me know if you have more questions on above.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.