Do not see the logs while submitting Spark Application as a usr in YARN mode

arpan57 — Fri, 16 Sep 2022 10:24:42 GMT

Guys,

I am trying to run the Spark PI application in YARN cluster mode, here are the steps briefly.

(This works for service users like spark, hdfs, hive,etc)
- try doing kinit using spark headless keytab
- run sparkPI example, check application ID
- check yarn logs using yarn logs -applicationId <ID>
I see the logs / PI value,etc.

Now follow these steps:
- kdestory
- kinit adm_user (user in active directory)

- check klist and user has valid TGT.
- run same spark PI application in YARN Cluster mode
- get the application id from history server or from console.
- Execute : yarn logs -applicationId <appID> from above step.

I see following :
16/06/09 12:11:53 INFO impl.TimelineClientImpl: Timeline service address: http://host:8188/ws/v1/timeline/
16/06/09 12:11:54 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
/app-logs/<USER_WHO_WAS_USED_TO_KINIT>/logs/application_1465312809105_0024 does not have any log files.

I checked /logs/<KINIT_USER>/logs/applicationID, indeed it does not have any logs.

Could you please suggest what is missing?

Thanks in advance,

Arpan

Re: Do not see the logs while submitting Spark Application as a usr in YARN mode

arpan57 — Tue, 30 Aug 2016 14:09:53 GMT

This got fixed, it was strange issue with casing.

question Do not see the logs while submitting Spark Application as a usr in YARN mode in Archives of Support Questions (Read Only)

Do not see the logs while submitting Spark Application as a usr in YARN mode

Re: Do not see the logs while submitting Spark Application as a usr in YARN mode