Guys,
I am trying to run the Spark PI application in YARN cluster mode, here are the steps briefly.
(This works for service users like spark, hdfs, hive,etc)
- try doing kinit using spark headless keytab
- run sparkPI example, check application ID
- check yarn logs using yarn logs -applicationId <ID>
I see the logs / PI value,etc.
Now follow these steps:
- kdestory
- kinit adm_user (user in active directory)
- check klist and user has valid TGT.
- run same spark PI application in YARN Cluster mode
- get the application id from history server or from console.
- Execute : yarn logs -applicationId <appID> from above step.
I see following :
16/06/09 12:11:53 INFO impl.TimelineClientImpl: Timeline service address: http://host:8188/ws/v1/timeline/
16/06/09 12:11:54 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
/app-logs/<USER_WHO_WAS_USED_TO_KINIT>/logs/application_1465312809105_0024 does not have any log files.
I checked /logs/<KINIT_USER>/logs/applicationID, indeed it does not have any logs.
Could you please suggest what is missing?
Thanks in advance,
Arpan
