Support Questions
Find answers, ask questions, and share your expertise

Do not see the logs while submitting Spark Application as a usr in YARN mode

Explorer

Guys,

 

I am trying to run the Spark PI application in YARN cluster mode, here are the steps briefly.

(This works for service users like spark, hdfs, hive,etc)
- try doing kinit using spark headless keytab
- run sparkPI example, check application ID
- check yarn logs using yarn logs -applicationId <ID>
I see the logs / PI value,etc. 

Now follow these steps:
- kdestory
- kinit adm_user (user in active directory)

- check klist and user has valid TGT.
- run same spark PI application in YARN Cluster mode
- get the application id from history server or from console.
- Execute : yarn logs -applicationId <appID> from above step.

I see following :
16/06/09 12:11:53 INFO impl.TimelineClientImpl: Timeline service address: http://host:8188/ws/v1/timeline/
16/06/09 12:11:54 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
/app-logs/<USER_WHO_WAS_USED_TO_KINIT>/logs/application_1465312809105_0024 does not have any log files.

I checked /logs/<KINIT_USER>/logs/applicationID, indeed it does not have any logs.

Could you please suggest what is missing?

 

Thanks in advance,

Arpan

 

1 ACCEPTED SOLUTION

Accepted Solutions

Explorer
This got fixed, it was strange issue with casing.

View solution in original post

1 REPLY 1

Explorer
This got fixed, it was strange issue with casing.

View solution in original post