question where is spark application logs ? & default path of that directory ? in Support Questions

where is spark application logs ? & default path of that directory ?

ComNic — Tue, 28 Mar 2023 07:40:36 GMT

Re: where is spark application logs ? & default path of that directory ?

VidyaSargur — Tue, 28 Mar 2023 09:12:45 GMT

@ComNic, Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @RangaReddy @Babasaheb who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: where is spark application logs ? & default path of that directory ?

RangaReddy — Tue, 28 Mar 2023 09:22:21 GMT

Hi @ComNic

When you run the spark application in yarn mode, Spark application will launch executors/containers in different node. In each node, logs are stored. When log aggregation is happen all logs are merged and we can access the logs by running following command:

yarn logs -applicationId <Application_ID>

The container logs should be under yarn.nodemanager.log-dirs path.

Re: where is spark application logs ? & default path of that directory ?

RangaReddy — Thu, 30 Mar 2023 09:23:51 GMT

Hi @ComNic

Please accept above solution if you got answer for your question.

Re: where is spark application logs ? & default path of that directory ?

VidyaSargur — Mon, 03 Apr 2023 04:48:22 GMT

@ComNic, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.