Skip to main content
ComNic
Bronze ⭐️⭐️⭐️
Mar 28, 2023 7:40 AM UTC
Solved

where is spark application logs ? & default path of that directory ?

Where are the Spark application logs located, and what is the default path of that directory?

RangaReddy
Gold ⭐️
Mar 28, 2023 9:22 AM UTC

Hi @ComNic 

 

When you run the spark application in yarn mode, Spark application will launch executors/containers in different node. In each node, logs are stored. When log aggregation is happen all logs are merged and we can access the logs by running following command:

 

yarn logs -applicationId <Application_ID>

The container logs should be under yarn.nodemanager.log-dirs path. 

4 replies

VidyaSargur
Community Manager
Community Manager
Mar 28, 2023 9:12 AM UTC

@ComNic, Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @RangaReddy @Babasaheb  who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Translate
Regards, Vidya Sargur Madabhushi| Senior Community Manager | Was your question answered? Make sure to mark the answer as the Best Answer. If you find a reply useful, say thanks by clicking on the thumbs up button.
RangaReddy
Gold ⭐️
Gold ⭐️
Mar 28, 2023 9:22 AM UTC

Hi @ComNic 

 

When you run the spark application in yarn mode, Spark application will launch executors/containers in different node. In each node, logs are stored. When log aggregation is happen all logs are merged and we can access the logs by running following command:

 

yarn logs -applicationId <Application_ID>

The container logs should be under yarn.nodemanager.log-dirs path. 

Translate
RangaReddy
Gold ⭐️
Gold ⭐️
Mar 30, 2023 9:23 AM UTC

Hi @ComNic 

 

Please accept above solution if you got answer for your question.

Translate
VidyaSargur
Community Manager
Community Manager
Apr 3, 2023 4:48 AM UTC

@ComNic, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Translate
Regards, Vidya Sargur Madabhushi| Senior Community Manager | Was your question answered? Make sure to mark the answer as the Best Answer. If you find a reply useful, say thanks by clicking on the thumbs up button.