Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Spark Driver and Executor logs in yarn cluster mode for CDP clusters
Labels:
- Labels:
-
Apache Spark
Expert Contributor
Created 01-15-2024 04:43 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Where is the location for the Spark driver and executor logs when the Spark job is executed in YARN cluster mode?
Does yarn logs -applicationId CMD capture the logs? I can't find log4j-active.log for the driver.
1 REPLY 1
Master Collaborator
Created 02-04-2024 08:11 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
To locate Spark logs, follow these steps:
- Access the Spark UI: Open the Spark UI in your web browser.
- Identify Nodes: Navigate to the Executors tab to view information about the driver and executor nodes involved in the Spark application.
- Determine Log Directory: Within the Spark UI, find the Hadoop settings section and locate the value of the yarn.nodemanager.log-dirs property. This specifies the base directory for Spark logs on the cluster.
- Access Log Location: Using a terminal or SSH, log in to the relevant node (driver or executor) where the logs you need are located.
- Navigate to Application Log Directory: Within the yarn.nodemanager.log-dirs directory, access the subdirectory for the specific application using the pattern application_${appid}, where ${appid} is the unique application ID of the Spark job.
- Find Container Logs: Within the application directory, locate the individual container log directories named container_{$contid}, where ${contid} is the container ID.
- Review Log Files: Each container directory contains the following log files generated by that container:
- stderr: Standard error output
- stdin: Standard input (if applicable)
- syslog: System-level logs
