Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Not applicable

Default location for spark event log history is hdfs:///spark-history (Spark) and hdfs:///spark2-history/ (spark2).

This will help to debug spark-history page load issue or if you have huge count of event log files you can archive it by creating the new active location.

Following are the steps to change this default location.

1. Create new directory on hdfs for e.g

$hdfs dfs -mkdir /spark2-history_new 
$hdfs dfs -chown spark:hadoop /spark2-history_new

2. Login to Amabri==>Spark==>config.

3. Update following parameters with new path "hdfs:///spark2-history_new/ " as follow.

83457-screen-shot-2018-07-26-at-12613-pm.png

4. Save the configuration.

5. Restart Spark service to enable new changes.

6. Run spark job, your new event log file will get save in new location. Same you can view using the spark history UI.

864 Views
0 Kudos
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 06:52 AM
Updated by:
 
Contributors
Top Kudoed Authors