Support Questions
Find answers, ask questions, and share your expertise

the spark configure about changing log_dir of spark_worker on CDH 5.0.0

the spark configure about changing log_dir of spark_worker on CDH 5.0.0

New Contributor

 

I'm have installed CDH 5.0.0 and add spark service.

I found the spark worker can't config log_dir

so I modify spark_worker below configure (spark_worker->advance)

 

"SPARK_WORKER_role_env_safety_valve" ;log_dir=/dfs/var/log/sparkwork

"log4j_safety_valve";"log.dir=/dfs/var/log/spark"

 

after restart service, the log4j.properties of process directory have changed ,but the cloudera-monitor.properties not changed

 

forexample  

/var/run/cloudera-scm-agent/process/1193-spark-SPARK_WORKER

 

log4j.properties :

log.threshold=INFO
main.logger=RFA
root.logger=${log.threshold},${main.logger}
log4j.rootLogger=${root.logger}
log.dir=/dfs/var/log/spark
log.file=spark-worker-pd1test6c8g.log
max.log.file.size=200MB
max.log.file.backup.index=10
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
log4j.appender.RFA.File=${log.dir}/${log.file}
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
log4j.appender.RFA.MaxFileSize=${max.log.file.size}
log4j.appender.RFA.MaxBackupIndex=${max.log.file.backup.index}
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
~                                                                                   

cloudera-monitor.properties:

host = pd1test6c8g
service_name = spark
service_version = 5
role_name = spark-SPARK_WORKER-3b58c709a04e72b5176c0806f0ab1543
collect_interval = 60
log_dir = /var/log/spark

 

THE Filed  "environment" of TABLE  "processes"  OF "scm" DATABASE infomation.example:

 

"{"SPARK_IDENT_STRING":"spark","SPARK_WORKER_PORT":"7078","log_dir":"/dfs/var/log/sparkwork","SPARK_WORKER_MEMORY":"67108864","SPARK_WORKER_WEBUI_PORT":"18081","CDH_VERSION":"5","SPARK_WORKER_DIR":"/dfs/var/run/spark/work","ADDITIONAL_ARGS":"","ZK_QUORUM":"pd1test6c8g:2181,pd2test6c8g:2181,pnn2test6c16g:2181","SPARK_DAEMON_MEMORY":"67108864","CM_CSD_SCRIPT":"scripts/control.sh"}" 

 

 

I don't understnd why the log_dir of cloudera-monitor.properties  not change?