Support Questions

mike_bronson7 · ‎08-13-2018

on the following answer , we have explain how to rotate gc.log ( logs ) for metrics collector service

https://community.hortonworks.com/questions/194534/automatic-deletion-of-rotated-gclog-files.html

Now we need to do the same on the following services

HDFS

KAFKA

what should be the approach on that services ?

Michael-Bronson

apappu · ‎08-13-2018

@Michael Bronson

Just adding KAFKA_GC_LOG_OPTS="-XX:+UseGCLogFileRotation-XX:NumberOfGCLogFiles=5-XX:GCLogFileSize=2M" will not be sufficient - will have to use that variable during the service start.

Regarding HDFS:

In hadoop-env template wherever you find -XX:+PrintGCDetails parameter you can add -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M

For example:

From:

HADOOP_JOBTRACKER_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{jtnode_opt_newsize}} -XX:MaxNewSize={{jtnode_opt_maxnewsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xmx{{jtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -Dhadoop.mapreduce.jobsummary.logger=INFO,JSA ${HADOOP_JOBTRACKER_OPTS}"

To:

HADOOP_JOBTRACKER_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{jtnode_opt_newsize}} -XX:MaxNewSize={{jtnode_opt_maxnewsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xmx{{jtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -Dhadoop.mapreduce.jobsummary.logger=INFO,JSA ${HADOOP_JOBTRACKER_OPTS}"

View solution in original post

mike_bronson7 · ‎08-13-2018

for HDFS now its more clear , but for kafka , where I need to set the "-XX:+UseGCLogFileRotation-XX:NumberOfGCLogFiles=5-XX:GCLogFileSize=2M"  ?

Michael-Bronson

apappu · ‎08-13-2018

@Michael Bronson

For Kafka its not even enabled GC logging I guess - hence you can ignore that part. but you can set that like below.

In kafka-env section,

From

export KAFKA_KERBEROS_PARAMS="-Djavax.security.auth.useSubjectCredsOnly=false

to:

export KAFKA_KERBEROS_PARAMS="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M

If you satisfied with answers then please mark my comment as correct answer. this will help others as well.

Cloudera Community

Support Questions

how to rotate gc files for ambari services