Archives of Support Questions (Read Only)

mike_bronson7 · ‎08-13-2018

on the following answer , we have explain how to rotate gc.log ( logs ) for metrics collector service

https://community.hortonworks.com/questions/194534/automatic-deletion-of-rotated-gclog-files.html

Now we need to do the same on the following services

HDFS

KAFKA

what should be the approach on that services ?

Michael-Bronson

apappu · ‎08-13-2018

@Michael Bronson

Just adding KAFKA_GC_LOG_OPTS="-XX:+UseGCLogFileRotation-XX:NumberOfGCLogFiles=5-XX:GCLogFileSize=2M" will not be sufficient - will have to use that variable during the service start.

Regarding HDFS:

In hadoop-env template wherever you find -XX:+PrintGCDetails parameter you can add -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M

For example:

From:

HADOOP_JOBTRACKER_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{jtnode_opt_newsize}} -XX:MaxNewSize={{jtnode_opt_maxnewsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xmx{{jtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -Dhadoop.mapreduce.jobsummary.logger=INFO,JSA ${HADOOP_JOBTRACKER_OPTS}"

To:

HADOOP_JOBTRACKER_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{jtnode_opt_newsize}} -XX:MaxNewSize={{jtnode_opt_maxnewsize}} -Xloggc:{{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xmx{{jtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -Dhadoop.mapreduce.jobsummary.logger=INFO,JSA ${HADOOP_JOBTRACKER_OPTS}"

View solution in original post

mike_bronson7 · ‎08-13-2018

for HDFS now its more clear , but for kafka , where I need to set the "-XX:+UseGCLogFileRotation-XX:NumberOfGCLogFiles=5-XX:GCLogFileSize=2M"  ?

Michael-Bronson

apappu · ‎08-13-2018

@Michael Bronson

For Kafka its not even enabled GC logging I guess - hence you can ignore that part. but you can set that like below.

In kafka-env section,

From

export KAFKA_KERBEROS_PARAMS="-Djavax.security.auth.useSubjectCredsOnly=false

to:

export KAFKA_KERBEROS_PARAMS="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M

If you satisfied with answers then please mark my comment as correct answer. this will help others as well.

Cloudera Community

Archives of Support Questions (Read Only)

how to rotate gc files for ambari services