@Michael Bronson As per JVM specification if we have duplicate JVM flags set for a process then the parameter which is set at the end will take precedence. Hence it is always better to set our own custom values at the end of the JVM options (instead of middle), Else there are chances that our values might get overridden. So better to set the JVM parameters after the default parameters. as following: hadoop-env templte (inside {% else %} block because our JDK might be JDK8) export HADOOP_DATANODE_OPTS="-server -XX:ParallelGCThreads=4 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -Xloggc:/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xms{{dtnode_heapsize}} -Xmx{{dtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HADOOP_DATANODE_OPTS} -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10K -verbose:gc" ... View more

@Michael Bronson I see that your GC log file is around "3658" (around 3K) currently so it should get rotated after 10K. ... View more

@Michael Bronson Also after setting the value for this parameter to >8K please make sure that the correct value is being reflected to the JVM options when you run the "ps -ef | grep $PROCESS" command ... View more

@Michael Bronson I see that you are setting the value to 2K as -XX:GCLogFileSize=2K However as per the JVM option the minimum value for log rotation should be greater than 8KB, Can you please try with 8KB greater value. The size of the log file at which point the log will be rotated, must be >= 8K. https://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html ... View more

@Arjun Das Additionally in your Stacktrace i see the API " com.mysql.cj " being invoked. You might be using a higher version of mysql-connector-java JDBC Driver. As the new API " com.mysql.cj " can be seen in the StackTrace. Please make sure that you are using the correct MySQL JDBC driver. Like if you are using HDP 2.6.5 then MySQL 5.7 Database and respective JDBC driver version needs to be used. Similar HCC thread: https://community.hortonworks.com/questions/211807/install-mysql-connector-for-hive-metastore.html ... View more

@Arjun Das As this error seems to be coming while attmepting to connect to MySQL database. java.net.ConnectException: Connection refused (Connection refused) . com.mysql.cj.protocol.StandardSocketFactory.connect So please check few things: 0. The IPTables/Firewall is disabled on mysql server side so that ambari host can access the MySQL. # service iptables stop # sysctl firewalld stop 1. MySQL process is runnign fine and from ambari server host are you able to connect to it? # telnet $MYSQL_HOST $MYSQL_PORT Like: # telnet mysqlserver.example.com 3306 2. Also please make sure that you are using the correct MySQL JDBC driver JAR (correct version of Driver) and correct mysql Driver classname and the ambari server setip is executed fine as mentioned in: # ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/mysql/mysql-connector-java.jar https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_using_ambari_with_mysql_or_mariadb.html Still if you find any issue then please share the complete stack trace (partial stacktrace will not give much idea) and also please share the ambari.properties. # grep 'jdbc' /etc/ambari-server/conf/ambari.properties . ... View more

@Taehyeon Lee You can find more detailed Ambari Upgrade Path here: Ambari 2.1.2 to Ambari 2.5 https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-upgrade/content/preparing_to_upgrade_ambari.html Then Ambari 2.5 to Ambari 2.6 https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/preparing_to_upgrade_ambari.html and then From ambari 2.6 to Ambari 2.7 https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-upgrade/content/preparing_to_upgrade_ambari.html ... View more

@li Hao Additionally the "host_in_memory_aggregation" value is actually getting populated from "ams-site" as following: # grep 'timeline.metrics.host.inmemory.aggregation' /var/lib/ambari-server/resources/common-services/KAFKA//0.8.1/package/scripts/params.py host_in_memory_aggregation = str(default("/configurations/ams-site/timeline.metrics.host.inmemory.aggregation", True)).lower() . Hence please check your "ams-site" config in ambati UI to see if it has correct boolean values (true / false)? Ambari UI --> Ambari Metrics --> Configs --> Advanced --> Advanced ams-site --> "Enable in-memory aggreation on monitors" Property Correct the above property value and then restrat AMS service followed by Kafka restart and then you should no longer see the mentioned error. . ... View more