Optimized Mysql DB config for Ambari, Hive to handle more than 1500 concurrent connections

[#1]

The article provides a optimized mysql config (Usage of enterprise RDBMS is encouraged ) that can be used, in case mysql is used as the backed DB for Ambari, Hive and
Ranger. The configuration is good enough to handle 1000 concurrent users on HS2 and Ambari.

The config file my.cnf is generally located at /etc/my.cnf

my.cnf

******************************************************************************************************************************

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html

[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock


max_allowed_packet = 32M
max_connections = 2000
open_files_limit = 10000
tmp_table_size = 64M
max_heap_table_size = 64M
query_cache_type = ON
query_cache_size = 128M
table_open_cache_instances = 16
back_log = 1000

default_storage_engine = InnoDB

innodb-buffer-pool-size = 10G
innodb_log_file_size = 1024M
innodb_log_buffer_size = 16M
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_buffer_pool_instances = 4
innodb_thread_concurrency = 10
innodb_checksum_algorithm=crc32

# MYISAM in case there is one

key-buffer-size = 64M


# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

# Recommended in standard MySQL setup
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

*****************************************************************************************************

Key Take Aways :

1.innodb-buffer-pool-size = 10G This is one of the most important parameters, which says how much data can be cached in RAM.

2. mysqldump -u root --all-databases -p  > dump.sql will dump all the databases in a file (which can be used to recovery) and in the process will fetch all the data from disk and cache it in RAM, making all queries lightening fast.

3. max_connections = 2000 concurrent connection as the database is connected by all HS2 instances. This can become a limiting factor and may result in connection timeouts for clients connecting to HS2.

4. Do keep in mind the number of connection the mysql can handle can directly impact the number of users the HS2 server can handle concurrently.

5. For a details of all config parameters of mysql kindly follow the amazing file http://www.speedemy.com/files/mysql/my.cnf

6. https://community.hortonworks.com/articles/80635/optimize-ambari-performance-for-large-clusters.html