Problem: There's a known bug in Ambari 2.4 and 2.5 that causes "ambari-server upgrade" to fail if the agent RPM is not upgraded first. E.g. stack trace: Using python /usr/bin/python Setup ambari-server Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 33, in from ambari_server.dbConfiguration import DATABASE_NAMES, LINUX_DBMS_KEYS_LIST File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 28, in from ambari_server.serverConfiguration import decrypt_password_for_alias, get_ambari_properties, get_is_secure, \ File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 36, in from ambari_commons.os_utils import run_os_command, search_file, set_file_permissions, parse_log4j_file ImportError: cannot import name parse_log4j_file Cause: This occurs because os_utils.py and other python files inside of /usr/lib/ambari-agent/lib/ambari_commons are upgraded by the agent's RPM and are used by the server's scripts to find which database to use. Solution: Note: Always back up your Ambari database before the upgrade. If ambari-agent is also present on the Ambari Server host, run "yum upgrade ambari-agent" (or equivalent for your OS). ... View more

Whether you're creating an Ambari cluster from scratch, taking over an existing cluster, or growing your cluster over time, it is imperative to tune Ambari and MySQL to work at a large scale of 1000-3000 Ambari Agents. Ambari Server Configs First, increase the memory used by Ambari. For large clusters, 8 GB of memory should be sufficient. If you have more than 10 concurrent users, increase it to 16 GB. Edit /var/lib/ambari-server/ambari-env.sh and change the -Xmn setting. export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms2048m -Xmx8192m Edit /etc/ambari-server/conf/ambari.properties with the following configs # The size of the Jetty connection pool used for handling incoming Ambari Agent requests. # 10 hosts => 25 # 50 hosts => 35 # 100 hosts => 75 # 500 hosts => 100 agent.threadpool.size.max=100 # Determines whether current alerts should be cached. # Enabling this can increase performance on large cluster, but can also result in lost alert data # if the cache is not flushed frequently. alerts.cache.enabled=true # The size of the alert cache. # Less than 50 hosts => 50000 # More than 50 hosts => 100000 alerts.cache.size=100000 # The number of threads used to handle alerts received from the Ambari Agents. # The value should be increased as the size of the cluster increases. # Less than 50 hosts => 2 # More than 50 hosts => 4 alerts.execution.scheduler.maxThreads=4 After performing these changes, restart Ambari Server. Move an existing Ambari DB from a spinning disk to a SSD It is highly suggested to use a Solid State Drive for the Ambari Database since this will be much faster. Check the throughput of the disk in which Ambari’s database (Postgres, MySQL, MariaDB, or Oracle) is on.Ideally, it should be a Solid State Drive or support at least 200 IOPS and be either on the same host as Ambari or only a 1-2 hops away. Type Details IOPS Throughput HDD 10,000 rpm SAS drive 175-210 100 MB/s SSD solid-state 500+ 500+ MB/s 1. ambari-server stop 2. Take a backup of the Ambari database, mysqldump -u root ambari > /tmp/ambari.sql 3. Stop MySQL server, copy its data, and change the directory. service mysqld stop cp -R -p /var/lib/mysql /mnt/disks/ssd/mysql cat /etc/my.cnf sed -ie 's/\/var\/lib\/mysql/\/mnt\/disks\/ssd\/mysql/g' /etc/my.cnf 4. Create symlink for sock file and start MySQL ln -s /mnt/disks/ssd/mysql/mysql.sock /var/lib/mysql/mysql.sock service mysqldstart 5. Ensure Ambari DB is accessible. mysql -u root -p show databases; use ambari; show tables; select count(*) from hosts; MySQL Optimizations First and foremost, if you're on an older version of MySQL, you can try to update it to MySQL 5.6 or 5.7, which has a lot of performance improvements. Connect to the MySQL DB and inspect these variables. E.g., SHOW VARIABLES LIKE 'name';These suggested values assume that only Ambari Database’s is on the MySQL Server.If you have other databases in the same MySQL Server, increment by these values. WARNING: Never stop MySQL server while Ambari Server is running. Variable Suggested Value innodb_log_buffer_size 512M innodb_buffer_pool_size 16G innodb_file_io_threads (deprecated in MySQL 5.5) 16 innodb_log_file_size 5M innodb_thread_concurrency 32 join_buffer_size 512M key_buffer_size 16G max_connections 500 max_allowed_packet 1024M max_heap_table_size 64M query_cache_limit 16M query_cache_size 512M read_rnd_buffer_size 128M sort_buffer_size 128M table_open_cache 1024 thread_cache_size 128 thread_stack 256K To change these values.1. Stop MySQL: service mysqld stop 2. Edit the configs in /etc/my.cnf , under the “[mysqld]” section (note, it may be in a different location). 3. Start MySQL: service mysqld start ... View more