Slowness of Ambari Server UI

[#1]

I am fairly new to Ambari. Some of the rest API values are high which are causing slowness while using Ambari UI . If resetting the alerting threshold values will stop ambari from sending alerts. Can anyone suggest a way by which we can avoid the slowness of the Ambari UI?

[#2]

@Anurag Mishra

If you are getting ambari performance related errors then instead of alerting threshold values to suppress the alerts it will be best to findout what is causing the performance issue.

1. Check the AMbari Server Memory Uses?

# $JAVA_HOME/bin/jmap -heap $AMBARI_PID

2. If needed then enable the GC logging for ambari server by editing the "/var/lib/ambari-server/ambari-env.sh" and adding the following options related to GC logging in the "AMBARI_JVM_ARGS" as following:

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms2048m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Xloggc:/var/log/ambari-server/ambari_server_gc.log-`date +'%Y%m%d%H%M'` -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps'

3. Check the ambari server logs to find out if there are any repeated WARNING or Errors.

.

4. If the cluster size if large (many nodes) then please check if the threadpool setting inside the "/etc/ambari-server/conf/ambari.properties" is correct or not? Default values are 25.

# grep 'threadpool' /etc/ambari-server/conf/ambari.properties 
client.threadpool.size.max=25
agent.threadpool.size.max=25

.

5. If the cluster size if more than 200 nodes then tuning the Cache will help, Calculate the new, larger cache size, using the following relationship, where <cluster_size> is the number of nodes in the cluster.

ecCacheSizeValue=60*<cluster_size>

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-administration/content/ch_tuning_...

6. Also please check the Database logs. To see if there are any slowness from the Database or Network side. Sometimes if the DB is located on a remote machine then either due to N/W load or DB slow responsiveness ambari queries run slowly.

.

7. During slowness of ambari try running the following Curl call to see how much time does it take to get the cluster details:

# time curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://amb25101.example.com:8080/api/v1/clusters/plain_cluster
real    0m0.234s
user    0m0.009s
sys    0m0.017s

.

8. Additionally from Ambari 2.5 onwards you can also use the AMS (Grafana) to monitor Ambari Database/JVM Metrics.

• To enable Ambari Server metrics, make sure the following config file exists during Ambari Server start/restart - /etc/ambari-server/conf/metrics.properties.
• Currently, only 2 metric sources have been implemented - JVM Metric Source and Database Metric Source.
• To add / remove a metric source to be tracked the following config needs to be modified in the metrics.properties file.
  metric.sources=jvm,database
• Source specific configs are discussed in the metrics source section.

[#3]

time taken to fetch cluster details :

real 0m33.275s

user 0m0.009s

sys 0m0.181s

[#4]

I have checked the ambari server logs . The following was found to be in a critical state:

Performance Overview: Database Access (Request By Status): 1ms (OK) Database Access (Task Status Aggregation): 0ms (OK) REST API (Cluster): 20,519ms (CRITICAL) and threshold is set for 5 secs only.

Can you suggest any solution based on the above information ?

[#5]

@Anurag Mishra

REST API (Cluster): 20,519ms (CRITICAL)

The ambari rest API calls seems to be using 20 seconds which is high.

That usually happens if the cluster size if large and the client api thread pool is not set properly.

# grep 'threadpool' /etc/ambari-server/conf/ambari.properties 
client.threadpool.size.max=25
agent.threadpool.size.max=25

.

Can you please let us know your cluster size and also can you try increasing the above thread pool values to a bit high like just for testing

client.threadpool.size.max=100

.

client.threadpool.size.max - this property is being used by port listening for REST calls

[#6]

@Jay SenSharma our cluster size isnt very big , comprising of only 12 nodes and the threadpool sizes are:

client.threadpool.size.max=25

agent.threadpool.size.max=25

in this case, is it recommended to increase the above values ?

[#7]

@Jay SenSharma I used the top command to check the CPU utilizaton. one of the process ids (which is a java command) has cpu utilization upto 350% , however the overall cpu utilization for the cluster is idle at around 93-95 %. Can you please let me know if the slowness of the Ambari UI is because of the process id that has higher CPU utilization. If yes, what might be the issue that it is taking such a large amount of CPU utilization ?

[#8]

@Jay SenSharma our cluster size isnt very big , comprising of only 12 nodes and the threadpool sizes are:

client.threadpool.size.max=25

agent.threadpool.size.max=25

in this case, is it recommended to increase the above values ?