- Ambari manages it's Http Session information's using "AMBARISESSIONID". So if we want to logout the logged in user then we will need to find out it's "AMBARISESSIONID" cookie value. We can get it using browser debug tools like for Google chrome we can use Menu -> More Tools --> Developer Tools The we can go to the "Network" tab of the developer tool and then on the right hand side we should see the Cookies and other request headers. Here i have found that the value for the "AMBARISESSIONID" Http Session Cookie is "1t4o6q0ph8j23etnl3e6ig5b3" hence now we can simply make a curl call as following in order to get this user logged out: $ curl -u admin:admin -i -H 'X-Requested-By:ambari' -H "Cookie: AMBARISESSIONID=1t4o6q0ph8j23etnl3e6ig5b3" -X GET http://c6401.ambari.apache.org:8080/api/v1/logout HTTP/1.1 200 OK X-Frame-Options: DENY X-XSS-Protection: 1; mode=block User: admin Content-Type: text/plain Content-Length: 0 Server: Jetty(8.1.19.v20160209) - As soon as the above curl command is executed we will see that the user is logged out from the ambari UI on the browser. . We can also enable the Http Session logging in the ambari server log by editing the file "/etc/ambari-server/conf/log4j.properties" and adding the following line at the end. log4j.logger.org.eclipse.jetty.server.session=DEBUG Now When we will restart ambari server then we can see that the logged in users sessions are printed there: 07 Feb 2017 05:21:20,283 DEBUG [ambari-client-thread-29] session:275 - Got Session ID 1t4o6q0ph8j23etnl3e6ig5b3 from cookie With the above approach we can get all the active Session IDs and can force logout them all. However enabling DEBUG logging will causes excessive log event generation so we need to be careful there. ... View more

While accessing the Zeppelin View is shows the "zeppelin service is not running". But when we check at the back end then we find that the Zeppelin server was running. PID file has the correct PID information mentioned in it. - As we see that Zepplin Notebok is running via ambari as well. But still when we try to access the Zeppelin View it shows failure saying "Zeppelin service is not running". We will need to check the Zeppelin log to see if it shows any error? We see the following kind of error: INFO [2016-12-01 09:42:08,926] ({main} ZeppelinServer.java[setupWebAppContext]:266) - ZeppelinServer Webapp path: /usr/hdp/current/zeppelin-server/webapps INFO [2016-12-01 09:42:09,331] ({main} ZeppelinServer.java[main]:114) - Starting zeppelin server INFO [2016-12-01 09:42:09,333] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210 WARN [2016-12-01 09:42:09,367] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@69b794e2{/,null,null}{/usr/hdp/current/zeppelin-server/lib/zeppelin-web-0.6.0.2.5.0.0-1133.war} java.lang.IllegalStateException: Failed to delete temp dir /usr/hdp/2.5.0.0-1133/zeppelin/webapps at org.eclipse.jetty.webapp.WebInfConfiguration.configureTempDirectory(WebInfConfiguration.java:372) at org.eclipse.jetty.webapp.WebInfConfiguration.resolveTempDirectory(WebInfConfiguration.java:260) at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:69) at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:468) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:504) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.server.Server.start(Server.java:387) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:354) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:116) . . INFO [2016-12-01 09:42:09,390] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@28cab3cc{HTTP/1.1}{0.0.0.0:9995} INFO [2016-12-01 09:42:09,391] ({main} Server.java[doStart]:379) - Started @1094ms . However the port was opened fine but the the Zeppelin WebAppContext was not initialized properly due to the above error. So Zeppeline View was showing "Service check failed" error with message "zeppelin service is not running". In this case check we will need to check if the "/usr/hdp/2.5.0.0-1133/zeppelin/webapps" has proper permission/ownership as "zeppelin:hadoop" the user who is running zeppelin something like following: INCORRECT: # ls -l /usr/hdp/2.5.0.0-1133/zeppelin/web* drwxr-xr-x. 3 root root 4096 Dec 1 09:37 webapps . CORRECT: # ls -l /usr/hdp/2.5.0.0-1133/zeppelin/web* drwxr-xr-x. 10 zeppelin hadoop 4096 Dec 1 09:59 webapp . ... View more

Ambari allows its users to have different configurations for different hosts for different components via configuration groups. Ambari initially assigns all hosts in your cluster to one, default configuration group for each service you install. When using Configuration groups, it enforces configuration properties that allow override, based on installed components for the selected service and group. For more information on this please refer to: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_config_groups.html Here we will try to make a very small change in "flume-conf" content and will apply it to specific host ("erie3.example.com") where as all the other hosts will be using the default "flume-conf" configuration. We can simply use the following ambari API to list all the config_groups. http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/config_groups Example: http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups . Step-1). ======== Lets see the current "flume-conf" from ambari which looks something like following and applied to all the Ambari Hosts: # Flume agent config sandbox.sources = eventlog sandbox.channels = file_channel sandbox.sinks = sink_to_hdfs # Define / Configure source sandbox.sources.eventlog.type = exec sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo.log sandbox.sources.eventlog.restart = true sandbox.sources.eventlog.batchSize = 1000 #sandbox.sources.eventlog.type = seq # HDFS sinks sandbox.sinks.sink_to_hdfs.type = hdfs sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 1000 # Use a channel which buffers events in memory sandbox.channels.file_channel.type = file sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint sandbox.channels.file_channel.dataDirs = /var/flume/data # Bind the source and sink to the channel sandbox.sources.eventlog.channels = file_channel sandbox.sinks.sink_to_hdfs.channel = file_channel . Step-2). ======== Now suppose for Host ("erie3.example.com") we want to run the flume with a slightly different properly like [sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log] So in order to achieve that we will need to create a "config_group" json data which we will need to push to Ambari. Here we will create a file like "/tmp/erie3_flume_conf.json" [ { "ConfigGroup": { "cluster_name": "ErieCluster", "group_name": "cfg_group_test1", "tag": "FLUME", "description": "FLUME configs for Changes", "hosts": [ { "host_name": "erie3.example.com" } ], "desired_configs": [ { "type": "flume-conf", "tag": "nextgen1", "properties": { "content": " # Flume agent config\r\n sandbox.sources = eventlog \r\n sandbox.channels = file_channel \r\n sandbox.sinks = sink_to_hdfs \r\n \r\n # Define / Configure source \r\n sandbox.sources.eventlog.type = exec \r\n sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log \r\n sandbox.sources.eventlog.restart = true \r\n sandbox.sources.eventlog.batchSize = 1000 \r\n #sandbox.sources.eventlog.type = seq \r\n \r\n # HDFS sinks \r\n sandbox.sinks.sink_to_hdfs.type = hdfs \r\n sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream \r\n sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events \r\n sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog \r\n sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log \r\n sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 2000 \r\n # Use a channel which buffers events in memory \r\n sandbox.channels.file_channel.type = file \r\n sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint \r\n sandbox.channels.file_channel.dataDirs = /var/flume/data \r\n # Bind the source and sink to the channel \r\n sandbox.sources.eventlog.channels = file_channel \r\n sandbox.sinks.sink_to_hdfs.channel = file_channel \r\n " } }, { "type": "flume-env", "tag": "nextgen1" } ] } } ] **Notice:** If our configuration contains new line then the json data should use "\r\n" characters sequence. - Also notice that the above JSON configuration is specified for host "erie3.example.com" as following, (However we can have more comma separated hosts there): . "hosts": [ { "host_name": "erie3.example.com" } ] - We have also specified the name for our config group ["group_name": "cfg_group_test1",] . Step-3). ======== Lets now PUT these changes to ambari and see if it works: curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups OUTPUT: $ curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups { "resources" : [ { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups/252", "ConfigGroup" : { "id" : 252 } } ] } . ... View more

- Many times while working on some issues we want to take a quick look at the "/etc/ambari-server/conf/ambari.properties" without actually doing a SSH to the ambari host. - In order to access the Ambari Service using the following URL: (here "erie1.example.com" is ambari hostname) http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER . - We will find that "ambari.properties" values are being returned as part of the JSON response something like following: { "href" : "http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER", "RootServiceComponents" : { "component_name" : "AMBARI_SERVER", "component_version" : "2.4.1.0", "server_clock" : 1482322132, "service_name" : "AMBARI", "properties" : { "AAA" : "BBB", "agent.package.install.task.timeout" : "1800", "agent.stack.retry.on_repo_unavailability" : "false", "agent.stack.retry.tries" : "5", "agent.task.timeout" : "900", "agent.threadpool.size.max" : "25", "ambari-server.user" : "root", "ambari.python.wrap" : "ambari-python-wrap", "bootstrap.dir" : "/var/run/ambari-server/bootstrap", "bootstrap.script" : "/usr/lib/python2.6/site-packages/ambari_server/bootstrap.py", "bootstrap.setup_agent.script" : "/usr/lib/python2.6/site-packages/ambari_server/setupAgent.py", "check_database_skipped" : "false", "client.threadpool.size.max" : "25", "common.services.path" : "/var/lib/ambari-server/resources/common-services", "custom.action.definitions" : "/var/lib/ambari-server/resources/custom_action_definitions", "extensions.path" : "/var/lib/ambari-server/resources/extensions", "http.strict-transport-security" : "max-age=31536000", "http.x-frame-options" : "DENY", "http.x-xss-protection" : "1; mode=block", "java.home" : "/usr/jdk64/jdk1.8.0_60", "java.releases" : "jdk1.8,jdk1.7", "java.version" : "1.8", "jce.download.supported" : "true", "jce.name" : "jce_policy-8.zip", "jdk.download.supported" : "true", "jdk.name" : "jdk-8u60-linux-x64.tar.gz", "jdk1.7.desc" : "Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7", "jdk1.7.dest-file" : "jdk-7u67-linux-x64.tar.gz", "jdk1.7.home" : "/usr/jdk64/", "jdk1.7.jcpol-file" : "UnlimitedJCEPolicyJDK7.zip", "jdk1.7.jcpol-url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip", "jdk1.7.re" : "(jdk.*)/jre", "jdk1.7.url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz", "jdk1.8.desc" : "Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8", "jdk1.8.dest-file" : "jdk-8u60-linux-x64.tar.gz", "jdk1.8.home" : "/usr/jdk64/", "jdk1.8.jcpol-file" : "jce_policy-8.zip", "jdk1.8.jcpol-url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip", "jdk1.8.re" : "(jdk.*)/jre", "jdk1.8.url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u60-linux-x64.tar.gz", "jdk_location" : "http://erie1.example.com:8080/resources/", "kerberos.keytab.cache.dir" : "/var/lib/ambari-server/data/cache", "metadata.path" : "/var/lib/ambari-server/resources/stacks", "mpacks.staging.path" : "/var/lib/ambari-server/resources/mpacks", "org.apache.ambari.contrib.snmp.script" : "/root/SNMP_Demo/snmp_mib_script.sh", "pid.dir" : "/var/run/ambari-server", "recommendations.artifacts.lifetime" : "1w", "recommendations.dir" : "/var/run/ambari-server/stack-recommendations", "resources.dir" : "/var/lib/ambari-server/resources", "rolling.upgrade.skip.packages.prefixes" : "", "security.server.disabled.ciphers" : "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5", "security.server.disabled.protocols" : "SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1.0|TLSv1.1", "security.server.keys_dir" : "/var/lib/ambari-server/keys", "server.connection.max.idle.millis" : "900000", "server.execution.scheduler.isClustered" : "false", "server.execution.scheduler.maxDbConnections" : "5", "server.execution.scheduler.maxThreads" : "5", "server.execution.scheduler.misfire.toleration.minutes" : "480", "server.fqdn.service.url" : "http://169.254.169.254/latest/meta-data/public-hostname", "server.http.session.inactive_timeout" : "1800", "server.jdbc.connection-pool" : "internal", "server.jdbc.database" : "postgres", "server.jdbc.database_name" : "ambari", "server.jdbc.postgres.schema" : "ambari", "server.jdbc.user.name" : "ambari", "server.jdbc.user.passwd" : "/etc/ambari-server/conf/password.dat", "server.os_family" : "redhat6", "server.os_type" : "redhat6", "server.persistence.type" : "local", "server.stages.parallel" : "true", "server.task.timeout" : "1200", "server.tmp.dir" : "/var/lib/ambari-server/data/tmp", "server.version.file" : "/var/lib/ambari-server/resources/version", "shared.resources.dir" : "/usr/lib/ambari-server/lib/ambari_commons/resources", "skip.service.checks" : "false", "stackadvisor.script" : "/var/lib/ambari-server/resources/scripts/stack_advisor.py", "ulimit.open.files" : "10000", "user.inactivity.timeout.default" : "0", "user.inactivity.timeout.role.readonly.default" : "0", "views.ambari.request.connect.timeout.millis" : "60000", "views.ambari.request.read.timeout.millis" : "45000", "views.http.strict-transport-security" : "max-age=31536000", "views.http.x-frame-options" : "SAMEORIGIN", "views.http.x-xss-protection" : "1; mode=blocki", "views.request.connect.timeout.millis" : "30000", "views.request.read.timeout.millis" : "60000", "webapp.dir" : "/usr/lib/ambari-server/web" } }, "hostComponents" : [ { "href" : "http://erie1.example.com:8080/api/v1/services/AMBARI/hosts/erie1.example.com/hostComponents/AMBARI_SERVER", "RootServiceHostComponents" : { "component_name" : "AMBARI_SERVER", "host_name" : "erie1.example.com", "service_name" : "AMBARI" } } ] } . **NOTE:** The HttpResponse that we are getting only shows the current state of the "/etc/ambari-server/conf/ambari.properties", Which means even if we make a new change in the properties file (like adding a new property AAA = BBB) it will be immediately reflected when we get the HttpResponse. But ambari-server will read this properties file only during the restart. So any changes made to the ambari.properties requires a restart. . . ... View more

In Ambari it's possible to supply custom properties for both eclipselink and the database driver through "/etc/ambari-server/conf/ambari.properties". Following are the eclipselink properties that can be configured using a prefix of "server.persistence.properties.". The following eclipselink properties should be defined with the prefix prepended in front of them. server.persistence.properties.eclipselink.jdbc.batch-writing.size=25 server.persistence.properties.eclipselink.profiler=QueryMonitor - The "QueryMonitor" is used to measure query executions and cache hits. This can be useful for performance analysis in a complex system. The batch writing, this value is the number of statements to batch (default: 100) . With the "QueryMonitor" we can get the following kind of logging inside the "ambari-server.out" org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByHostAndName 0 1336 org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByNameAndNoHost 0 33 org.apache.ambari.server.orm.entities.AlertCurrentEntity-findAll 0 6 org.apache.ambari.server.orm.entities.AlertCurrentEntity-update 0 1365 org.apache.ambari.server.orm.entities.AlertDefinitionEntity-AlertDefinitionEntity.findAllEnabledInCluster 0 org.apache.ambari.server.orm.entities.StackEntity-StackEntity.findByNameAndVersion 0 11 org.apache.ambari.server.orm.entities.StackEntity-findByPrimaryKey 975 0 org.apache.ambari.server.orm.entities.StageEntity-findAll 0 1 org.apache.ambari.server.orm.entities.StageEntity-findByPrimaryKey 33 17 org.apache.ambari.server.orm.entities.TopologyHostGroupEntity-findByPrimaryKey 8 4 org.apache.ambari.server.orm.entities.TopologyHostGroupEntity-topologyHostGroupEntities 0 3 org.apache.ambari.server.orm.entities.TopologyHostInfoEntity-topologyHostInfoEntities 0 8 . Instead of "QueryMonitor" We also use native EclipseLink "PerformanceMonitor" to count how many queries are actually hitting the DB. The performance monitor can also be enabled in ambari through "/etc/ambari-server/conf/ambari.properties" using the below property: server.persistence.properties.eclipselink.profiler=PerformanceMonitor . In general the PerformanceMonitor statics contains three sets of information: Info: Statistics that are constant informational data, such as the session name, or time of login. Counter: Statistics that are cumulative counters of total operations, such as cache hits, or query executions. Timer: Statistics that are cumulative measurements of total time (in nano seconds) for a specific type of operation, reading, writing, database operations. Statistics are generally grouped in total and also by query type, query class, and query name. Counters and timers are generally recorded for the same operations, so the time per operation could also be calculated. . We can see the output of the performance of eclipselink in the "/var/log/ambari-server/ambari-server.out" as following Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components 3,526,260 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:ObjectBuilding 1,490,069 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:QueryPreparation 249,963 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:RowFetch 59,732 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:SqlGeneration 66,874 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:SqlPrepare 102,031 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:StatementExecute 1,347,424 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations 5,639,061 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:QueryPreparation 294,815 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:RowFetch 10,683 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:SqlGeneration 70,524 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:SqlPrepare 120,638 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:StatementExecute 4,918,040 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups 1,659,392 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:ObjectBuilding 376,399 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:QueryPreparation 511,305 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:RowFetch 21,179 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:SqlGeneration 66,966 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:SqlPrepare 26,478 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:StatementExecute 626,749 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null 3,762,530 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:ObjectBuilding 2,979,561 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:QueryPreparation 216,243 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:RowFetch 21,124 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:SqlGeneration 65,250 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:SqlPrepare 36,813 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:StatementExecute 597,791 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities 19,861,290 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:ObjectBuilding 6,763,784 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:QueryPreparation 341,067 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:RowFetch 526,064 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:SqlGeneration 97,978 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:SqlPrepare 946,199 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:StatementExecute 8,789,782 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion 17,151,509 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:ObjectBuilding 2,218,445 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:QueryPreparation 4,475,469 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:RowFetch 380,386 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:SqlGeneration 1,612,237 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:SqlPrepare 2,357,710 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:StatementExecute 5,216,896 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities 2,140,734 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:ObjectBuilding 778,102 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:QueryPreparation 169,452 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:RowFetch 32,767 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:SqlGeneration 45,134 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:SqlPrepare 48,039 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:StatementExecute 831,105 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities 3,773,810 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:ObjectBuilding 840,565 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:QueryPreparation 166,987 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:RowFetch 65,320 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:SqlGeneration 45,956 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:SqlPrepare 141,334 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:StatementExecute 2,081,051 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities 1,944,700 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:ObjectBuilding 1,098,335 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:QueryPreparation 257,534 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:RowFetch 18,207 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:SqlGeneration 56,939 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:SqlPrepare 15,456 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:StatementExecute 468,304 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities 2,475,972 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:ObjectBuilding 686,443 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:QueryPreparation 207,373 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:RowFetch 38,754 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:SqlGeneration 49,914 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:SqlPrepare 70,248 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:StatementExecute 1,141,554 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities 90,176,197 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:ObjectBuilding 87,029,141 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:QueryPreparation 190,624 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:RowFetch 102,952 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:SqlGeneration 50,110 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:SqlPrepare 154,729 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:StatementExecute 2,048,949 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId 408,099 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:ObjectBuilding 76,592 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:QueryPreparation 558,022 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:RowFetch 7,387 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:SqlGeneration 91,874 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:SqlPrepare 17,673 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:StatementExecute 214,411 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null 5,592,879 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:ObjectBuilding 2,160,103 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:QueryPreparation 1,914,031 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:RowFetch 405,417 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:SqlGeneration 61,219 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:SqlPrepare 24,379 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:StatementExecute 831,073 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances 34,420,101 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:ObjectBuilding 20,104,351 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:QueryPreparation 605,238 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:RowFetch 627,922 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:SqlGeneration 111,088 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:SqlPrepare 1,265,273 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:StatementExecute 8,921,351 Timer:ReadObjectQuery 805,462,215 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:blueprint 64,723 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity 1,757,487 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:ObjectBuilding 420,889 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:QueryPreparation 537,105 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:RowFetch 33,245 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:SqlGeneration 77,139 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:SqlPrepare 36,096 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:StatementExecute 1,073,086 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity 8,415,350 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:ObjectBuilding 395,952 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:QueryPreparation 2,150,084 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:RowFetch 70,515 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:SqlGeneration 112,038 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:SqlPrepare 117,634 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:StatementExecute 7,604,570 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null 166,183,273 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:ObjectBuilding 5,736,905 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:QueryPreparation 14,543,234 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:RowFetch 733,256 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:SqlGeneration 3,601,668 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:SqlPrepare 2,600,547 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:StatementExecute 43,609,711 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:clusterEntity 21,161,901 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:readClusterEntity 33,654,045 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:readClusterEntity:QueryPreparation 499,548 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:readClusterEntity:SqlGeneration 173,979 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterServiceEntity:clusterServiceEntity 2,353,290 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:clusterStateEntity 2,469,007 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:ObjectBuilding 797,353 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:QueryPreparation 368,203 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:RowFetch 56,276 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:SqlGeneration 109,224 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:SqlPrepare 65,662 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:StatementExecute 864,789 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ConfigGroupEntity:configGroupEntity 124,377 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity 29,109,973 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:ObjectBuilding 11,526,425 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:QueryPreparation 315,835 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:RowFetch 1,523,159 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:SqlGeneration 70,174 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:SqlPrepare 770,074 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:StatementExecute 12,513,542 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity 368,361,796 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:ObjectBuilding 117,592,614 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:QueryPreparation 290,883 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:RowFetch 1,907,832 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:SqlGeneration 84,574 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:SqlPrepare 6,321,676 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:StatementExecute 48,701,304 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentStateEntity:readHostComponentStateEntity 79,391,633 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentStateEntity:readHostComponentStateEntity:QueryPreparation 348,985 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentStateEntity:readHostComponentStateEntity:SqlGeneration 99,721 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostEntity:hostEntity 7,036,550 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroup 497,612 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:hostRoleCommand 854,614 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:hostRoleCommandEntity 83,187,729 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity 28,222,961 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:ObjectBuilding 43,005,022 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:QueryPreparation 174,345 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:RowFetch 13,071,909 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:SqlGeneration 50,848 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:SqlPrepare 1,461,549 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:StatementExecute 22,634,229 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:hostStateEntity 3,296,814 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity 6,485,183 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:ObjectBuilding 703,591 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:QueryPreparation 258,035 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:RowFetch 61,728 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:SqlGeneration 57,985 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:SqlPrepare 201,965 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:StatementExecute 1,514,444 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity 98,288,876 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:ObjectBuilding 7,443,764 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:QueryPreparation 2,337,384 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:RowFetch 1,226,558 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:SqlGeneration 1,314,624 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:SqlPrepare 8,209,609 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:StatementExecute 60,086,307 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion 1,187,366 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:ObjectBuilding 355,655 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:QueryPreparation 485,336 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:RowFetch 17,455 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:SqlGeneration 119,557 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:SqlPrepare 23,492 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:StatementExecute 644,119 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:ObjectBuilding 3,925,222 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:QueryPreparation 182,150 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:RowFetch 124,766 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:SqlGeneration 47,169 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:SqlPrepare 220,917 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:StatementExecute 7,113,579 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:request 12,361,888 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestOperationLevelEntity:requestOperationLevel 2,944,947 . ... View more

Many times we find that the operations (like start/stop/restart ...etc) are failing from ambari UI. In such cases if we want to troubleshoot what Ambari UI did to perform that operation or How the commands were executed. Then we can manually execute those same operations from the individual host with the help of "/var/lib/ambari-agent/data/command-xxx.json" file. - When we perform any operation from ambari UI like (Starting / Stopping Datanode) then we will notice that ambari shows the operation progress in the UI. There we can see basically the following two files. Example: stderr: /var/lib/ambari-agent/data/errors-952.txt stdout: /var/lib/ambari-agent/data/output-952.txt - Apart from the above files there is one more important file which ambari agent uses to execute the instructions/commands that are sent by the AmbariServer. We can find that specific file in the ambari-agent's "/var/lib/ambari-agent/data/command-xxx.json" file. /var/lib/ambari-agent/data/command-952.json - Here the "command-xxx.json" file has the command ID (xxx) same as the "errors-xxx.txt" & "output-xxx.txt" (as command-952.json, errors-952.txt, output-952.txt) - The "command-xxx.json" file contains lots of information's in it specially the "localComponents", "configuration_attributes", "configurationTags" and the command type. In this file we can find the data snippet something like following: }, "public_hostname": "c6402.ambari.apache.org", "commandId": "53-0", "hostname": "c6402.ambari.apache.org", "kerberosCommandParams": [], "serviceName": "HDFS", "role": "DATANODE", "forceRefreshConfigTagsBeforeExecution": false, "requestId": 53, "agentConfigParams": { "agent": { "parallel_execution": 0 } }, "clusterName": "ClusterDemo", "commandType": "EXECUTION_COMMAND", "taskId": 952, "roleParams": { "component_category": "SLAVE" }, "conf . . How to execute the same command from the host ("c6402.ambari.apache.org") where the operation was actually performed? . Step-1). ======= Login to the host in which the command was executed. Here it is "c6402.ambari.apache.org" which we can see in the ambari UI operations history. While stopping DataNode. ssh root@c6402.ambari.apache.org Step-2). ======= As in the operation we were performing DataNode Start operation hence we will be executing the "datanode.py" script. As following: [root@c6402 ambari-agent]# PATH=$PATH:/var/lib/ambari-agent/ [root@c6402 ambari-agent]# python2.6 /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py START /var/lib/ambari-agent/data/command-952.json /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package /tmp/Jay/tmp.txt ERROR /tmp/Jay/ . ** NOTICE: ** Here we have modified the PATH variable temporarily, Because if we will not edit it then while running the above command we might see the following error: We will need to set the PATH just to make sure that when we will try to execute the commands we have the "ambari-sudo.sh" present in the PATH. Else we might see the following kind of error while trying to execute the commands. This is because ambari agent executes these commands with the help of "" script. So that script must be available in the PATH. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 174, in <module> DataNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 58, in start import params File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params.py", line 25, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py", line 20, in <module> import status_params File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/status_params.py", line 53, in <module> hadoop_conf_dir = conf_select.get_hadoop_conf_dir() File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/conf_select.py", line 477, in get_hadoop_conf_dir select(stack_name, "hadoop", version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/conf_select.py", line 315, in select shell.checked_call(_get_cmd("set-conf-dir", package, version), logoutput=False, quiet=False, sudo=True) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'ambari-python-wrap /usr/bin/conf-select set-conf-dir --package hadoop --stack-version 2.5.0.0-1245 --conf-version 0' returned 127. /bin/bash: ambari-sudo.sh: command not found . - Here the "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py" script will have the following arguments: Script expects at least 6 arguments Usage: datanode.py <COMMAND> <JSON_CONFIG> <BASEDIR> <STROUTPUT> <LOGGING_LEVEL> <TMP_DIR> <COMMAND> command type (INSTALL/CONFIGURE/START/STOP/SERVICE_CHECK...) <JSON_CONFIG> path to command json file. Ex: /var/lib/ambari-agent/data/command-2.json <BASEDIR> path to service metadata dir. Ex: /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package <STROUTPUT> path to file with structured command output (file will be created). Ex:/tmp/my.txt <LOGGING_LEVEL> log level for stdout. Ex:DEBUG,INFO <TMP_DIR> temporary directory for executable scripts. Ex: /var/lib/ambari-agent/tmp . - Once we have executed the above commands then we can see that the DataNode is started exactly the same way how we start it from Ambari UI. It also helps us in troubleshooting if ambari-server & agent were not communicating well and to isolate the issue. Example: (OUTPUT) ================= [root@c6402 Jay]# python2.6 /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py START /var/lib/ambari-agent/data/command-952.json /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package /tmp/Jay/tmp.txt DEBUG /tmp/Jay/ 2016-12-17 08:35:27,489 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245 2016-12-17 08:35:27,489 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0 2016-12-17 08:35:27,489 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-12-17 08:35:27,506 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '') 2016-12-17 08:35:27,507 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-12-17 08:35:27,523 - checked_call returned (0, '') 2016-12-17 08:35:27,523 - Ensuring that hadoop has the correct symlink structure 2016-12-17 08:35:27,524 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-12-17 08:35:27,529 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-1245, current_cluster_version=2.5.0.0-1245 -> 2.5.0.0-1245 2016-12-17 08:35:27,530 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245 2016-12-17 08:35:27,531 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0 2016-12-17 08:35:27,531 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-12-17 08:35:27,548 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '') 2016-12-17 08:35:27,549 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-12-17 08:35:27,568 - checked_call returned (0, '') 2016-12-17 08:35:27,568 - Ensuring that hadoop has the correct symlink structure 2016-12-17 08:35:27,568 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-12-17 08:35:27,574 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1} 2.5.0.0-12452016-12-17 08:35:27,588 - checked_call returned (0, '2.5.0.0-1245', '') 2016-12-17 08:35:27,591 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2016-12-17 08:35:27,597 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2016-12-17 08:35:27,599 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,606 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml 2016-12-17 08:35:27,607 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,615 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,622 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml 2016-12-17 08:35:27,622 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,631 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'} 2016-12-17 08:35:27,632 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,639 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml 2016-12-17 08:35:27,639 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,644 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,651 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml 2016-12-17 08:35:27,652 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,658 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {'final': {'dfs.datanode.failed.volumes.tolerated': 'true', 'dfs.datanode.data.dir': 'true', 'dfs.namenode.name.dir': 'true', 'dfs.support.append': 'true', 'dfs.webhdfs.enabled': 'true'}}, 'configurations': ...} 2016-12-17 08:35:27,665 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml 2016-12-17 08:35:27,666 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,715 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'fs.defaultFS': 'true'}}, 'owner': 'hdfs', 'configurations': ...} 2016-12-17 08:35:27,721 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml 2016-12-17 08:35:27,721 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,738 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'} 2016-12-17 08:35:27,739 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'group': 'hadoop', 'mode': 0751} 2016-12-17 08:35:27,740 - Directory['/var/lib/ambari-agent/data/datanode'] {'create_parents': True, 'mode': 0755} 2016-12-17 08:35:27,744 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/proc/sys/fs/binfmt_misc', '/var/lib/nfs/rpc_pipefs']. 2016-12-17 08:35:27,744 - Mount point for directory /hadoop/hdfs/data is / 2016-12-17 08:35:27,744 - Mount point for directory /hadoop/hdfs/data is / 2016-12-17 08:35:27,744 - Last mount for /hadoop/hdfs/data in the history file is / 2016-12-17 08:35:27,744 - Will manage /hadoop/hdfs/data since it's on the same mount point: / 2016-12-17 08:35:27,745 - Forcefully ensuring existence and permissions of the directory: /hadoop/hdfs/data 2016-12-17 08:35:27,745 - Directory['/hadoop/hdfs/data'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'ignore_failures': True, 'mode': 0755, 'owner': 'hdfs'} 2016-12-17 08:35:27,749 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/proc/sys/fs/binfmt_misc', '/var/lib/nfs/rpc_pipefs']. 2016-12-17 08:35:27,749 - Mount point for directory /hadoop/hdfs/data is / 2016-12-17 08:35:27,749 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': '\n# This file keeps track of the last known mount-point for each dir.\n# It is safe to delete, since it will get regenerated the next time that the component of the service starts.\n# However, it is not advised to delete this file since Ambari may\n# re-create a dir that used to be mounted on a drive but is now mounted on the root.\n# Comments begin with a hash (#) symbol\n# dir,mount_point\n/hadoop/hdfs/data,/\n', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2016-12-17 08:35:27,750 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755} 2016-12-17 08:35:27,751 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True} 2016-12-17 08:35:27,752 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True} 2016-12-17 08:35:27,752 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'} 2016-12-17 08:35:27,760 - Skipping File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] due to not_if 2016-12-17 08:35:27,761 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'} 2016-12-17 08:35:27,767 - Skipping Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] due to not_if 2016-12-17 08:35:27,796 - Command: /usr/bin/hdp-select status hadoop-hdfs-datanode > /tmp/tmp6ogtMa Output: hadoop-hdfs-datanode - 2.5.0.0-1245 . . ... View more

[Related Article On Ambari Server Tuning : https://community.hortonworks.com/articles/131670/ambari-server-performance-tuning-troubleshooting-c.html ] -The jcmd utility comes with the JDK and is present inside the "$JAVA_HOME/bin". It is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings, troubleshoot, and diagnose JVM and Java Applications. Following are the conditions for using this utility. - 1. It must be used on the same machine where the JVM is running - 2. Only a user who own the JVM process can connect to is using this utility. This utility can help us in getting many details about the JVM process. Some of the most useful information's are as following: Syntax: jcmd $PID $ARGUMENT Example1: Classes taking the most memory are listed at the top, and classes are listed in a descending order. /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.class_histogram > /tmp/22421_ClassHistogram.txt Example2: Generate Heap Dump /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.heap_dump /tmp/test123.hprof Example3: Explicitly request JVM to trigger a Garbage Collection Cycle. /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.run Example4: Generate Thread dump. usr/jdk64/jdk1.8.0_60/bin/jcmd $PID Thread.print Example5: List JVM properties. /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID VM.system_properties Example6: The Command line options along with the CLASSPATH setting. /usr/jdk64/jdk1.8.0_60/bin/jcmd $ PID VM.command_line **NOTE:** To use few specific features offered by "jcmd" tool the "-XX:+UnlockDiagnosticVMOptions" JVM option need to be enabled. . When to Collect Thread Dumps? --------------------------------------------------------- Here now we will see a very common scenario when we find that the JVM process is talking a lots of time in processing the request. Many times we see that the JVM process is stuck/slow or completely Hung. In such scenario in order to investigate the root cause of the slowness we need to collect the thread dumps of the JVM process which will tell us about the various activities those threads are actually performing. Sometimes some threads are involved in some very high CPU intensive operations which also might cause a slowness in getting the response. So We should collect the thread dump as well as the CPU data using "top" command. Few things to consider while collecting the thread dumps: - 1. Collect the thread dump when we see the issue (slowness, stuck/ hung scenario ...etc). . - 2. Mostly a single thread dump is not very useful. So whenever we collect the thread dump then we should collect at least 5-6 thread dumps. In some interval like collect 5-6 thread dumps in 10 seconds interval. Like that we will get around 5-6 thread dumps in 1 minute. - 3. If we are also investigating that few threads might be consuming high CPU cycles then in order to find the APIs that are actually consuming the high CPU we must collect the Thread dump as well as the "top" command output data almost at the same time. . - In order to make this easy we can use a simple script "threaddump_cpu_with_cmd.sh" and use it for out troubleshooting & JVM data collection. The following script can be downloaded from: https://github.com/jaysensharma/MiddlewareMagicDemos/tree/master/HDP_Ambari/JVM #!/bin/sh # Takes the JavaApp PID as an argument. # Make sure you set JAVA_HOME # Create thread dumps a specified number of times (i.e. LOOP) and INTERVAL. # Thread dumps will be collected in the file "jcmd_threaddump.out", in the same directory from where this script is been executed. # Usage: # sudo - $user_Who_Owns_The_JavaProcess # ./threaddump_cpu_with_cmd.sh <JAVA_APP_PID> # # # Example: # NameNode PID is "5752" and it is started by user "hdfs" then run this utility as following: # # su -l hdfs -c "/tmp/threaddump_cpu_with_cmd.sh 5752" ################################################################################################ # Number of times to collect data. Means total number of thread dumps. LOOP=10 # Interval in seconds between data points. INTERVAL=10 # Where to generate the threddump & top output files. WHERE_TO_GENERATE_OUTPUT_FILES="/tmp" # Setting the Java Home, by giving the path where your JDK is kept # USERS MUST SET THE JAVA_HOME before running this scripta s following: JAVA_HOME=/usr/jdk64/jdk1.8.0_60 echo "Writing CPU data log files to Directory: $WHERE_TO_GENERATE_OUTPUT_FILES" for ((i=1; i <= $LOOP; i++)) do #$JAVA_HOME/bin/jstack -l $1 >> jstack_threaddump.out $JAVA_HOME/bin/jcmd $1 Thread.print >> $WHERE_TO_GENERATE_OUTPUT_FILES/jcmd_threaddump.out _now=$(date) echo "${_now}" >> $WHERE_TO_GENERATE_OUTPUT_FILES/top_highcpu.out top -b -n 1 -H -p $1 >> $WHERE_TO_GENERATE_OUTPUT_FILES/top_highcpu.out echo "Collected 'top' output and Thread Dump #" $i if [ $i -lt $LOOP ]; then echo "Sleeping for $INTERVAL seconds." sleep $INTERVAL fi done - Get the file "jcmd_threaddump.out" and "top_highcpu.out" for analysis. . How to analyze the Thread dump Data? --------------------------------------------------------- You may have a look at one of my old blog article which explains "High CPU Utilization Finding Cause?" http://middlewaremagic.com/weblogic/?p=4884 . . Common Errors with "jcmd" utility. --------------------------------------------------------- While running the JCMD we might see the below mentioned error. Here the "5752" is the NameNode PID. [root@c6401 keys]# /usr/jdk64/jdk1.8.0_60/bin/jcmd 5752 help 5752: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106) at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63) at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208) at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:147) at sun.tools.jcmd.JCmd.main(JCmd.java:131) This error occurred because, JCMD utility allows to connect only to the JVM process that we own the process. In this case we see that the "NameNode" process is being owned by the "hdfs" user where as in the above command we are trying to connect to the NameNode process Via "jcmd" utility as "root" user. The root user here does not own the process, Hence we see the error. - "hdfs" user owned process # su -l hdfs -c "/usr/jdk64/jdk1.8.0_60/bin/jcmd -l" 5752 org.apache.hadoop.hdfs.server.namenode.NameNode 5546 org.apache.hadoop.hdfs.tools.DFSZKFailoverController 5340 org.apache.hadoop.hdfs.server.datanode.DataNode 4991 org.apache.hadoop.hdfs.qjournal.server.JournalNode . - "root" user owned process [root@c6401 keys]# /usr/jdk64/jdk1.8.0_60/bin/jcmd -l 1893 com.hortonworks.support.tools.server.SupportToolServer 6470 com.hortonworks.smartsense.activity.ActivityAnalyzerFacade 16774 org.apache.ambari.server.controller.AmbariServer 29100 sun.tools.jcmd.JCmd -l 6687 org.apache.zeppelin.server.ZeppelinServer More information about this utility can be found at: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html . . ... View more