While accessing the Zeppelin View is shows the "zeppelin service is not running". But when we check at the back end then we find that the Zeppelin server was running. PID file has the correct PID information mentioned in it. - As we see that Zepplin Notebok is running via ambari as well. But still when we try to access the Zeppelin View it shows failure saying "Zeppelin service is not running". We will need to check the Zeppelin log to see if it shows any error? We see the following kind of error: INFO [2016-12-01 09:42:08,926] ({main} ZeppelinServer.java[setupWebAppContext]:266) - ZeppelinServer Webapp path: /usr/hdp/current/zeppelin-server/webapps INFO [2016-12-01 09:42:09,331] ({main} ZeppelinServer.java[main]:114) - Starting zeppelin server INFO [2016-12-01 09:42:09,333] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210 WARN [2016-12-01 09:42:09,367] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@69b794e2{/,null,null}{/usr/hdp/current/zeppelin-server/lib/zeppelin-web-0.6.0.2.5.0.0-1133.war} java.lang.IllegalStateException: Failed to delete temp dir /usr/hdp/2.5.0.0-1133/zeppelin/webapps at org.eclipse.jetty.webapp.WebInfConfiguration.configureTempDirectory(WebInfConfiguration.java:372) at org.eclipse.jetty.webapp.WebInfConfiguration.resolveTempDirectory(WebInfConfiguration.java:260) at org.eclipse.jetty.webapp.WebInfConfiguration.preConfigure(WebInfConfiguration.java:69) at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:468) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:504) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.server.Server.start(Server.java:387) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:354) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:116) . . INFO [2016-12-01 09:42:09,390] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@28cab3cc{HTTP/1.1}{0.0.0.0:9995} INFO [2016-12-01 09:42:09,391] ({main} Server.java[doStart]:379) - Started @1094ms . However the port was opened fine but the the Zeppelin WebAppContext was not initialized properly due to the above error. So Zeppeline View was showing "Service check failed" error with message "zeppelin service is not running". In this case check we will need to check if the "/usr/hdp/2.5.0.0-1133/zeppelin/webapps" has proper permission/ownership as "zeppelin:hadoop" the user who is running zeppelin something like following: INCORRECT: # ls -l /usr/hdp/2.5.0.0-1133/zeppelin/web* drwxr-xr-x. 3 root root 4096 Dec 1 09:37 webapps . CORRECT: # ls -l /usr/hdp/2.5.0.0-1133/zeppelin/web* drwxr-xr-x. 10 zeppelin hadoop 4096 Dec 1 09:59 webapp . ... View more

Ambari allows its users to have different configurations for different hosts for different components via configuration groups. Ambari initially assigns all hosts in your cluster to one, default configuration group for each service you install. When using Configuration groups, it enforces configuration properties that allow override, based on installed components for the selected service and group. For more information on this please refer to: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/using_host_config_groups.html Here we will try to make a very small change in "flume-conf" content and will apply it to specific host ("erie3.example.com") where as all the other hosts will be using the default "flume-conf" configuration. We can simply use the following ambari API to list all the config_groups. http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/config_groups Example: http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups . Step-1). ======== Lets see the current "flume-conf" from ambari which looks something like following and applied to all the Ambari Hosts: # Flume agent config sandbox.sources = eventlog sandbox.channels = file_channel sandbox.sinks = sink_to_hdfs # Define / Configure source sandbox.sources.eventlog.type = exec sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo.log sandbox.sources.eventlog.restart = true sandbox.sources.eventlog.batchSize = 1000 #sandbox.sources.eventlog.type = seq # HDFS sinks sandbox.sinks.sink_to_hdfs.type = hdfs sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 1000 # Use a channel which buffers events in memory sandbox.channels.file_channel.type = file sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint sandbox.channels.file_channel.dataDirs = /var/flume/data # Bind the source and sink to the channel sandbox.sources.eventlog.channels = file_channel sandbox.sinks.sink_to_hdfs.channel = file_channel . Step-2). ======== Now suppose for Host ("erie3.example.com") we want to run the flume with a slightly different properly like [sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log] So in order to achieve that we will need to create a "config_group" json data which we will need to push to Ambari. Here we will create a file like "/tmp/erie3_flume_conf.json" [ { "ConfigGroup": { "cluster_name": "ErieCluster", "group_name": "cfg_group_test1", "tag": "FLUME", "description": "FLUME configs for Changes", "hosts": [ { "host_name": "erie3.example.com" } ], "desired_configs": [ { "type": "flume-conf", "tag": "nextgen1", "properties": { "content": " # Flume agent config\r\n sandbox.sources = eventlog \r\n sandbox.channels = file_channel \r\n sandbox.sinks = sink_to_hdfs \r\n \r\n # Define / Configure source \r\n sandbox.sources.eventlog.type = exec \r\n sandbox.sources.eventlog.command = tail -F /var/log/eventlog-demo-new-location.log \r\n sandbox.sources.eventlog.restart = true \r\n sandbox.sources.eventlog.batchSize = 1000 \r\n #sandbox.sources.eventlog.type = seq \r\n \r\n # HDFS sinks \r\n sandbox.sinks.sink_to_hdfs.type = hdfs \r\n sandbox.sinks.sink_to_hdfs.hdfs.fileType = DataStream \r\n sandbox.sinks.sink_to_hdfs.hdfs.path = /flume/events \r\n sandbox.sinks.sink_to_hdfs.hdfs.filePrefix = eventlog \r\n sandbox.sinks.sink_to_hdfs.hdfs.fileSuffix = .log \r\n sandbox.sinks.sink_to_hdfs.hdfs.batchSize = 2000 \r\n # Use a channel which buffers events in memory \r\n sandbox.channels.file_channel.type = file \r\n sandbox.channels.file_channel.checkpointDir = /var/flume/checkpoint \r\n sandbox.channels.file_channel.dataDirs = /var/flume/data \r\n # Bind the source and sink to the channel \r\n sandbox.sources.eventlog.channels = file_channel \r\n sandbox.sinks.sink_to_hdfs.channel = file_channel \r\n " } }, { "type": "flume-env", "tag": "nextgen1" } ] } } ] **Notice:** If our configuration contains new line then the json data should use "\r\n" characters sequence. - Also notice that the above JSON configuration is specified for host "erie3.example.com" as following, (However we can have more comma separated hosts there): . "hosts": [ { "host_name": "erie3.example.com" } ] - We have also specified the name for our config group ["group_name": "cfg_group_test1",] . Step-3). ======== Lets now PUT these changes to ambari and see if it works: curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups OUTPUT: $ curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Flume_Config_Group/erie3_flume_conf.json http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups { "resources" : [ { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/config_groups/252", "ConfigGroup" : { "id" : 252 } } ] } . ... View more

- If we want to run a service check for a particular service using Ambari APIs then one most important thins we will need to know in order to perform the service check using the APIs, that is the request payload. For every service the Payload of the request can be something in the following format: { "RequestInfo": { "context":"HDFS Service Check", "command":"HDFS_SERVICE_CHECK" }, "Requests/resource_filters":[ { "service_name":"HDFS" } ] } . If we do not know what will be the exact payload for different services then we can simply get it by using the "Browser Debugger Tools". It should show the "Form Data" section. - Now once we know the payload that we can use to POST for the service check execution to ambari then we can simply tun the command as following: Syntax: curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/PATH/TO/hdfs_service_check_payload.txt http://AMBARI_HOST:8080/api/v1/clusters/CLUSTER_NAME/requests . Example: $ curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/Users/jsensharma/Cases/Articles/Service_Checks_Using_APIs/hdfs_service_check_payload.txt http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests/424", "Requests" : { "id" : 424, "status" : "Accepted" } } . Here the file "hdfs_service_check_payload.txt" contains the payload mentioned above. . ... View more

- Many times while working on some issues we want to take a quick look at the "/etc/ambari-server/conf/ambari.properties" without actually doing a SSH to the ambari host. - In order to access the Ambari Service using the following URL: (here "erie1.example.com" is ambari hostname) http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER . - We will find that "ambari.properties" values are being returned as part of the JSON response something like following: { "href" : "http://erie1.example.com:8080/api/v1/services/AMBARI/components/AMBARI_SERVER", "RootServiceComponents" : { "component_name" : "AMBARI_SERVER", "component_version" : "2.4.1.0", "server_clock" : 1482322132, "service_name" : "AMBARI", "properties" : { "AAA" : "BBB", "agent.package.install.task.timeout" : "1800", "agent.stack.retry.on_repo_unavailability" : "false", "agent.stack.retry.tries" : "5", "agent.task.timeout" : "900", "agent.threadpool.size.max" : "25", "ambari-server.user" : "root", "ambari.python.wrap" : "ambari-python-wrap", "bootstrap.dir" : "/var/run/ambari-server/bootstrap", "bootstrap.script" : "/usr/lib/python2.6/site-packages/ambari_server/bootstrap.py", "bootstrap.setup_agent.script" : "/usr/lib/python2.6/site-packages/ambari_server/setupAgent.py", "check_database_skipped" : "false", "client.threadpool.size.max" : "25", "common.services.path" : "/var/lib/ambari-server/resources/common-services", "custom.action.definitions" : "/var/lib/ambari-server/resources/custom_action_definitions", "extensions.path" : "/var/lib/ambari-server/resources/extensions", "http.strict-transport-security" : "max-age=31536000", "http.x-frame-options" : "DENY", "http.x-xss-protection" : "1; mode=block", "java.home" : "/usr/jdk64/jdk1.8.0_60", "java.releases" : "jdk1.8,jdk1.7", "java.version" : "1.8", "jce.download.supported" : "true", "jce.name" : "jce_policy-8.zip", "jdk.download.supported" : "true", "jdk.name" : "jdk-8u60-linux-x64.tar.gz", "jdk1.7.desc" : "Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7", "jdk1.7.dest-file" : "jdk-7u67-linux-x64.tar.gz", "jdk1.7.home" : "/usr/jdk64/", "jdk1.7.jcpol-file" : "UnlimitedJCEPolicyJDK7.zip", "jdk1.7.jcpol-url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip", "jdk1.7.re" : "(jdk.*)/jre", "jdk1.7.url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz", "jdk1.8.desc" : "Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8", "jdk1.8.dest-file" : "jdk-8u60-linux-x64.tar.gz", "jdk1.8.home" : "/usr/jdk64/", "jdk1.8.jcpol-file" : "jce_policy-8.zip", "jdk1.8.jcpol-url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip", "jdk1.8.re" : "(jdk.*)/jre", "jdk1.8.url" : "http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u60-linux-x64.tar.gz", "jdk_location" : "http://erie1.example.com:8080/resources/", "kerberos.keytab.cache.dir" : "/var/lib/ambari-server/data/cache", "metadata.path" : "/var/lib/ambari-server/resources/stacks", "mpacks.staging.path" : "/var/lib/ambari-server/resources/mpacks", "org.apache.ambari.contrib.snmp.script" : "/root/SNMP_Demo/snmp_mib_script.sh", "pid.dir" : "/var/run/ambari-server", "recommendations.artifacts.lifetime" : "1w", "recommendations.dir" : "/var/run/ambari-server/stack-recommendations", "resources.dir" : "/var/lib/ambari-server/resources", "rolling.upgrade.skip.packages.prefixes" : "", "security.server.disabled.ciphers" : "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5", "security.server.disabled.protocols" : "SSL|SSLv2|SSLv2Hello|SSLv3|TLSv1.0|TLSv1.1", "security.server.keys_dir" : "/var/lib/ambari-server/keys", "server.connection.max.idle.millis" : "900000", "server.execution.scheduler.isClustered" : "false", "server.execution.scheduler.maxDbConnections" : "5", "server.execution.scheduler.maxThreads" : "5", "server.execution.scheduler.misfire.toleration.minutes" : "480", "server.fqdn.service.url" : "http://169.254.169.254/latest/meta-data/public-hostname", "server.http.session.inactive_timeout" : "1800", "server.jdbc.connection-pool" : "internal", "server.jdbc.database" : "postgres", "server.jdbc.database_name" : "ambari", "server.jdbc.postgres.schema" : "ambari", "server.jdbc.user.name" : "ambari", "server.jdbc.user.passwd" : "/etc/ambari-server/conf/password.dat", "server.os_family" : "redhat6", "server.os_type" : "redhat6", "server.persistence.type" : "local", "server.stages.parallel" : "true", "server.task.timeout" : "1200", "server.tmp.dir" : "/var/lib/ambari-server/data/tmp", "server.version.file" : "/var/lib/ambari-server/resources/version", "shared.resources.dir" : "/usr/lib/ambari-server/lib/ambari_commons/resources", "skip.service.checks" : "false", "stackadvisor.script" : "/var/lib/ambari-server/resources/scripts/stack_advisor.py", "ulimit.open.files" : "10000", "user.inactivity.timeout.default" : "0", "user.inactivity.timeout.role.readonly.default" : "0", "views.ambari.request.connect.timeout.millis" : "60000", "views.ambari.request.read.timeout.millis" : "45000", "views.http.strict-transport-security" : "max-age=31536000", "views.http.x-frame-options" : "SAMEORIGIN", "views.http.x-xss-protection" : "1; mode=blocki", "views.request.connect.timeout.millis" : "30000", "views.request.read.timeout.millis" : "60000", "webapp.dir" : "/usr/lib/ambari-server/web" } }, "hostComponents" : [ { "href" : "http://erie1.example.com:8080/api/v1/services/AMBARI/hosts/erie1.example.com/hostComponents/AMBARI_SERVER", "RootServiceHostComponents" : { "component_name" : "AMBARI_SERVER", "host_name" : "erie1.example.com", "service_name" : "AMBARI" } } ] } . **NOTE:** The HttpResponse that we are getting only shows the current state of the "/etc/ambari-server/conf/ambari.properties", Which means even if we make a new change in the properties file (like adding a new property AAA = BBB) it will be immediately reflected when we get the HttpResponse. But ambari-server will read this properties file only during the restart. So any changes made to the ambari.properties requires a restart. . . ... View more

###Finding Stale Configs - We can use the Ambari APIs to find the stale configurations. (Notice: ?HostRoles/stale_configs=true) http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true - If there is no stale configuration present in the cluster then we might see the items as empty "[]". Output may be as following: { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true", "items" : [ ] } . - Now i intentionally made some changes to the "Advanced core-site.xml" from ambari and added the following properties to it, Just to see if it changes the stale configuration list. hadoop.proxyuser.joy.hosts=* hadoop.proxyuser.joy.groups=* . - As we made some changes and the affected components are not yet restarted hence we will see that there are few stale configuration present in the cluster then we might see the output something like following (Notice: "stale_configs" : true ) { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/host_components?HostRoles/stale_configs=true", "items" : [ { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie3.example.com/host_components/APP_TIMELINE_SERVER", "HostRoles" : { "cluster_name" : "ErieCluster", "component_name" : "APP_TIMELINE_SERVER", "host_name" : "erie3.example.com", "stale_configs" : true }, "host" : { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie3.example.com" } }, { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie1.example.com/host_components/DATANODE", "HostRoles" : { "cluster_name" : "ErieCluster", "component_name" : "DATANODE", "host_name" : "erie1.example.com", "stale_configs" : true }, "host" : { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie1.example.com" } }, . . .======== SKIPPED some part of the output ======== . { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie2.example.com/host_components/ZKFC", "HostRoles" : { "cluster_name" : "ErieCluster", "component_name" : "ZKFC", "host_name" : "erie2.example.com", "stale_configs" : true }, "host" : { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/hosts/erie2.example.com" } } ] } . ###How to fix the Stale Config using Ambari APIs? ## Until Ambari 2.6.x ## Now we will see how we can fix the mentioned stale configuration changes. We will need to restart the affected components in order to refresh the stale configurations. We can use the following ambari APIs (it is applicable for Ambari 2.4 and above) # curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true"}]}' http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests . Example output, when the request will be successfully accepted by ambari: OUTPUT ======= { "href" : "http://erie1.example.com:8080/api/v1/clusters/ErieCluster/requests/427", "Requests" : { "id" : 427, "status" : "Accepted" } } . NOTE: The ["command":"RESTART"] will cause the affected services to be restarted completely (example NameNodes & DataNodes) hence before running the above command make sure that you do it in a maintenance windows (downtime expected) ## From Ambari 2.7.x onwards ## If you are using Ambari 2.7.0 then in the API call users should also pass the "HostRoles/cluster_name=XXXXX" cluster_name property as it is mandatory. # curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"RequestInfo":{"command":"RESTART","context":"Restart all required services","operation_level":"host_component"},"Requests/resource_filters":[{"hosts_predicate":"HostRoles/stale_configs=true&HostRoles/cluster_name=DemoCluster"}]}' http://latest1.example.com:8080/api/v1/clusters/DemoCluster/requests OUTPUT ======== { "href" : "http://latest1.example.com:8080/api/v1/clusters/DemoCluster/requests/314", "Requests" : { "id" : 314, "status" : "Accepted" } } . ... View more

In Ambari it's possible to supply custom properties for both eclipselink and the database driver through "/etc/ambari-server/conf/ambari.properties". Following are the eclipselink properties that can be configured using a prefix of "server.persistence.properties.". The following eclipselink properties should be defined with the prefix prepended in front of them. server.persistence.properties.eclipselink.jdbc.batch-writing.size=25 server.persistence.properties.eclipselink.profiler=QueryMonitor - The "QueryMonitor" is used to measure query executions and cache hits. This can be useful for performance analysis in a complex system. The batch writing, this value is the number of statements to batch (default: 100) . With the "QueryMonitor" we can get the following kind of logging inside the "ambari-server.out" org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByHostAndName 0 1336 org.apache.ambari.server.orm.entities.AlertCurrentEntity-AlertCurrentEntity.findByNameAndNoHost 0 33 org.apache.ambari.server.orm.entities.AlertCurrentEntity-findAll 0 6 org.apache.ambari.server.orm.entities.AlertCurrentEntity-update 0 1365 org.apache.ambari.server.orm.entities.AlertDefinitionEntity-AlertDefinitionEntity.findAllEnabledInCluster 0 org.apache.ambari.server.orm.entities.StackEntity-StackEntity.findByNameAndVersion 0 11 org.apache.ambari.server.orm.entities.StackEntity-findByPrimaryKey 975 0 org.apache.ambari.server.orm.entities.StageEntity-findAll 0 1 org.apache.ambari.server.orm.entities.StageEntity-findByPrimaryKey 33 17 org.apache.ambari.server.orm.entities.TopologyHostGroupEntity-findByPrimaryKey 8 4 org.apache.ambari.server.orm.entities.TopologyHostGroupEntity-topologyHostGroupEntities 0 3 org.apache.ambari.server.orm.entities.TopologyHostInfoEntity-topologyHostInfoEntities 0 8 . Instead of "QueryMonitor" We also use native EclipseLink "PerformanceMonitor" to count how many queries are actually hitting the DB. The performance monitor can also be enabled in ambari through "/etc/ambari-server/conf/ambari.properties" using the below property: server.persistence.properties.eclipselink.profiler=PerformanceMonitor . In general the PerformanceMonitor statics contains three sets of information: Info: Statistics that are constant informational data, such as the session name, or time of login. Counter: Statistics that are cumulative counters of total operations, such as cache hits, or query executions. Timer: Statistics that are cumulative measurements of total time (in nano seconds) for a specific type of operation, reading, writing, database operations. Statistics are generally grouped in total and also by query type, query class, and query name. Counters and timers are generally recorded for the same operations, so the time per operation could also be calculated. . We can see the output of the performance of eclipselink in the "/var/log/ambari-server/ambari-server.out" as following Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components 3,526,260 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:ObjectBuilding 1,490,069 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:QueryPreparation 249,963 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:RowFetch 59,732 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:SqlGeneration 66,874 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:SqlPrepare 102,031 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupComponentEntity:components:StatementExecute 1,347,424 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations 5,639,061 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:QueryPreparation 294,815 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:RowFetch 10,683 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:SqlGeneration 70,524 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:SqlPrepare 120,638 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupConfigEntity:configurations:StatementExecute 4,918,040 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups 1,659,392 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:ObjectBuilding 376,399 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:QueryPreparation 511,305 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:RowFetch 21,179 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:SqlGeneration 66,966 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:SqlPrepare 26,478 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroups:StatementExecute 626,749 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null 3,762,530 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:ObjectBuilding 2,979,561 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:QueryPreparation 216,243 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:RowFetch 21,124 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:SqlGeneration 65,250 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:SqlPrepare 36,813 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.HostVersionEntity:null:StatementExecute 597,791 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities 19,861,290 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:ObjectBuilding 6,763,784 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:QueryPreparation 341,067 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:RowFetch 526,064 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:SqlGeneration 97,978 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:SqlPrepare 946,199 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ServiceComponentDesiredStateEntity:serviceComponentDesiredStateEntities:StatementExecute 8,789,782 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion 17,151,509 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:ObjectBuilding 2,218,445 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:QueryPreparation 4,475,469 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:RowFetch 380,386 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:SqlGeneration 1,612,237 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:SqlPrepare 2,357,710 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.StackEntity:StackEntity.findByNameAndVersion:StatementExecute 5,216,896 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities 2,140,734 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:ObjectBuilding 778,102 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:QueryPreparation 169,452 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:RowFetch 32,767 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:SqlGeneration 45,134 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:SqlPrepare 48,039 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostGroupEntity:topologyHostGroupEntities:StatementExecute 831,105 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities 3,773,810 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:ObjectBuilding 840,565 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:QueryPreparation 166,987 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:RowFetch 65,320 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:SqlGeneration 45,956 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:SqlPrepare 141,334 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostInfoEntity:topologyHostInfoEntities:StatementExecute 2,081,051 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities 1,944,700 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:ObjectBuilding 1,098,335 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:QueryPreparation 257,534 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:RowFetch 18,207 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:SqlGeneration 56,939 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:SqlPrepare 15,456 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostRequestEntity:topologyHostRequestEntities:StatementExecute 468,304 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities 2,475,972 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:ObjectBuilding 686,443 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:QueryPreparation 207,373 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:RowFetch 38,754 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:SqlGeneration 49,914 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:SqlPrepare 70,248 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyHostTaskEntity:topologyHostTaskEntities:StatementExecute 1,141,554 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities 90,176,197 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:ObjectBuilding 87,029,141 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:QueryPreparation 190,624 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:RowFetch 102,952 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:SqlGeneration 50,110 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:SqlPrepare 154,729 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyLogicalTaskEntity:topologyLogicalTaskEntities:StatementExecute 2,048,949 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId 408,099 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:ObjectBuilding 76,592 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:QueryPreparation 558,022 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:RowFetch 7,387 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:SqlGeneration 91,874 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:SqlPrepare 17,673 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:TopologyRequestEntity.findByClusterId:StatementExecute 214,411 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null 5,592,879 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:ObjectBuilding 2,160,103 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:QueryPreparation 1,914,031 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:RowFetch 405,417 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:SqlGeneration 61,219 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:SqlPrepare 24,379 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.TopologyRequestEntity:null:StatementExecute 831,073 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances 34,420,101 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:ObjectBuilding 20,104,351 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:QueryPreparation 605,238 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:RowFetch 627,922 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:SqlGeneration 111,088 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:SqlPrepare 1,265,273 Timer:ReadAllQuery:org.apache.ambari.server.orm.entities.ViewInstanceEntity:instances:StatementExecute 8,921,351 Timer:ReadObjectQuery 805,462,215 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:blueprint 64,723 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity 1,757,487 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:ObjectBuilding 420,889 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:QueryPreparation 537,105 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:RowFetch 33,245 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:SqlGeneration 77,139 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:SqlPrepare 36,096 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.BlueprintEntity:readBlueprintEntity:StatementExecute 1,073,086 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity 8,415,350 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:ObjectBuilding 395,952 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:QueryPreparation 2,150,084 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:RowFetch 70,515 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:SqlGeneration 112,038 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:SqlPrepare 117,634 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:clusterConfigEntity:StatementExecute 7,604,570 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null 166,183,273 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:ObjectBuilding 5,736,905 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:QueryPreparation 14,543,234 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:RowFetch 733,256 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:SqlGeneration 3,601,668 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:SqlPrepare 2,600,547 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterConfigEntity:null:StatementExecute 43,609,711 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:clusterEntity 21,161,901 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:readClusterEntity 33,654,045 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:readClusterEntity:QueryPreparation 499,548 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterEntity:readClusterEntity:SqlGeneration 173,979 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterServiceEntity:clusterServiceEntity 2,353,290 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:clusterStateEntity 2,469,007 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:ObjectBuilding 797,353 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:QueryPreparation 368,203 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:RowFetch 56,276 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:SqlGeneration 109,224 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:SqlPrepare 65,662 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ClusterStateEntity:readClusterStateEntity:StatementExecute 864,789 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ConfigGroupEntity:configGroupEntity 124,377 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity 29,109,973 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:ObjectBuilding 11,526,425 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:QueryPreparation 315,835 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:RowFetch 1,523,159 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:SqlGeneration 70,174 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:SqlPrepare 770,074 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.ExecutionCommandEntity:readExecutionCommandEntity:StatementExecute 12,513,542 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity 368,361,796 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:ObjectBuilding 117,592,614 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:QueryPreparation 290,883 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:RowFetch 1,907,832 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:SqlGeneration 84,574 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:SqlPrepare 6,321,676 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentDesiredStateEntity:readHostComponentDesiredStateEntity:StatementExecute 48,701,304 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentStateEntity:readHostComponentStateEntity 79,391,633 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentStateEntity:readHostComponentStateEntity:QueryPreparation 348,985 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostComponentStateEntity:readHostComponentStateEntity:SqlGeneration 99,721 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostEntity:hostEntity 7,036,550 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostGroupEntity:hostGroup 497,612 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:hostRoleCommand 854,614 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:hostRoleCommandEntity 83,187,729 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity 28,222,961 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:ObjectBuilding 43,005,022 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:QueryPreparation 174,345 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:RowFetch 13,071,909 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:SqlGeneration 50,848 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:SqlPrepare 1,461,549 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostRoleCommandEntity:readHostRoleCommandEntity:StatementExecute 22,634,229 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:hostStateEntity 3,296,814 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity 6,485,183 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:ObjectBuilding 703,591 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:QueryPreparation 258,035 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:RowFetch 61,728 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:SqlGeneration 57,985 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:SqlPrepare 201,965 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.HostStateEntity:readHostStateEntity:StatementExecute 1,514,444 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity 98,288,876 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:ObjectBuilding 7,443,764 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:QueryPreparation 2,337,384 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:RowFetch 1,226,558 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:SqlGeneration 1,314,624 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:SqlPrepare 8,209,609 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.MetainfoEntity:readMetainfoEntity:StatementExecute 60,086,307 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion 1,187,366 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:ObjectBuilding 355,655 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:QueryPreparation 485,336 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:RowFetch 17,455 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:SqlGeneration 119,557 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:SqlPrepare 23,492 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RepositoryVersionEntity:repositoryVersion:StatementExecute 644,119 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:ObjectBuilding 3,925,222 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:QueryPreparation 182,150 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:RowFetch 124,766 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:SqlGeneration 47,169 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:SqlPrepare 220,917 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:readRequestEntity:StatementExecute 7,113,579 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestEntity:request 12,361,888 Timer:ReadObjectQuery:org.apache.ambari.server.orm.entities.RequestOperationLevelEntity:requestOperationLevel 2,944,947 . ... View more

Many times we find that the operations (like start/stop/restart ...etc) are failing from ambari UI. In such cases if we want to troubleshoot what Ambari UI did to perform that operation or How the commands were executed. Then we can manually execute those same operations from the individual host with the help of "/var/lib/ambari-agent/data/command-xxx.json" file. - When we perform any operation from ambari UI like (Starting / Stopping Datanode) then we will notice that ambari shows the operation progress in the UI. There we can see basically the following two files. Example: stderr: /var/lib/ambari-agent/data/errors-952.txt stdout: /var/lib/ambari-agent/data/output-952.txt - Apart from the above files there is one more important file which ambari agent uses to execute the instructions/commands that are sent by the AmbariServer. We can find that specific file in the ambari-agent's "/var/lib/ambari-agent/data/command-xxx.json" file. /var/lib/ambari-agent/data/command-952.json - Here the "command-xxx.json" file has the command ID (xxx) same as the "errors-xxx.txt" & "output-xxx.txt" (as command-952.json, errors-952.txt, output-952.txt) - The "command-xxx.json" file contains lots of information's in it specially the "localComponents", "configuration_attributes", "configurationTags" and the command type. In this file we can find the data snippet something like following: }, "public_hostname": "c6402.ambari.apache.org", "commandId": "53-0", "hostname": "c6402.ambari.apache.org", "kerberosCommandParams": [], "serviceName": "HDFS", "role": "DATANODE", "forceRefreshConfigTagsBeforeExecution": false, "requestId": 53, "agentConfigParams": { "agent": { "parallel_execution": 0 } }, "clusterName": "ClusterDemo", "commandType": "EXECUTION_COMMAND", "taskId": 952, "roleParams": { "component_category": "SLAVE" }, "conf . . How to execute the same command from the host ("c6402.ambari.apache.org") where the operation was actually performed? . Step-1). ======= Login to the host in which the command was executed. Here it is "c6402.ambari.apache.org" which we can see in the ambari UI operations history. While stopping DataNode. ssh root@c6402.ambari.apache.org Step-2). ======= As in the operation we were performing DataNode Start operation hence we will be executing the "datanode.py" script. As following: [root@c6402 ambari-agent]# PATH=$PATH:/var/lib/ambari-agent/ [root@c6402 ambari-agent]# python2.6 /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py START /var/lib/ambari-agent/data/command-952.json /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package /tmp/Jay/tmp.txt ERROR /tmp/Jay/ . ** NOTICE: ** Here we have modified the PATH variable temporarily, Because if we will not edit it then while running the above command we might see the following error: We will need to set the PATH just to make sure that when we will try to execute the commands we have the "ambari-sudo.sh" present in the PATH. Else we might see the following kind of error while trying to execute the commands. This is because ambari agent executes these commands with the help of "" script. So that script must be available in the PATH. Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 174, in <module> DataNode().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 58, in start import params File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params.py", line 25, in <module> from params_linux import * File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py", line 20, in <module> import status_params File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/status_params.py", line 53, in <module> hadoop_conf_dir = conf_select.get_hadoop_conf_dir() File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/conf_select.py", line 477, in get_hadoop_conf_dir select(stack_name, "hadoop", version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/conf_select.py", line 315, in select shell.checked_call(_get_cmd("set-conf-dir", package, version), logoutput=False, quiet=False, sudo=True) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of 'ambari-python-wrap /usr/bin/conf-select set-conf-dir --package hadoop --stack-version 2.5.0.0-1245 --conf-version 0' returned 127. /bin/bash: ambari-sudo.sh: command not found . - Here the "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py" script will have the following arguments: Script expects at least 6 arguments Usage: datanode.py <COMMAND> <JSON_CONFIG> <BASEDIR> <STROUTPUT> <LOGGING_LEVEL> <TMP_DIR> <COMMAND> command type (INSTALL/CONFIGURE/START/STOP/SERVICE_CHECK...) <JSON_CONFIG> path to command json file. Ex: /var/lib/ambari-agent/data/command-2.json <BASEDIR> path to service metadata dir. Ex: /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package <STROUTPUT> path to file with structured command output (file will be created). Ex:/tmp/my.txt <LOGGING_LEVEL> log level for stdout. Ex:DEBUG,INFO <TMP_DIR> temporary directory for executable scripts. Ex: /var/lib/ambari-agent/tmp . - Once we have executed the above commands then we can see that the DataNode is started exactly the same way how we start it from Ambari UI. It also helps us in troubleshooting if ambari-server & agent were not communicating well and to isolate the issue. Example: (OUTPUT) ================= [root@c6402 Jay]# python2.6 /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py START /var/lib/ambari-agent/data/command-952.json /var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package /tmp/Jay/tmp.txt DEBUG /tmp/Jay/ 2016-12-17 08:35:27,489 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245 2016-12-17 08:35:27,489 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0 2016-12-17 08:35:27,489 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-12-17 08:35:27,506 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '') 2016-12-17 08:35:27,507 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-12-17 08:35:27,523 - checked_call returned (0, '') 2016-12-17 08:35:27,523 - Ensuring that hadoop has the correct symlink structure 2016-12-17 08:35:27,524 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-12-17 08:35:27,529 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-1245, current_cluster_version=2.5.0.0-1245 -> 2.5.0.0-1245 2016-12-17 08:35:27,530 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.5.0.0-1245 2016-12-17 08:35:27,531 - Checking if need to create versioned conf dir /etc/hadoop/2.5.0.0-1245/0 2016-12-17 08:35:27,531 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2016-12-17 08:35:27,548 - call returned (1, '/etc/hadoop/2.5.0.0-1245/0 exist already', '') 2016-12-17 08:35:27,549 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.5.0.0-1245', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2016-12-17 08:35:27,568 - checked_call returned (0, '') 2016-12-17 08:35:27,568 - Ensuring that hadoop has the correct symlink structure 2016-12-17 08:35:27,568 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2016-12-17 08:35:27,574 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1} 2.5.0.0-12452016-12-17 08:35:27,588 - checked_call returned (0, '2.5.0.0-1245', '') 2016-12-17 08:35:27,591 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2016-12-17 08:35:27,597 - File['/etc/security/limits.d/hdfs.conf'] {'content': Template('hdfs.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2016-12-17 08:35:27,599 - XmlConfig['hadoop-policy.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,606 - Generating config: /usr/hdp/current/hadoop-client/conf/hadoop-policy.xml 2016-12-17 08:35:27,607 - File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,615 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,622 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-client.xml 2016-12-17 08:35:27,622 - File['/usr/hdp/current/hadoop-client/conf/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,631 - Directory['/usr/hdp/current/hadoop-client/conf/secure'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'cd_access': 'a'} 2016-12-17 08:35:27,632 - XmlConfig['ssl-client.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf/secure', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,639 - Generating config: /usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml 2016-12-17 08:35:27,639 - File['/usr/hdp/current/hadoop-client/conf/secure/ssl-client.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,644 - XmlConfig['ssl-server.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {}, 'configurations': ...} 2016-12-17 08:35:27,651 - Generating config: /usr/hdp/current/hadoop-client/conf/ssl-server.xml 2016-12-17 08:35:27,652 - File['/usr/hdp/current/hadoop-client/conf/ssl-server.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,658 - XmlConfig['hdfs-site.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'configuration_attributes': {'final': {'dfs.datanode.failed.volumes.tolerated': 'true', 'dfs.datanode.data.dir': 'true', 'dfs.namenode.name.dir': 'true', 'dfs.support.append': 'true', 'dfs.webhdfs.enabled': 'true'}}, 'configurations': ...} 2016-12-17 08:35:27,665 - Generating config: /usr/hdp/current/hadoop-client/conf/hdfs-site.xml 2016-12-17 08:35:27,666 - File['/usr/hdp/current/hadoop-client/conf/hdfs-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,715 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'mode': 0644, 'configuration_attributes': {'final': {'fs.defaultFS': 'true'}}, 'owner': 'hdfs', 'configurations': ...} 2016-12-17 08:35:27,721 - Generating config: /usr/hdp/current/hadoop-client/conf/core-site.xml 2016-12-17 08:35:27,721 - File['/usr/hdp/current/hadoop-client/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'} 2016-12-17 08:35:27,738 - File['/usr/hdp/current/hadoop-client/conf/slaves'] {'content': Template('slaves.j2'), 'owner': 'hdfs'} 2016-12-17 08:35:27,739 - Directory['/var/lib/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'group': 'hadoop', 'mode': 0751} 2016-12-17 08:35:27,740 - Directory['/var/lib/ambari-agent/data/datanode'] {'create_parents': True, 'mode': 0755} 2016-12-17 08:35:27,744 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/proc/sys/fs/binfmt_misc', '/var/lib/nfs/rpc_pipefs']. 2016-12-17 08:35:27,744 - Mount point for directory /hadoop/hdfs/data is / 2016-12-17 08:35:27,744 - Mount point for directory /hadoop/hdfs/data is / 2016-12-17 08:35:27,744 - Last mount for /hadoop/hdfs/data in the history file is / 2016-12-17 08:35:27,744 - Will manage /hadoop/hdfs/data since it's on the same mount point: / 2016-12-17 08:35:27,745 - Forcefully ensuring existence and permissions of the directory: /hadoop/hdfs/data 2016-12-17 08:35:27,745 - Directory['/hadoop/hdfs/data'] {'group': 'hadoop', 'cd_access': 'a', 'create_parents': True, 'ignore_failures': True, 'mode': 0755, 'owner': 'hdfs'} 2016-12-17 08:35:27,749 - Host contains mounts: ['/', '/proc', '/sys', '/dev/pts', '/dev/shm', '/boot', '/proc/sys/fs/binfmt_misc', '/var/lib/nfs/rpc_pipefs']. 2016-12-17 08:35:27,749 - Mount point for directory /hadoop/hdfs/data is / 2016-12-17 08:35:27,749 - File['/var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist'] {'content': '\n# This file keeps track of the last known mount-point for each dir.\n# It is safe to delete, since it will get regenerated the next time that the component of the service starts.\n# However, it is not advised to delete this file since Ambari may\n# re-create a dir that used to be mounted on a drive but is now mounted on the root.\n# Comments begin with a hash (#) symbol\n# dir,mount_point\n/hadoop/hdfs/data,/\n', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2016-12-17 08:35:27,750 - Directory['/var/run/hadoop'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0755} 2016-12-17 08:35:27,751 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True} 2016-12-17 08:35:27,752 - Directory['/var/log/hadoop/hdfs'] {'owner': 'hdfs', 'group': 'hadoop', 'create_parents': True} 2016-12-17 08:35:27,752 - File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] {'action': ['delete'], 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'} 2016-12-17 08:35:27,760 - Skipping File['/var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'] due to not_if 2016-12-17 08:35:27,761 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] {'environment': {'HADOOP_LIBEXEC_DIR': '/usr/hdp/current/hadoop-client/libexec'}, 'not_if': 'ambari-sudo.sh -H -E test -f /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid && ambari-sudo.sh -H -E pgrep -F /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid'} 2016-12-17 08:35:27,767 - Skipping Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode''] due to not_if 2016-12-17 08:35:27,796 - Command: /usr/bin/hdp-select status hadoop-hdfs-datanode > /tmp/tmp6ogtMa Output: hadoop-hdfs-datanode - 2.5.0.0-1245 . . ... View more

[Related Article On Ambari Server Tuning : https://community.hortonworks.com/articles/131670/ambari-server-performance-tuning-troubleshooting-c.html ] -The jcmd utility comes with the JDK and is present inside the "$JAVA_HOME/bin". It is used to send diagnostic command requests to the JVM, where these requests are useful for controlling Java Flight Recordings, troubleshoot, and diagnose JVM and Java Applications. Following are the conditions for using this utility. - 1. It must be used on the same machine where the JVM is running - 2. Only a user who own the JVM process can connect to is using this utility. This utility can help us in getting many details about the JVM process. Some of the most useful information's are as following: Syntax: jcmd $PID $ARGUMENT Example1: Classes taking the most memory are listed at the top, and classes are listed in a descending order. /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.class_histogram > /tmp/22421_ClassHistogram.txt Example2: Generate Heap Dump /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.heap_dump /tmp/test123.hprof Example3: Explicitly request JVM to trigger a Garbage Collection Cycle. /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID GC.run Example4: Generate Thread dump. usr/jdk64/jdk1.8.0_60/bin/jcmd $PID Thread.print Example5: List JVM properties. /usr/jdk64/jdk1.8.0_60/bin/jcmd $PID VM.system_properties Example6: The Command line options along with the CLASSPATH setting. /usr/jdk64/jdk1.8.0_60/bin/jcmd $ PID VM.command_line **NOTE:** To use few specific features offered by "jcmd" tool the "-XX:+UnlockDiagnosticVMOptions" JVM option need to be enabled. . When to Collect Thread Dumps? --------------------------------------------------------- Here now we will see a very common scenario when we find that the JVM process is talking a lots of time in processing the request. Many times we see that the JVM process is stuck/slow or completely Hung. In such scenario in order to investigate the root cause of the slowness we need to collect the thread dumps of the JVM process which will tell us about the various activities those threads are actually performing. Sometimes some threads are involved in some very high CPU intensive operations which also might cause a slowness in getting the response. So We should collect the thread dump as well as the CPU data using "top" command. Few things to consider while collecting the thread dumps: - 1. Collect the thread dump when we see the issue (slowness, stuck/ hung scenario ...etc). . - 2. Mostly a single thread dump is not very useful. So whenever we collect the thread dump then we should collect at least 5-6 thread dumps. In some interval like collect 5-6 thread dumps in 10 seconds interval. Like that we will get around 5-6 thread dumps in 1 minute. - 3. If we are also investigating that few threads might be consuming high CPU cycles then in order to find the APIs that are actually consuming the high CPU we must collect the Thread dump as well as the "top" command output data almost at the same time. . - In order to make this easy we can use a simple script "threaddump_cpu_with_cmd.sh" and use it for out troubleshooting & JVM data collection. The following script can be downloaded from: https://github.com/jaysensharma/MiddlewareMagicDemos/tree/master/HDP_Ambari/JVM #!/bin/sh # Takes the JavaApp PID as an argument. # Make sure you set JAVA_HOME # Create thread dumps a specified number of times (i.e. LOOP) and INTERVAL. # Thread dumps will be collected in the file "jcmd_threaddump.out", in the same directory from where this script is been executed. # Usage: # sudo - $user_Who_Owns_The_JavaProcess # ./threaddump_cpu_with_cmd.sh <JAVA_APP_PID> # # # Example: # NameNode PID is "5752" and it is started by user "hdfs" then run this utility as following: # # su -l hdfs -c "/tmp/threaddump_cpu_with_cmd.sh 5752" ################################################################################################ # Number of times to collect data. Means total number of thread dumps. LOOP=10 # Interval in seconds between data points. INTERVAL=10 # Where to generate the threddump & top output files. WHERE_TO_GENERATE_OUTPUT_FILES="/tmp" # Setting the Java Home, by giving the path where your JDK is kept # USERS MUST SET THE JAVA_HOME before running this scripta s following: JAVA_HOME=/usr/jdk64/jdk1.8.0_60 echo "Writing CPU data log files to Directory: $WHERE_TO_GENERATE_OUTPUT_FILES" for ((i=1; i <= $LOOP; i++)) do #$JAVA_HOME/bin/jstack -l $1 >> jstack_threaddump.out $JAVA_HOME/bin/jcmd $1 Thread.print >> $WHERE_TO_GENERATE_OUTPUT_FILES/jcmd_threaddump.out _now=$(date) echo "${_now}" >> $WHERE_TO_GENERATE_OUTPUT_FILES/top_highcpu.out top -b -n 1 -H -p $1 >> $WHERE_TO_GENERATE_OUTPUT_FILES/top_highcpu.out echo "Collected 'top' output and Thread Dump #" $i if [ $i -lt $LOOP ]; then echo "Sleeping for $INTERVAL seconds." sleep $INTERVAL fi done - Get the file "jcmd_threaddump.out" and "top_highcpu.out" for analysis. . How to analyze the Thread dump Data? --------------------------------------------------------- You may have a look at one of my old blog article which explains "High CPU Utilization Finding Cause?" http://middlewaremagic.com/weblogic/?p=4884 . . Common Errors with "jcmd" utility. --------------------------------------------------------- While running the JCMD we might see the below mentioned error. Here the "5752" is the NameNode PID. [root@c6401 keys]# /usr/jdk64/jdk1.8.0_60/bin/jcmd 5752 help 5752: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106) at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63) at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208) at sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:147) at sun.tools.jcmd.JCmd.main(JCmd.java:131) This error occurred because, JCMD utility allows to connect only to the JVM process that we own the process. In this case we see that the "NameNode" process is being owned by the "hdfs" user where as in the above command we are trying to connect to the NameNode process Via "jcmd" utility as "root" user. The root user here does not own the process, Hence we see the error. - "hdfs" user owned process # su -l hdfs -c "/usr/jdk64/jdk1.8.0_60/bin/jcmd -l" 5752 org.apache.hadoop.hdfs.server.namenode.NameNode 5546 org.apache.hadoop.hdfs.tools.DFSZKFailoverController 5340 org.apache.hadoop.hdfs.server.datanode.DataNode 4991 org.apache.hadoop.hdfs.qjournal.server.JournalNode . - "root" user owned process [root@c6401 keys]# /usr/jdk64/jdk1.8.0_60/bin/jcmd -l 1893 com.hortonworks.support.tools.server.SupportToolServer 6470 com.hortonworks.smartsense.activity.ActivityAnalyzerFacade 16774 org.apache.ambari.server.controller.AmbariServer 29100 sun.tools.jcmd.JCmd -l 6687 org.apache.zeppelin.server.ZeppelinServer More information about this utility can be found at: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html . . ... View more

As we know that the logs are important however many times we see that the logs consumes a lots of disk space. So if we want to use the log4j provided approach to control the logging behavior in a much efficient way, then in that case we can take advantage of the "apache-log4j-extra" package. More information about the extra packages can be found in : https://logging.apache.org/log4j/extras/ In this article we will see how to use the log compression (zip) feature of log4j to compress the NameNode logs on a daily basis automatically. In order to achieve the same we will need the following Steps: Step-1). As the extras features are not shipped with the default log4j implementation hence the users will need to download the "Apache Extras™ for Apache log4j" (Like: apache-log4j-extras) : https://logging.apache.org/log4j/extras/download.html Example: For example download the jar "apache-log4j-extras-1.2.17.jar" and place it inside the Hadoop library location. /usr/hdp/2.4.2.0-258/hadoop/lib/apache-log4j-extras-1.2.17.jar Step-2). Create a log4j appender like "ZIPRFA" using class "org.apache.log4j.rolling.RollingFileAppender" where we will define the "rollingPolicy". For more information about various Rolling Policies users can refer to : https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/ - Login to Ambari and then in the HDFS advanced configuration, Add the following Appender in the "Advanced hdfs-log4j" some where at the end. #### New Appender to Zip the Log Files Based on Daily Rotation log4j.appender.ZIPRFA=org.apache.log4j.rolling.RollingFileAppender log4j.appender.ZIPRFA.File=${hadoop.log.dir}/${hadoop.log.file} log4j.appender.ZIPRFA.layout=org.apache.log4j.PatternLayout log4j.appender.ZIPRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n log4j.appender.ZIPRFA.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy log4j.appender.ZIPRFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file} log4j.appender.ZIPRFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-.%d{yyyyMMdd}.log.gz Step-3). Also we will need to make sure that the NameNode should use the above mentioned Appender then we will need to add the "HADOOP_NAMENODE_OPTS" to include the "-Dhadoop.root.logger=INFO,ZIPRFA" something like following: export HADOOP_NAMENODE_OPTS="${SHARED_HADOOP_NAMENODE_OPTS} -XX:OnOutOfMemoryError=\"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node\" -Dorg.mortbay.jetty.Request.maxFormContentSize=-1 ${HADOOP_NAMENODE_OPTS} -Dhadoop.root.logger=INFO,ZIPRFA" Step-4). Now Restart the NameNode and double check sure that the "-Dhadoop.root.logger=INFO,ZIPRFA" property is added properly somewhere at the end. We can confirm the same using the "ps -ef | grep NameNode" output hdfs 27497 1 3 07:07 ? 00:01:27 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_namenode -Xmx1024m -Dhdp.version=2.4.2.0-258 -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhdp.version= -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.4.2.0-258/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhdp.version=2.4.2.0-258 -Dhadoop.log.dir=/var/log/hadoop/hdfs -Dhadoop.log.file=hadoop-hdfs-namenode-jss1.openstacklocal.log -Dhadoop.home.dir=/usr/hdp/2.4.2.0-258/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -Xloggc:/var/log/hadoop/hdfs/gc.log-201608060706 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms1024m -Xmx1024m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -Dorg.mortbay.jetty.Request.maxFormContentSize=-1 -server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -Xloggc:/var/log/hadoop/hdfs/gc.log-201608060706 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms1024m -Xmx1024m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -Dorg.mortbay.jetty.Request.maxFormContentSize=-1 -server -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hadoop/hdfs/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -Xloggc:/var/log/hadoop/hdfs/gc.log-201608060706 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -Xms1024m -Xmx1024m -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT -XX:OnOutOfMemoryError="/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node" -Dorg.mortbay.jetty.Request.maxFormContentSize=-1 -Dhadoop.root.logger=INFO,ZIPRFA -Dhadoop.root.logger=INFO,ZIPRFA -Dhadoop.root.logger=INFO,ZIPRFA -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode Step-5). Now as soon as the date changes we should be able to see that the old NameNode log file got zipped as following: [root@jayhost hdfs]# /var/log/hadoop/hdfs [root@jayhost hdfs]# ls -lart *.gz -rw-r--r--. 1 hdfs hadoop 32453 Aug 5 06:32 hadoop-hdfs-namenode-jayhost.openstacklocal.log-.20160804.log.gz ... View more

If we are working on some cloud environment like "OpenStack" then we might encounter an issue in which after installing the ambari agents we will see that some of the Quick Links in the ambari console is not having the expected hostname. For example if we navigate to Ambari --> HDFS --> "Resource Links" then we may see the incorrect hostnames. Actually those are detected as "public_host_name" by the ambari "/usr/lib/python2.6/site-packages/ambari_agent/hostname.py" script during every restart. So in case of OpenStack it is possible that the "public_host_name" might be different from the "host_name" (`hostname -f`) Here if we see that the Resource Link will show the links with the hostname "ambari1.example.com" (which is `hostname -f` but it is showing "ambari1.novalocal" hence the links are not working. In such case we might find that the ambari "hosts" table in the ambari database has different values for the "public_host_name" and the "host_name" select public_host_name, host_name from hosts ---------------------------------------------- public_host_name | host_name ---------------------------------------------- ambari3.novalocal | ambari3.example.com ambari2.novalocal | ambari2.example.com ambari5.novalocal | ambari5.example.com ambari1.novalocal | ambari1.example.com ambari4.novalocal | ambari4.example.com In this case we can refer to the following document to fix the hostname (or define a custom hostname) https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_ambari_reference_guide/content/_how_to_customize_the_name_of_a_host.html . . ** Permanently fix the public hostname: (Recommended) 1. Create a file with name : "/var/lib/ambari-agent/public_hostname.sh" then in that file add the following line: #!/bin/sh echo `hostname -f` 2. Make sure that the file "/var/lib/ambari-agent/public_hostname.sh" has proper execute permission. Example: chmod 755 "/var/lib/ambari-agent/public_hostname.sh" 3. On every ambari-agent host edit the file "/etc/ambari-agent/conf/ambari-agent.ini" and in the [agent] section add the following line: ## Added following to customize the public hostname public_hostname_script=/var/lib/ambari-agent/public_hostname.sh NOTE: Users can also use the property "hostname_script" to customize the internal hostname. 3. Make sure that the changes are pushed to all the hosts present in the ambari cluster. 4. Now restart the agents. ambari-agent restart . . . Temporarily fix the public hostname using Ambari APIs: Using Ambari APIs we can temporarily fix the host name (which will not survive the ambari agent restart). 1. List all the hostnames : https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts?fields=Hosts/public_host_name Example output: items: [{ href: "https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts/ambari1.example.com", Hosts: { cluster_name: "ClusterDemo", host_name: "ambari1.example.com", public_host_name: "ambari1.novalocal" } }, Now lets change the public hostname "ambari1.novalocal" to "ambari1.example.com" : - (A). Get the output of the URL: https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts/ambari1.example.com - (B). Now from the output remove the very first line containing "href:" line - (C). In the output file change the public_host_name to the desired name. and then run the following command: curl --insecure -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"Hosts" : {"public_host_name" : "ambari1.example.com"}}' https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts/ambari1.example.com . How ambari agent finds the hostname ? Users can refer to the "/usr/lib/python2.6/site-packages/ambari_agent/hostname.py" script to know how ambari agents gets the "public_host_name" and "host_name". Here we will see a simple python script "findHostname.py" to understand the logic which agent is using to determine the internal "host_name" and the "public_host_name" import socket import subprocess import urllib2 cached_hostname = None cached_public_hostname = None try: print "\n####### finding 'hostname' using 'urllib2.urlopen' approach ########" handle = urllib2.urlopen('http://169.254.169.254/latest/meta-data/public-hostname', '', 2) str = handle.read() print 'str = ' + str handle.close() cached_public_hostname = str.lower() print '[Try] cached_public_hostname = ' + cached_public_hostname print("Read public hostname '" + cached_public_hostname + "' from http://169.254.169.254/latest/meta-data/public-hostname") print "\n\n####### finding 'hostname' using 'socket.getfqdn()' approach ########" cached_hostname = socket.getfqdn() print '[Try] cached_hostname = ' + cached_hostname except: cached_public_hostname = socket.getfqdn().lower() print '[except] cached_public_hostname = ' + cached_public_hostname print("Read public hostname '" + cached_public_hostname + "' using socket.getfqdn()") After running the above python script we may see the following output: # python findHostname.py ####### finding 'hostname' using 'urllib2.urlopen' approach ######## str = ambari2.novalocal [Try] cached_public_hostname = ambari2.novalocal Read public hostname 'ambari2.novalocal' from http://169.254.169.254/latest/meta-data/public-hostname ####### finding 'hostname' using 'socket.getfqdn()' approach ######## [Try] cached_hostname = ambari2.example.com ... View more