Resources

SYMPTOM:hive CLI is in hung state for so long, an impatient user did CRTL+C to exit of it and complain about the hive CLI slowness.ROOT CAUSE: user is running hive CLI in Kerberos enabled security, we asked them to enable debug logging at the console using hive --hiveconf hive.root.logger=debug,console and saw the following GSS exception due to ticket expiration.WARN hive.metastore: Failed to connect to the MetaStore Server...org.apache.thrift.transport.TTransportException: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Na

ROOT CAUSE: YARN UI shows total memory vs used memory wrong when there are reserved resources. The memory total shown when there is no reserved resources will be the correct one. We can also compare that with sum of all NodeManager memory resource. This won't have any impact on YARN scheduler logic. This behavior has been reported in bug in YARN UI and got fixed by https://issues.apache.org/jira/browse/YARN-3432 and https://issues.apache.org/jira/browse/YARN-3243 REFERENCES: https://issues.apache.org/jira/browse/YARN-3432https://issues.apache.org/jira/browse/YARN-3243

It would sometimes be useful to adjust the log level when executing hbase client commands from command line without having to restart components. This article explains how that can be achieved:You can set DEBUG level for any client hbase command by exporting the following environment variable before it is executed:HBASE_ROOT_LOGGER=hbase.root.logger=DEBUG,consoleFor example:# export HBASE_ROOT_LOGGER=hbase.root.logger=DEBUG,console# hbase hbck2016-12-24 15:18:03,240 DEBUG [main] util.Shell: setsid exited with exit code 02016-12-24 15:18:03,363 DEBUG [main] lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, about=, sampleName=Ops, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)2016-12-24 15:18:03,373 DEBUG [main] lib.MutableMetricsFactory: field org.apache.hadoop.m

It would sometimes be useful to adjust the log level when executing hive commands from command line without having to restart components. This article explains how that can be achieved:You can provide the log level on the command line to hive as followshive -hiveconf hive.root.logger=DEBUG,console -e "show tables ;"For example:# hive -hiveconf hive.root.logger=DEBUG,console -e "show tables ;" |more16/12/24 15:10:23 DEBUG util.VersionInfo: version: 2.7.3.2.5.3.0-3716/12/24 15:10:27 [main]: DEBUG common.LogUtils: Using hive-site.xml found on CLASSPATH at /etc/hive/2.5.3.0-37/0/hive-site.xml16/12/24 15:10:27 [main]: DEBUG session.SessionState: SessionState user: nullLogging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties16/12/24 15:10:27 [main]: INFO SessionState: Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties16/12/24 15:10:27 [main]: DEBUG parse.VariableSubstitution: Substitution is on: hive16/12/24 15:10:

It's sometimes useful to adjust the log level when executing commands from the client to get more info about what is happening. This article explains how that can be achieved: You can set DEBUG level for any client command by exporting the following environment variable before it is executed: HADOOP_ROOT_LOGGER=hadoop.root.logger=DEBUG,console Now try executing a client command and watch the stream of DEBUG info come to your terminal. For example: # hdfs dfs -ls16/12/24 15:01:34 DEBUG util.Shell: setsid exited with exit code 016/12/24 15:01:34 DEBUG conf.Configuration: parsing URL jar:file:/usr/hdp/2.5.3.0-37/hadoop/hadoop-common-2.7.3.2.5.3.0-37.jar!/core-default.xml16/12/24 15:01:34 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@149494d816/12/24 15:01:34 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/2.5.3.0-37/0/core-site.xml16/12/24 15:01:34 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream

SYMPTOMS: /tmp filling up causes multiple services to stop functioning. ROOT CAUSE: The issue happens due to internal Smartsense bug ST-2551.SOLUTION: Upgrade to Smartsense 1.3.1 WORKAROUND: To workaround this issue we need to manually modify two files related to Smartsense, so that the tmp files will not be generated in /tmp directory anymore 1. File : /usr/hdp/share/hst/hst-agent/lib/hst_agent/anonymize.py Change from : ANONYMIZATION_JAVA_COMMAND = "{0}" + os.sep + "bin" + os.sep + "java" +\ " -Dlog.file.name="+ ANONYMIZATION_LOG_FILE_NAME +\ " -cp {1} {2} {3}" Change to : ANONYMIZATION_JAVA_COMMAND = "{0}" + os.sep + "bin" + os.sep + "java" +\ " -Djava.io.tmpdir=/grid/02/smartsense/hst-agent/data/tmp" +\ " -Dlog.file.name="+ ANONYMIZATION_LOG_FILE_NAME +\ " -cp {1} {2} {3}" Make sure the tmp dir value is same as this property agent.tmp_dir in hst-agent-conf.2. File : /usr/sbin/hst-server.py Change from : SERVER_START_CMD = "{0}" + os.sep + "bin" + os.sep +\ "java -server -XX:NewRat

ENVIRONMENT: All Ambari versions prior to 2.4.xSYMPTOMS: Intermittent loss of heartbeat to cluster nodes, freeze of ambari-agent service, intermittent issues in Ambari alerts and service status updates in Ambari dashboard.Ambari-agent logs:- INFO 2016-08-21 19:10:20,080 Heartbeat.py:78 - Building Heartbeat: {responseId = 139566, timestamp = 1471821020080, commandsInProgress = False, componentsMapped = True}ERROR 2016-08-21 19:10:20,102 HostInfo.py:228 - Checking java processes failedTraceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/HostInfo.py", line 211, in javaProcs cmd = open(os.path.join('/proc', pid, 'cmdline'), 'rb').read()IOError: [Errno 2] No such file or directory: '/proc/24270/cmdline'Top command output: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP TIME DATA COMMAND 10098 root 20 0 54.4g 53g 4540 S 54.5 14.0 18000:11 224 300,00 54g /usr/bin/python2 /usr/lib/python2.6/site-packages/ambari_agent/main.py start --expected-hostname=12

ENVIRONMENT: HDP 2.3.2 , Ambari 2.2.0,JDK 1.7.0_67-b01,Kernel: 3.13.0-48-generic ERRORS: Last few lines in the NM log before it hit SIGSEGV shows that there was Container Localizer running for a new container: 2016-10-20 01:29:05,810 INFO localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(711)) - Created localizer for container_e14_1475595980406_28807_01_000021 [...] 2016-10-20 01:29:08,308 INFO localizer.LocalizedResource (LocalizedResource.java:handle(203)) - Resource hdfs://user/tmp/hive/xxx/5b0f04c6-ba2d-47dc-85c2-88179a1db407/hive_2016-10-20_01-28-15_091_3309851709548218363-3928/-mr-10007/df6632b4-ec58-4cdf-8ffb-c81460abc266/reduce.xml(->/hadoop/yarn/local/usercache/xxx/filecache/150663/reduce.xml) transitioned from DOWNLOADING to LOCALIZED - The exception says: Current thread (0x00007f2c66cc7000): JavaThread "ContainerLocalizer Downloader" [_thread_in_Java, id=14260, stack(0x00007f2c740a3000,0x00007f2c741a4000)] siginfo:si_signo=SIGSEGV: si_errno=0

ENVIRONMENT: HDP 2.5.0 , Ambari 2.4.1 ERRORS:- Logs from Ambari server:- resource_management.core.exceptions.ExecutionFailed: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/2.5.3.0-37/hadoop/sbin/hadoop-daemon.sh --config /usr/hdp/2.5.3.0-37/hadoop/conf start namenode -rollingUpgrade started'' returned 1. -bash: line 0: ulimit: core file size: cannot modify limit: Operation not permitted starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-llab90hdpc2m3.out ROOT CAUSE:- Not yet known,reported as Internal bug. (BUG-70647) WORKAROUND:- Add the following entries to /etc/security/limits.conf to complete the upgrade. soft core unlimitedhard core unlimited OR as root user, run the following command on Ambari server host. ulimit -c unlimited Please note that its not a recommended setting for any of the HDP components and just a workaround to complete the upgrade. Please revert the setting afterwards.In case unsure of the execution/i

PROBLEM: Ambari service check for Solr fails when the active namenode is nn2. From the std-err, you will see log belowERROR: stderr: /var/lib/ambari-agent/data/errors-803.txt Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/SOLR/5.5.2.2.5/package/scripts/service_check.py", line 48, in <module> ServiceCheck().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/SOLR/5.5.2.2.5/package/scripts/service_check.py", line 43, in service_check user=params.solr_config_user File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_actio

SYMPTOM: Ranger is installed and managed using Ambari. Services failed to start and giving below errorERROR:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 185, in <module>HiveServer().execute()File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in executemethod(env)File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 85, in startsetup_ranger_hive(rolling_upgrade=rolling_restart)File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive.py", line 50, in setup_ranger_hivehdp_version_override = hdp_version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/setup_ranger_plugin_xml.py", line 82, in setup_ranger_pluginpolicy_user)File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/ranger_functions.py", lin

SYMPTOM: Ambari Hive View does not reconnect when its connection to Metastore is interrupted.ERROR: 03 May 2016 13:33:39,172 ERROR qtp-ambari-client-41951 ServiceFormattedException:96 - org.apache.ambari.view.hive.client.HiveClientException: H100 Unable to submit statement show databases like '*': org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipeorg.apache.ambari.view.hive.client.HiveClientException: H100 Unable to submit statement show databases like '*': org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipeat org.apache.ambari.view.hive.client.Connection$3.body(Connection.java:608)at org.apache.ambari.view.hive.client.Connection$3.body(Connection.java:590)at org.apache.ambari.view.hive.client.HiveCall.call(HiveCall.java:101)at org.apache.ambari.view.hive.client.Connection.execute(Connection.java:590)at org.apache.ambari.view.hive.client.Connection.executeSync(Connection.java:629)at org.apache.ambari.view.hive.clien

while investigating a performance issue with topology assignments, I figured out these high-level steps which storm uses for topologies assignments.1. for backward compatibility for the old topologies Backward ClientJarTransformerRunner starts and invoke StormShadeTransformer which transformed the jar write into /tmp/<some_random_string>.jar2. StormSubmitter start uploading topology jar to nimbus inbox using NimbusCliento.a.s.StormSubmitter - Uploading topology jar /tmp/27ed633ac9aa11e6a850fa163e19dd06.jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-b1eca4ae-d021-4e93-aaf1-986c9a5772ad.jarStart uploading file '/tmp/27ed633ac9aa11e6a850fa163e19dd06.jar' to '/hadoop/storm/nimbus/inbox/stormjar-b1eca4ae-d021-4e93-aaf1-986c9a5772ad.jar'o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /hadoop/storm/nimbus/inbox/stormjar-b1eca4ae-d021-4e93-aaf1-986c9a5772ad.jar3. nimbus client submit topology to Nimbus using thrift callo.a.s.StormSubmitter -

SYMPTOM: Script runs fine in grunt, however, it fails when executed in Hue Pig editor. Script is as below -Pig script: A = load '/tmp/baseball'; dump A; ERROR: Below are the error logs -ROOT CAUSE: The property templeton.libjars was pointing to the wrong jar files: /usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar RESOLUTION: After changing path for "templeton.libjars" to /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar the script ran successful in the Hue Pig editor.Please do check - https://community.hortonworks.com/articles/15958/templetonlibjars-property-changed-its-value-after.html

SYMPTOM:hiveserver2 logs are filled with following exceptions:2016-12-22 16:36:49,643 WARN ipc.Client (Client.java:run(685)) - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation

SYMPTOM: Hive jobs failing on Production Aggregation cluster by giving j"ava.net.UnknownHostException: Matrix-Aggr" Error. Matrix-Aggr is the nameservice for Namenode HA.ERROR: Error log is as below -Caused by: java.lang.IllegalArgumentException: java.net.UnknownHostException: Matrix-Aggr at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:312) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:665) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:601) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2619) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2653) at org.apache.had

SYMPTOM: During processes like adding a service, or upgrading, Ambari UI complains package not found for installation. From the log we can see that Ambari is searching for a repo higher version than cluster's current HDP version For example: -- Current version : Ambari Version 1.7.0 and HDP 2.2.0 -- But Ambari is searching in repo version for 2.2.8 ROOT CAUSE: It seems that /var/lib/ambari-server/resources/stacks/HDP/<VERSION>/repos/repoinfo.xml file has been updated with wrong latest version info.RESOLUTION: 1. comment out the below line in file /var/lib/ambari-server/resources/stacks/HDP/<VERSION>/repos/repoinfo.xml'<latest>http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json</latest>'2. Open Ambari database, and check content of table metainfo . For example :In case the metainfo_key "repo:/HDP/2.2/redhat6/HDP-<VERSION>:baseurl" is missing, use following command to add content to tab

Environment: HDP 2.4.3 , Ambari 2.4.0SYMPTOMS: Region server logs are as follows:-2016-10-03 15:13:55,611 INFO [main] regionserver.HRegionServer: STOPPED: Unexpected exception during initialization, aborting2016-10-03 15:13:55,649 ERROR [main] token.AuthenticationTokenSecretManager: Zookeeper initialization failedorg.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase-secure/tokenauth/keysat org.apache.zookeeper.KeeperException.create(KeeperException.java:113)at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:575)at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:554)Zookeeper logs:- 2016-10-04 15:48:45,702 - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@137] - Failed to set name based on Kerberos authe

PROBLEM:When we go to Grafana UI:Under HBase - Tables: 1. NUM FLUSHES 2. NUM WRITE REQUESTS 3. NUM READ REQUESTSUnder HBase - Users: 1. Num Get Requests 2. Num Scan Next RequestsWe just see:Problem! java.lang.Exception: Invalid number of functions specified.grafana.log show:[I] Completed X.X.X.X - "GET /ws/v1/timeline/metrics HTTP/1.1" 400 Bad Request 144 bytes in 7653us[I] Completed X.X.X.X - "GET /ws/v1/timeline/metrics HTTP/1.1" 400 Bad Request 144 bytes in 3316us[I] Completed X.X.X.X - "GET /ws/v1/timeline/metrics HTTP/1.1" 400 Bad Request 144 bytes in 1734usRESOLUTION:1. Login with Grafana admin.2. Set transform=none in panels.

SYMPTOM: Trying to add components using ambari ui but its failing. We are using RHN satellite repos to download packages.The HDP.repo and HDP_UTILS.repo were configured with "enable=0" On all servers But they always be modified with "enable=1".Below are my repos Output: [HDP-2.5]name=HDP-2.5baseurl=http://172.26.64.249/hdp/centos6/HDP-2.5.3.0/path=/enabled=0[HDP-UTILS-1.1.0.21]name=HDP-UTILS-1.1.0.21baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos6path=/enabled=0 ROOT CAUSE: As ambari uses puppet it will always revert the repo files back to orignal.RESOLUTION: Modified the respective file depending upon os, in my case it was -/var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/templates/repo_suse_rhel.j2 and replaced - enabled=1 to enabled=0 Restarted ambari server after which services were able to install using RHN Satellite repository.

Kafka Producer (Python)yum install -y python-pippip install kafka-python//kafka producer sample codevim kafka_producer.pyfrom kafka import KafkaProducerfrom kafka.errors import KafkaErrorproducer = KafkaProducer(bootstrap_servers=['rkk1.hdp.local:6667'])topic = "kafkatopic"producer.send(topic, b'test message')//run itpython kafka_consumer.py //test it[root@rkk1 ~]# /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper `hostname`:2181 --topic kafkatopic{metadata.broker.list=rkk1.hdp.local:6667,rkk2.hdp.local:6667,rkk3.hdp.local:6667, request.timeout.ms=30000, client.id=console-consumer-41051, security.protocol=PLAINTEXT}test messageKafka Producer (Scala)mkdir kafkaproducerscalacd kafkaproducerscala/mkdir -p src/main/scalacd src/main/scalavim KafkaProducerScala.scalaobject KafkaProducerScala extends App { import java.util.Properties import org.apache.kafka.clients.producer._ val props = new Properties() props.put("bootstrap.server

SYMPTOM: hivemetastore crashing with outofmemoryerror during ACID compactions. ERROR [Thread-13]: compactor.Cleaner (Cleaner.java:run(140)) - Caught an exception in the main loop of compactor cleaner, java.lang.OutOfMemoryError: Java heap space ERROR [Thread-13]: compactor.Cleaner (Cleaner.java:run(140)) - Caught an exception in the main loop of compactor cleaner, java.lang.OutOfMemoryError: Java heap space ROOT CAUSE: Enabled heap dump on outofmemory, after Analysis the heap dump we found that there are lots of entries for FileSystem$Cache$Key,FileSystem objects which was causing a memory leak WORKAROUND: set fs.hdfs.impl.disable.cache=true set fs.file.impl.disable.cache=true RESOLUTION: this has been fixed in https://issues.apache.org/jira/browse/HIVE-13151, so apply a patch to avoid this.

SYMPTOM: Upon starting the App Timeline Service after an Ambari & HDP upgrade, the following errors were thrown and the service was unable to start:ERROR: 2015-08-02 22:56:24,311 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 116 missing files; e.g.: /tmp/hadoop/yarn/timeline/leveldb-timeline-store.ldb/001052.sst org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 116 missing files; e.g.: /tmp/hadoop/yarn/timeline/leveldb-timeline-store.ldb/001052.sst ROOT CAUSE: The issue is with corrupted SST's in the app timeline db path.RESOLUTION: Navigate to /hadoop/yarn/timeline/leveldb-timeline-store.ldb From there you will see a text file named "CURRENT" Please back this file up in /tmp and then remove the file as such: cp /hadoop/yarn/timeline/leveldb-timeline-store.ldb/CURRENT /

SYMPTOM: Yarn timeline logs are growing very fast and the disk is now 100% utilized. Below are my configs set for ATS -Configs:<property> <name>yarn.timeline-service.ttl-enable</name> <value>true</value></property><property> <name>yarn.timeline-service.ttl-ms</name> <value>1339200000</value>/property><property> <name>yarn.timeline-service.leveldb-timeline-store.ttl-interval-ms</name> <value>150000</value></property> ROOT CAUSE: This config does not affect the semantic of the ATS purging process. However, it affects the concrete behavior of a level-db based storage implementation to do purging. This config decides the time interval between two purges in a level-db based ATS storage (like leveldb storage and rolling leveldb storage).Here in this case, the customer set ttl to 1339200000 ms, 1339200 seconds, or 372 hours or 15.5 days. On a normal cluster with limited disk space b

- If we want to run a service check for a particular service using Ambari APIs then one most important thins we will need to know in order to perform the service check using the APIs, that is the request payload. For every service the Payload of the request can be something in the following format: { "RequestInfo": { "context":"HDFS Service Check", "command":"HDFS_SERVICE_CHECK" }, "Requests/resource_filters":[ { "service_name":"HDFS" } ] }.If we do not know what will be the exact payload for different services then we can simply get it by using the "Browser Debugger Tools". It should show the "Form Data" section.- Now once we know the payload that we can use to POST for the service check execution to ambari then we can simply tun the command as following:Syntax:curl -u admin:admin -H "X-Requested-By: ambari" -X POST -d @/PATH/TO/hdfs_service_check_payload.txt http://AMBARI_HOST:8080/api/v1/clust