YARN issues critical error "ATSv2 HBase Application" (fresh installation of Ambari 2.7.1, HDP 3.0.1)

[#1]

After fresh installation I got the following critical error alert in YARN:

Title: ATSv2 HBase Application

Response:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/alerts/alert_ats_hbase.py", line 183, in execute
    ats_hbase_app_info = make_valid_json(output)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/alerts/alert_ats_hbase.py", line 226, in make_valid_json
    raise Fail("Couldn't validate the received output for JSON parsing.")
Fail: Couldn't validate the received output for JSON parsing.

[#2]

@Zholaman Kubaliyev

Can you please check if the following "status" command is giving you proper output?

# su - yarn-ats -c "/usr/hdp/current/hadoop-yarn-client/bin/yarn app -status ats-hbase"

If "ats-hbase" service is down the you might see the error. Because the failing alert will also run the same query and then will attempt to parse the JSON output so if the status is not returning fine they you should see the alert.

Please check if your "ats-hbase" service is running?

Reference:
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/data-operating-system/content/options_to_re...

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/data-operating-system/content/enable_system...

[#3]

@Zholaman Kubaliyev

Ambair also allows to disable alerts .. so if it is your test cluster and if you just want to suppress that alert then just click on that alert link in ambari UI and "Disable" button in the alerts page.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/managing-and-monitoring-ambari/content/amb_...

[#4]

@Jay Kumar SenSharma

Hi Jay Kumar!

The (# su - yarn-ats -c "/usr/hdp/current/hadoop-yarn-client/bin/yarn app -status ats-hbase") output :

[root@hadoop anaconda3]# su - yarn-ats -c "/usr/hdp/current/hadoop-yarn-client/bin/yarn app -status ats-hbase"
18/10/25 13:18:42 INFO client.RMProxy: Connecting to ResourceManager at hadoop.test.com/xxx.xxx.xxx.xxx:8050
18/10/25 13:18:42 INFO client.AHSProxy: Connecting to Application History server at hadoop.test.com/xxx.xxx.xxx.xxx:10200
18/10/25 13:18:42 INFO client.RMProxy: Connecting to ResourceManager at hadoop.test.com/xxx.xxx.xxx.xxx:8050
18/10/25 13:18:42 INFO client.AHSProxy: Connecting to Application History server at hadoop.test.com/xxx.xxx.xxx.xxx:10200
18/10/25 13:18:42 INFO util.log: Logging initialized @1294ms
ats-hbase Failed : HTTP error code : 500
[root@hadoop anaconda3]#

As I understand the service "ats-hbase" is down. To start this service I should follow the instructions given in - https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/data-operating-system/content/enable_system...

Am I right ? How important this service (ats-hbase) for production ?

Thank you!

[#5]

• Move ats-hbase from the default queue to the yarn-system queue.
  yarn application -changeQueue yarn-system -appId <app-id>

  Here, <app-id> is the ID of the ats-hbase service.
  
  I stopped at this step and the alert notification still showing although I have restart the yarn and refresh yarn capacity scheduler, how I can find the app-id?
  I have try " yarn app -status ats-hbase " but it returns
  
  

  20/07/30 01:42:49 INFO client.RMProxy: Connecting to ResourceManager at hdpdev02.bps.go.id/10.0.45.112:8050
  20/07/30 01:42:49 INFO client.AHSProxy: Connecting to Application History server at hdpdev02.bps.go.id/10.0.45.112:10200
  20/07/30 01:42:50 INFO client.RMProxy: Connecting to ResourceManager at hdpdev02.bps.go.id/10.0.45.112:8050
  20/07/30 01:42:50 INFO client.AHSProxy: Connecting to Application History server at hdpdev02.bps.go.id/10.0.45.112:10200
  ats-hbase Failed : HTTP error code : 500
  
  NB: The cluster is a kerberized cluster

[#6]

I have the same problem，

Can you help me?

[#7]

Hi, could you post here you alert message ?

[#8]

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/alerts/alert_ats_hbase.py", line 183, in execute
    ats_hbase_app_info = make_valid_json(output)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/alerts/alert_ats_hbase.py", line 226, in make_valid_json
    raise Fail("Couldn't validate the received output for JSON parsing.")
 Fail: Couldn't validate the received output for JSON parsing.

[#9]

what version of OS, HDP do you use ?

[#10]

ambari2.7 、HDP3.1