question Re: Why am I getting disk usage alert? in Support Questions

Why am I getting disk usage alert?

fisseha_berhane — Mon, 19 Aug 2019 08:29:30 GMT

I created 7 AWS EC2 instances (1 for ambari server and 6 for ambari agents) and installed HDP 2.4 using ambari. Now, when I open the ambari dashboard, it shows me critical alerts related to disk usage. It shows that the hosts have less than 6GB disk size. For example, the first host in the attached image shows "4.56 GB/5.63 (88.1% used)". How can I solve this problem?

Re: Why am I getting disk usage alert?

fisseha_berhane — Mon, 19 Aug 2019 08:29:22 GMT

The metrics for the first host in the attached image is shown below.

Re: Why am I getting disk usage alert?

sshridhar — Thu, 21 Jul 2016 01:47:11 GMT

Can you get the o/p of command :

$hdfs dfsadmin -report

This will get us the actual state of disks. Also, we can then figure out if the alerts are stale.

- Also, if we are running out of disk, if would be good idea to increase your disk space.

Re: Why am I getting disk usage alert?

bmadala — Thu, 21 Jul 2016 01:48:14 GMT

Hi @Fish Berh

The default alert configuration for disk_usage is set like the one below:

I believe the minimum free space is kicking in here.

 "AlertDefinition" : {

    "cluster_name" : "xxxxxxxxxx",

    "component_name" : "AMBARI_AGENT",

    "description" : "This host-level alert is triggered if the amount of disk space used goes above specific thresholds. The default threshold values are 50% for WARNING and 80% for CRITICAL",

    "enabled" : true,

    "id" : 48,

    "ignore_host" : false,

    "interval" : 1,

    "label" : "Host Disk Usage",

    "name" : "ambari_agent_disk_usage",

    "scope" : "HOST",

    "service_name" : "AMBARI",

    "source" : {

      "parameters" : [

        {

          "name" : "minimum.free.space",

          "description" : "The overall amount of free disk space left before an alert is triggered.",

          "threshold" : "WARNING",

          "units" : "bytes",

          "display_name" : "Minimum Free Space",

          "type" : "NUMERIC",

          "value" : "5.0E9"

        },

        {

          "name" : "percent.used.space.warning.threshold",

          "description" : "The percent of disk space consumed before a warning is triggered.",

          "threshold" : "WARNING",

          "units" : "%",

          "display_name" : "Warning",

          "type" : "PERCENT",

          "value" : “0.5”

        },

        {

          "name" : "percent.free.space.critical.threshold",

          "description" : "The percent of disk space consumed before a critical alert is triggered.",

          "threshold" : "CRITICAL",

          "units" : "%",

          "display_name" : "Critical",

          "type" : "PERCENT",

          "value" : “0.8”

        }

      ],

      "path" : "alert_disk_space.py",

      "type" : "SCRIPT"

    }

  }

}

Hope this helps.

Re: Why am I getting disk usage alert?

fisseha_berhane — Mon, 19 Aug 2019 08:29:14 GMT

I am getting the report below in the hosts

Re: Why am I getting disk usage alert?

fisseha_berhane — Mon, 19 Aug 2019 08:29:07 GMT

But it is saying it has used 88.1% of 5.63 GB. But when I check the disk size in the terminals, I the report shown in the attachment

Re: Why am I getting disk usage alert?

sshridhar — Thu, 21 Jul 2016 02:31:53 GMT

@Fish Berh

Missed on mentioning 'su hdfs', to get the full report:

Can you once again do:

$su hdfs

$hdfs dfsadmin -report

Re: Why am I getting disk usage alert?

fisseha_berhane — Thu, 21 Jul 2016 07:16:12 GMT

When I use the command $su hdfs, it asks me for password. However, I have not set any password.

Re: Why am I getting disk usage alert?

bmadala — Thu, 21 Jul 2016 09:47:55 GMT

Hi @Fish Berh

Can you give the output of the command : df -h ?

Re: Why am I getting disk usage alert?

sshridhar — Thu, 21 Jul 2016 10:11:08 GMT

@Fish Berh Did you "su hdfs" as root/admin user ?

Re: Why am I getting disk usage alert?

fisseha_berhane — Mon, 19 Aug 2019 08:28:59 GMT

The size is 5.7G from this command

Re: Why am I getting disk usage alert?

fisseha_berhane — Mon, 19 Aug 2019 08:28:51 GMT

I changed to root using "sudo su -", then ran the command "su hdfs", but I get an error message that user "hdfs" is not known.

Re: Why am I getting disk usage alert?

bmadala — Thu, 21 Jul 2016 21:43:44 GMT

Hi @Fish Berh

This explains why the alert is generated. You need to increase the space to stop these alerts.

Re: Why am I getting disk usage alert?

fisseha_berhane — Thu, 21 Jul 2016 21:47:18 GMT

How can I do it? Each host has 107 GB disk size.

Re: Why am I getting disk usage alert?

bmadala — Fri, 22 Jul 2016 02:21:48 GMT

Hi @Fish Berh

You may want to discuss with your system admin how these disks have been created.