Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why am I getting disk usage alert?

avatar
Contributor

I created 7 AWS EC2 instances (1 for ambari server and 6 for ambari agents) and installed HDP 2.4 using ambari. Now, when I open the ambari dashboard, it shows me critical alerts related to disk usage. It shows that the hosts have less than 6GB disk size. For example, the first host in the attached image shows "4.56 GB/5.63 (88.1% used)". How can I solve this problem?

5914-screen-shot-2016-07-20-at-20926-pm.png

1 ACCEPTED SOLUTION

avatar
Contributor

Hi @Fish Berh

This explains why the alert is generated. You need to increase the space to stop these alerts.

View solution in original post

14 REPLIES 14

avatar
Contributor

The metrics for the first host in the attached image is shown below.

5915-screen-shot-2016-07-20-at-22746-pm.png

avatar
Expert Contributor

Can you get the o/p of command :

$hdfs dfsadmin -report

This will get us the actual state of disks. Also, we can then figure out if the alerts are stale.

- Also, if we are running out of disk, if would be good idea to increase your disk space.

avatar
Contributor

I am getting the report below in the hosts

5918-screen-shot-2016-07-20-at-30557-pm.png

avatar
Expert Contributor

@Fish Berh

Missed on mentioning 'su hdfs', to get the full report:

Can you once again do:

$su hdfs

$hdfs dfsadmin -report

avatar
Contributor

When I use the command $su hdfs, it asks me for password. However, I have not set any password.

avatar
Expert Contributor

@Fish Berh Did you "su hdfs" as root/admin user ?

avatar
Contributor

I changed to root using "sudo su -", then ran the command "su hdfs", but I get an error message that user "hdfs" is not known.

5961-screen-shot-2016-07-21-at-103503-am.png

avatar
Contributor

Hi @Fish Berh

The default alert configuration for disk_usage is set like the one below:

I believe the minimum free space is kicking in here.

 "AlertDefinition" : {

    "cluster_name" : "xxxxxxxxxx",

    "component_name" : "AMBARI_AGENT",

    "description" : "This host-level alert is triggered if the amount of disk space used goes above specific thresholds. The default threshold values are 50% for WARNING and 80% for CRITICAL",

    "enabled" : true,

    "id" : 48,

    "ignore_host" : false,

    "interval" : 1,

    "label" : "Host Disk Usage",

    "name" : "ambari_agent_disk_usage",

    "scope" : "HOST",

    "service_name" : "AMBARI",

    "source" : {

      "parameters" : [

        {

          "name" : "minimum.free.space",

          "description" : "The overall amount of free disk space left before an alert is triggered.",

          "threshold" : "WARNING",

          "units" : "bytes",

          "display_name" : "Minimum Free Space",

          "type" : "NUMERIC",

          "value" : "5.0E9"

        },

        {

          "name" : "percent.used.space.warning.threshold",

          "description" : "The percent of disk space consumed before a warning is triggered.",

          "threshold" : "WARNING",

          "units" : "%",

          "display_name" : "Warning",

          "type" : "PERCENT",

          "value" : “0.5”

        },

        {

          "name" : "percent.free.space.critical.threshold",

          "description" : "The percent of disk space consumed before a critical alert is triggered.",

          "threshold" : "CRITICAL",

          "units" : "%",

          "display_name" : "Critical",

          "type" : "PERCENT",

          "value" : “0.8”

        }

      ],

      "path" : "alert_disk_space.py",

      "type" : "SCRIPT"

    }

  }

}

Hope this helps.

avatar
Contributor

But it is saying it has used 88.1% of 5.63 GB. But when I check the disk size in the terminals, I the report shown in the attachment

5919-screen-shot-2016-07-20-at-31201-pm.png