Reply
New Contributor
Posts: 3
Registered: ‎08-01-2017

HDFS /tmp filesystem is filling up rapidly and expected to cause outage

In our Hadoop cluster (Cloudera distribution), we recently found that Hive Job started by a user create a 160 TB of files in '/tmp' location and it almost consumed remaining HDFS space and about to cause an outage. Later we troubleshoot and kill the particular job as we are unable to reach the user who started this job.

 

So now my question is - how could we able to set an alert for '/tmp' location if anyone created huge files or can we restrict the users using HDFS '/tmp' space?

 

Please share if you have any other suggestions.

Posts: 352
Topics: 11
Kudos: 54
Solutions: 30
Registered: ‎09-02-2016

Re: HDFS /tmp filesystem is filling up rapidly and expected to cause outage

@Srini4u

 

There are different options

 

1. If you have linux monitoring tools like Nagios, New Relic, ganglia, etc. You can set-up an alert for a file system (/tmp will be mounted on a file system) and trigger a mail if any file system running out of space

2. you can create a shell script to triger a mail based on the space availability and schedule via cron

Posts: 614
Topics: 3
Kudos: 91
Solutions: 59
Registered: ‎08-16-2016

Re: HDFS /tmp filesystem is filling up rapidly and expected to cause outage

I don't know HDFS quotas well enough but should fit the bill.

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html

In CM you can configure alerts to notify you when disk and HDFS is nearing capacity.
New Contributor
Posts: 3
Registered: ‎08-01-2017

Re: HDFS /tmp filesystem is filling up rapidly and expected to cause outage

@saranvisa, Thanks for your reply. 

 

I am talking about HDFS temp file system, not on the host machine temp file system. 

 

Please advise. 

New Contributor
Posts: 3
Registered: ‎08-01-2017

Re: HDFS /tmp filesystem is filling up rapidly and expected to cause outage

@mbigelow Thanks for your reply too. 

 

"hdfs dfsadmin -setSpaceQuota" wont work on HDFS temp location. Need to find out some other alternative. 

 

Highlighted
Posts: 614
Topics: 3
Kudos: 91
Solutions: 59
Registered: ‎08-16-2016

Re: HDFS /tmp filesystem is filling up rapidly and expected to cause outage

Why won't it work?  Have you tried /tmp and /tmp/hive/<user.name>.

 

The alternative if quotas can't be applied to /tmp or its subdirs is to set alerts for HDFS capacity or disk space on the disks hosting the DFS directories.

Announcements