Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why did the hdfs disk utilization skyrocketed from a few hundred GB to over 7TB in a matter of hours while ingesting data into HBASE by openTSDB?

avatar
Contributor

Background :

Customer has an 8 Node cluster on AWS with ephemeral storage, 5 of which are Hbase.

OpenTSDB and Grafana were installed on the cluster as well.

Customer was ingesting time series data with OpenTSDB, at a rate of ~50k records/second.

Symptom:

In a span of a couple of hours, the disk utilization of hdfs skyrocketed from a few hundred GB to over 6TB all of it in HBASE / openTSDB.

In attempting to troubleshoot – turning off all data ingest and stopping openTSDBand just running HBase caused the disk utilization to continue to grow unabated and out of control dozens of GB per minute, even when openTSDB was completely shut down.

1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Rising Star