The purpose of this document is how to leverage “R” to predict HDFS growth assuming we have access to the latest fsimage of a given cluster. This way we can forecast how much capacity would need to be added to the cluster ahead of time. In case of on-prem clusters, ordering H/W can be a lengthy process and for some organizations, it can take months till they can actually add capacity to their clusters. Giving teams managing Hadoop clusters an insight on HDFS growth rate can help with placing H/W orders ahead of time
A prereq. for this script, you would need access to a recent fsimage file, R and a machine with hadoop binaries installed on it to run OIV.
Example: hdfs oiv -i fsimage_xxxxx -p Delimited > fsimage.out
Loading required libraries
library(dplyr) library(anytime) library(lubridate) library(prophet) library(rmarkdown) --> only needed if you are using RStudio and planning on exporting the script to a PDF or Word format
Data Filtering & manipulation
The only three columns we need are Filesize, Replication and ModificationTime. Also by choozing FileSize > 0, removes all directories in the file from our calculation as well as zero size files.
files <- fsimage %>% filter(FileSize > 0) %>% select (FileSize, Replication, ModificationTime)
Here I am calculating actual file sizes on disk (file size * replication factor) and converting size to GB (not really required). Then filtering on which day I want to start my calculation from.
files_used <- mutate(files, RawSize = (((FileSize/1024)/1024)/1024) * Replication, MTime = anytime(ModificationTime)) %>% select (RawSize, MTime) %>% group_by(day=floor_date(MTime, "day")) %>% filter (day > '2017-04-01') %>% summarize(RawSize = sum(RawSize))
# Using Prophet library for prediction. # Change column names for prophet to work names(files_used)[1] <- "ds" names(files_used)[2] <- "y"
Based on all our data points, below graph shows the predicted usage over the next year.
m <- prophet(files_used, yearly.seasonality=TRUE) future <- make_future_dataframe(m, periods = 365) forecast <- predict (m, future) ## Plot Forecast plot(m, forecast) prophet_plot_components(m, forecast)
Percentage of HDFS files accessed in the last 30 days
files_used_access <- fsimage %>% filter(FileSize > 0) %>% select (FileSize, Path, ModificationTime) x = nrow(files_used_access) files_access <- mutate(files_used_access, MTime = anytime(ModificationTime)) %>% select (Path, MTime) %>% filter(MTime > today() - days(30)) y = nrow(files_access) z = (y / x) * 100 Percentage of files accessed in the last 30 days `r ceiling(z)`%
As an enhancement to this script would be using SparkR from a zeppelin notebook
Note: I am not R expert, I am just R curious 🙂 .
