The purpose of this document is how to leverage “R” to
predict HDFS growth assuming we have access to the latest fsimage of a given
cluster. This way we can forecast how much capacity would need to be added to
the cluster ahead of time. In case of on-prem clusters, ordering H/W can be a
lengthy process and for some organizations, it can take months till they can
actually add capacity to their clusters. Giving teams managing Hadoop clusters
an insight on HDFS growth rate can help with placing H/W orders ahead of time A prereq. for this script, you would need access to a
recent fsimage file, R and a machine with hadoop binaries installed on it to
run OIV. Example: hdfs oiv -i fsimage_xxxxx -p Delimited
> fsimage.out Loading required libraries library(dplyr)
library(anytime)
library(lubridate)
library(prophet)
library(rmarkdown) --> only needed if you are using RStudio and planning on exporting the script to a PDF or Word format Data Filtering & manipulation The only three columns we need are Filesize,
Replication and ModificationTime. Also by choozing FileSize > 0, removes all
directories in the file from our calculation as well as zero size files. files <- fsimage %>%
filter(FileSize > 0) %>%
select (FileSize, Replication, ModificationTime) Here I am calculating actual file sizes on disk (file
size * replication factor) and converting size to GB (not really required).
Then filtering on which day I want to start my calculation from. files_used <- mutate(files, RawSize = (((FileSize/1024)/1024)/1024) * Replication, MTime = anytime(ModificationTime)) %>%
select (RawSize, MTime) %>%
group_by(day=floor_date(MTime, "day")) %>%
filter (day > '2017-04-01') %>%
summarize(RawSize = sum(RawSize)) # Using Prophet library for prediction.
# Change column names for prophet to work
names(files_used)[1] <- "ds"
names(files_used)[2] <- "y"
Based on all our data points, below graph shows the predicted usage over the next year. m <- prophet(files_used, yearly.seasonality=TRUE)
future <- make_future_dataframe(m, periods = 365)
forecast <- predict (m, future)
## Plot Forecast
plot(m, forecast)
prophet_plot_components(m, forecast) Percentage of HDFS files accessed in the last 30 days files_used_access <- fsimage %>%
filter(FileSize > 0) %>%
select (FileSize, Path, ModificationTime)
x = nrow(files_used_access)
files_access <- mutate(files_used_access, MTime = anytime(ModificationTime)) %>%
select (Path, MTime) %>%
filter(MTime > today() - days(30))
y = nrow(files_access)
z = (y / x) * 100
Percentage of files accessed in the last 30 days `r ceiling(z)`% As an enhancement to this script would be using SparkR from a zeppelin notebook Note: I am not R expert, I am just R curious 🙂 .
... View more
