Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Should I scale Hadoop processing power alone(YarnClient + NodeManager + AppMaster + Container) but not storage (DataNode Blocks) ?

Should I scale Hadoop processing power alone(YarnClient + NodeManager + AppMaster + Container) but not storage (DataNode Blocks) ?

New Contributor

We have fixed cluster nodes(expanded manually if required) with adequate storage whenever job request came from client, system should check for available memory and then spin a host once the job completed it will delete the host from the cluster. The idea is to support true multitenacy since client don’t want to wait in a queue due to sort of processing power. Fair, FIFO & Capacity schedulers doesn’t help to solve this problem also we don’t want to unnecessarily reserve resources per tenant since we don’t know when the job will be placed.

2 REPLIES 2

Re: Should I scale Hadoop processing power alone(YarnClient + NodeManager + AppMaster + Container) but not storage (DataNode Blocks) ?

Super Guru

@Gopi Thulasidoss If I understand your question correctly you are essentially trying to have a elastic cluster functionality within a static cluster. Theoretically it can be done but not easy. that is why tool like cloudbreak are neat as they allow you to scale # of node based on workload. Things you will have to consider when you add and remove nodes is hdfs rebalancing. All in all I would say it is possible and you will have to manually configure via puppet/chef scripts and integrate all the management via ambari API

Re: Should I scale Hadoop processing power alone(YarnClient + NodeManager + AppMaster + Container) but not storage (DataNode Blocks) ?

New Contributor

@Sunile Manjee Yes your are correct, I am trying to have a elastic cluster but my idea is to spin a Node Manager(MR TaskTracker) alone since my static cluster has enough disk storage in that case should I need to rebalance hdfs ?

Don't have an account?
Coming from Hortonworks? Activate your account here