HDFS In-Memory Tier

[#1]

Looking to understand if there are any limits to using Host Groups to define specific machines in a cluster to serve as HDFS in Memory Tier. Cluster has machines that have RAM and SSD, in addition to machines that have RAM and SATA Drives.

Goal would be to use the in memory tier as storage for Spark data processing pipelines.

[#2 · Accepted solution]

@michael perez To add to Emil, to isolate CPU and Ram resources enable CPU scheduling. More info here.

CPU scheduling represents one aspect of YARN resource management capabilities that includes CGroups, node labels, archival storage, and memory as storage. CGroups should be used with CPU scheduling to constrain and manage CPU processes.