Support Questions
Find answers, ask questions, and share your expertise

HDFS In-Memory Tier

Looking to understand if there are any limits to using Host Groups to define specific machines in a cluster to serve as HDFS in Memory Tier. Cluster has machines that have RAM and SSD, in addition to machines that have RAM and SATA Drives.

Goal would be to use the in memory tier as storage for Spark data processing pipelines.

1 ACCEPTED SOLUTION

Super Guru

@michael perez To add to Emil, to isolate CPU and Ram resources enable CPU scheduling. More info here.

CPU scheduling represents one aspect of YARN resource management capabilities that includes CGroups, node labels, archival storage, and memory as storage. CGroups should be used with CPU scheduling to constrain and manage CPU processes.

View solution in original post

1 REPLY 1

Super Guru

@michael perez To add to Emil, to isolate CPU and Ram resources enable CPU scheduling. More info here.

CPU scheduling represents one aspect of YARN resource management capabilities that includes CGroups, node labels, archival storage, and memory as storage. CGroups should be used with CPU scheduling to constrain and manage CPU processes.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.