Support Questions
Find answers, ask questions, and share your expertise

Tiered storage at a node level?

Tiered storage at a node level?

New Contributor

I'm building a MPP demonstrator lab and want to know if its possible to do data tiering (heterogeneous storage) on each node. For my lab I want to build 4-8 compute nodes with:

2x Xeon Quadcore CPU 64-256GB RAM 4x SATA/SAS 1TB SSD 1x 1.6TB Intel Optane PCIe super fast SSD. (Something like a FusionIO drive of yore) 4x 6TB SATA/SAS Spinning disk?

Because I don't want 4(more) cold storage nodes, can cold storage be on the same node as the in memory and the near in memory speed disks? I know long term, we're going to need in memory, fast storage, and to efficiently move less used data to a slower tier. To test our code and methodologies I need to set up that storage somehow. Again, I want to avoid building a bunch of cold storage nodes in my lab cluster just to test theory.

Can I set up tiered storage on the nodes themselves?

1 REPLY 1
Highlighted

Re: Tiered storage at a node level?

Contributor

Please follow this instruction to deploy HDP on heterogeneous disks for Hot, Warm and Cold tiering in a single node.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hdfs-administration/content/configuring_...