Hello, I'm currently running some proof of concepts in a new HDP 2.4 cluster that I am virtualizing with a SAN back-end that is all flash. I've been reading articles such as this http://www.bluedata.com/blog/2015/12/separating-hadoop-compute-and-storage/ && http://www.infostor.com/disk-arrays/hadoop-storage-options-time-to-ditch-das.html My question is, are there any concerns and design considerations when doing this with HDP? Would this essentially be having a series of nodes running purely HDFS / RegionServers for HBASE for the storage implementation, and then a series of nodes for MR and YARN for compute processing? The whole concept of splitting compute and storage is very new to me. I'm used to having all machines be identical and the DAS method that I currently use in my production environment. Also, what would the configuration files look like for this? I assume the DataNode directories parameter would hold the shared storage / SAN endpoint for HDFS? That would mean the SAN would have to be set up with native HDFS volumes. Correct me if I'm wrong. I realize this may go against the past fundamentals people have on how to use this software, but like I said, this is purely for R&D PoC testing.
... View more
