My question on HDFS using SAN as the backend storage has 3 main parts 1. Is it feasible to use SAN as the back end storage for HDFS? 2. What are the pros and cons of using SAN or NAS for HDFS? 3. Has it been tested for performance and may be other aspects? ... View more

Is it possible to deploy HDP docker container in Mesos using Marathon? If so, where can i get the docker images from and the Marathon recipes? If its not possible with the combination above, what are the options to deploy HDP on Mesos? How is it going to be better than running on Yarn? ... View more

My understanding along with questions as below, AWS-HDCloud Manual scaling using Ambari or AWS UI possible. Auto Scaling 1. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 1.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. -------------------------------------------------------------------------------------------------------------------------------------------------------------- AWS-HDP on IaaS Manual scaling using Ambari is possible. Auto Scaling-Without CloudBreak 2. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 2.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. Auto Scaling-WithCloudBreak Auto-scaling may be possible, but question 2.1 applies here as well. -------------------------------------------------------------------------------------------------------------------------------------------------------------- Azure-HdInsights Manual scaling using Ambari or Azure UI possible. Auto Scaling 3. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 3.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. -------------------------------------------------------------------------------------------------------------------------------------------------------------- Azure-HDP in MarketPlace Manual scaling using Ambari or Azure UI possible. Auto Scaling 4. Is it possible to auto-scale in this option (while creating the cluster can i set auto-scaling group)? 4.1. In that case, how is the data re-balanced? i.e. if a new node is added, then compute may not gain data locality. -------------------------------------------------------------------------------------------------------------------------------------------------------------- Azure-HDP on IaaS Same questions as AWS-HDP on IaaS ... View more

What are the storage options possible when deploying HDP on Cloud? My understanding as follows, 1. Azure (HDInsight, HDP via CloudBreak, HDP in the MarketPlace) WASB - What about parallelism here? i.e. if i store a file here and run a map reduce job processing this file. Would i achieve the same effect as i achieve in HDFS storage? ADLS - Although not co-located, performance can be improved by means of parallelism. HDFS itself - I can move the data to the edge node then copy into HDFS What are my options to move my data into WASB, ADLS? This thread suggests NI-FI but my requirement is ephemeral and NIFI investment may not sell. 2. AWS (Below questions apply to HDCloud, HDP via CloudBreak to AWS) S3 - What about parallelism here? i.e. if i store a file here and run a map reduce job processing this file. Would i achieve the same effect as i achieve in HDFS storage? HDFS itself - I can move the data to the edge node then copy into HDFS And out of these storage options, which one is better over the other and for what reason? ... View more