02-20-2019 03:59 AM - last edited on 02-21-2019 06:01 AM by cjervis
Amazon has recently introduced partition placement groups
which finally solves the problem of multiple HDFS nodes being placed on the same underlying physical hardware.
Is it on the roadmap to add support for this to director ?
02-20-2019 10:48 AM
To my surprise, I think Altus Director can already work with partition placement groups. It is already possible to name a placement group for EC2 instances in an Altus Director instance template. From my interpretation of the AWS docs, if that placement group has a "partition" strategy, instances can be placed into it even without specifying a partition number (and today, Altus Director doesn't support picking a partition number).
I don't know how EC2 decides which partition in the placement group each instance should then be placed into. Maybe it's random or round robin, which might be good enough for spreading out HDFS datanodes if there are enough partitions. I encourage you to give it a shot and see what happens.
Explicit support isn't on the roadmap right now for Altus Director, but at first glance it doesn't seem too difficult to add it. The AWS plugin is open source, so you could also try adding it in yourself. Essentially, it would involve adding a new configuration property for EC2InstanceTemplate for the partition number and then including its value in RunInstanceRequest objects.
Good luck! If you do some experiments, I'd be interested to know how they turn out.