Recently we had a situation where
a customer wanted to migrate their nodes running master services (NameNode,
Zookeeper, ZKFC and the Journal nodes) of a HDP 2.2 cluster from VMs to a
physical servers. While swapping servers, they realized that changing host
names required a thorough process planning and some “down the stream”
configuration changes. For
instance, application servers that refer hostname in ZK quorum etc needed
update as well. On top of careful planning, these types of dependent
application changes also caused overhead. There were many manual processes that
could have led to misconfiguration. Having given the background, are there any
plans to address these types of operational challenges? One possible solution
was to create additional DNS aliases (canonical names for each node) and use
these DNS aliases in application configuration instead of hostnames. If this
process were to be implemented, what changes would we have to do with the
installation process and patching process in future? This would be helpful if
one has to swap the underlying server to a different name or change hostname or
IP or even move servers from one data center to another. Do you foresee any
challenge down the line in terms of maintenance or other operations? Though
this does come with additional work of generating alias for each node,
especially for large clusters such as one with 500 or
1000 nodes, it would provide value for the operations
team. Generating additional alias would be a onetime challenge
that could be addressed with text editing utilities. A small one time cost for
a long-term benefit
... View more
