There's a classic issue/exception in HDFS HA with ZKFC, that is, "IPC's epoch X is less than the last promised epoch Y". What are the best suggested steps to troubleshoot the problem? How to find the root cause? What are the possible reasons? Thanks. ... View more

In HDFS documentation, it said the steps to do rollback of rolling upgrade: Shutdown all NNs and DNs. Restore the pre-upgrade release in all machines. Start NN1 as Active with the "-rollingUpgrade rollback" option. Run `-bootstrapStandby' on NN2 and start it normally as standby. Start DNs with the "-rollback" option. Why do both NameNodes have different option? E.g. for NN1, -rollingUpgrade rollback, AND for NN2, '-bootstrapStandby' instead must be used. What's the difference of '-bootstrapstandby' and '-rollback'? In addition, it's expected to explain what happened behind hoods if any of the two is specified. Thank you! ... View more

In a typical HDFS HA deployment, Zookeeper, ZKFailoverController, DataNodes, NameNodes and JournalNodes, are all installed. What's the best order of starting/stoping these HDFS services to avoid running into services dependency issues? ... View more

In the process of exploring HDFS HA with ZKFC, I noticed 'dfs.ha.fencing.methods' is configured as 'shell(/bin/true)'. Would anyone explain what's the purpose of this conf? As a bonus, it's better to highlight high level failover flow within which how this conf is applied? Thanks. ... View more

For some unknown reasons, data read through DataNode could be very slow. In addition to troubleshooting root cause of slowness, are there any alternative ways (e.g. different input channels) but with the same semantics to make read potentially faster? Thanks. ... View more