we added journalnode and removed journalnode at the same time and also tried to add just journalnode in 2 different scenarios. In both scenarios we have oboserved both namenodes came down and also all the services went down. Over the course of doing we got many errors. is it normal ?
Please note, while adding/removing the journal nodes from the running cluster, you need to ensure a quorum of journal nodes available for NameNodes. (As cited in the shared document--> NameNode high availability requires that you must maintain at least three, active JournalNodes in your cluster.) It means NameNode ensures at least a quorum of Journal Nodes (2 of 3 journal nodes) available for edits log write at any given point Failing to write edits into a quorum of journal nodes, NameNode is expected to crash (shutdown itself). I believe this could be the scenario in your case.
So you need to add new journal nodes first to the cluster before removing the old Journal nodes one by one ensuring a quorum of journal nodes available in the cluster.
If you see NN crashed even after edits log write was successful on a quorum of JNs, then we need to check the NN log for any other issues.