question Re: Is it recommended to use same set of journal nodes for two hdfs instances ( namespace 1 and namespace 2) with different shared edits in production environment? in Archives of Support Questions (Read Only)

Is it recommended to use same set of journal nodes for two hdfs instances ( namespace 1 and namespace 2) with different shared edits in production environment?

kannanwebsphere — Fri, 16 Sep 2022 10:04:27 GMT

Namespace1:

<name>dfs.namenode.shared.edits.dir</name> <value>qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/mycluster1</value> </property>

Namespace2:

<name>dfs.namenode.shared.edits.dir</name> <value>qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/mycluster2</value> </property>

Please advise.

Re: Is it recommended to use same set of journal nodes for two hdfs instances ( namespace 1 and namespace 2) with different shared edits in production environment?

jstraub — Wed, 17 Feb 2016 14:59:10 GMT

It is definitely possible to do that, however I would not recommend it, especially in a production environment. These JN processes are just lightweight daemons, so you can place them on the same nodes with other master services. Using one Quorum for multiple clusters increases the risk and chance of affecting the health/stability of all the attached clusters. For example if Cluster A brings down your JN Quorum (for whatever reason), the Namenodes of Cluster B cant synchronize their state and will shutdown eventually because the Quorum is not available =>

2016-02-16 22:55:55,550 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: flush failed for required journal (JournalAndStream(mgr=QJM to [XXXXX:8485, XXXXXX:8485, xXXXX:8485], stream=QuorumOutputStream starting at txid 51260))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.

Re: Is it recommended to use same set of journal nodes for two hdfs instances ( namespace 1 and namespace 2) with different shared edits in production environment?

cnauroth — Thu, 18 Feb 2016 02:42:43 GMT

+1 Another consideration is upgrades. Sharing the same set of JournalNodes across multiple clusters would complicate upgrade plans, because an upgrade of software on those JournalNodes potentially impacts every cluster served by those JournalNodes.