This is not really required in a HA setup, since you already have two independent fsimage copies being made at 2x NNs. ... View more

The even number disallowance has nothing to do with availability. An odd number of nodes is required to form a "quorum". You cannot have definite majority to a vote request with even number of nodes in a quorum. In an odd set, there's always a majority to a vote's result, and thereby the requirement. Hope this helps you make sense on why CM is requiring you to configure your QJM properly, with an odd number of nodes. ... View more

This may happen if your YARN service is misconfigured w.r.t. memory resources. An MR job requires 1.5g for its AppMaster, and subsequently 1g for each task. Therefore, NodeManagers should be publishing at least 2.5g or 3g for MR jobs to run. Can you ensure your value in CM -> YARN -> Configuration, for field "yarn.nodemanager.resource.memory-mb" is set to at least 3 GB or higher? If it isn't, please set it appropriately as per your machine's ability, and restart the YARN service and retry the job submission. You can also alternatively reduce the default memory demands via client-side properties yarn.app.mapreduce.am.resource.mb (for the MR AppMaster) and set of mapreduce.map.memory.mb/mapreduce.reduce.memory.mb (for the Tasks). These properties accept integer values in MB units. ... View more

Wherever possible, it is always good to combine multiple goals into the same job to avoid extra I/O and additional job bootup/cleanup overheads. Optimisers in Hive and Pig try to do this as well. The reducer can emit different outputs via the MultipleOutputs classes. For map outputs, you'll need to build some form of "general" class and hierarchy that can cater to different styles of output and partition them appropriately. Its definitely possible, but needs some additional work. ... View more

Hive will contact a Hive MetaStore Service iff there's a "hive.metastore.uris" property in hive-site.xml. Your XML doesn't have one. Your XML carries the JDO properties of the DB instead, and is thereby the "local" mode config described at http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_hive_metastore_configure.html ... View more

The job scheduler would be the spot to look at. One way would be to use user-named pools in FairScheduler. You will need to define a FairScheduler <pool> configuration per user, that then control the maxMaps and maxReduces configuration for each of them, as defined at http://archive.cloudera.com/cdh4/cdh/4/mr1/fair_scheduler.html#Allocation+File+Format. ... View more