Is this for proof of concept/discovery?  This would be a tightly packed cluster you are proposing for most production environments IMHO .  Our account team provides Solutions Engineering /Solutions Architecture guidance for things like this as you begin to scope out revenue generating activity with a cluster and want enterprise support.   Review our blogs discussion here on hardware sizing: http://blog.cloudera.com/blog/2013/08/how-to-select-the-right-hardware-for-your-new-hadoop-cluster/   As a general rule of thumb, try to minimize the services co-located with the NameNodes (NN). JN's and ZK nodes are OK to co-locate with them, we recommend JN's be co-located with the NN's + 1.     You have a over-loaded mix of services here in the combination of HBase and Mapreduce services on the nodes (there will be battles for memory if you are under-sized.)   If this is a dev or non prod footprint you can get by with what I'm proposing below... HBase can take a lot of memory so you want to monitor. MR jobs are variable based on the complexity and size of what you are doing.   Secondary Name Node (SNN) as an architecture is less safe then using Name Node High Availability (NN HA).  SNN does not do what you think it does.  Read the hadoop operations guide book (3rd edition) to have a better sense of this.   Once you enable NN HA  you end up deploying 3 zookeeper instances and jounal nodes, so based on what you are presenting you are saddling up for future outage / loss of data if this ends up in prod this way unless you are really really careful (and even then you could get hit). This footprint viability really depends on your workload... so you might end up relocating things once you start observing activity, what I'm proposing below is, at best, a playpen configuration so  you can kick the tires and check stuff out.   You are using 3 separate DB implementations with Impala (Fast SQL perf), Hive (Slower SQL but more std SQL support) and HBase (Column based DB)... Is your design really requiring all 3 (research them a little bit more, add them if it makes sense after initial deployment). HUE is usually in the mix to facilitate end user web based access too. Realize you can move services after deployment.  Note that decommissioning a DataNode takes a while to replicate data blocks off the node to the rest. Read up on Hive and Hive2. Hive2 is for more secure implementations (plus other stuff)   NN Active:  Jounal Node, ZooKeeper JobTracker,  NN (standby):  Journal Node, ZK, JobTracker (if using JT HA?) HBaseMaster   ZK, All Hive + Metastore, Sqoop, Impala StateStore Daemon,  DataNode01:  TaskTracker, Impala Daemon, HBase RS DataNode02:  TaskTracker, Impala Daemon, HBase RS DataNode03:  TaskTracker, Impala Daemon,  HBase RS DataNode04:  TaskTracker, Impala Daemon,  HBase RS Admin01 (VM):  Cloudera Manager Cloudera Mgmt Services (Monitors), DB for Monitoring.  Note that using a VM for this will be a "heavy vm" with regards to network IO/Disk IO as cluster activity scales. ... View more

Yes,   There are dependancies, add your second DVD to the repo, as there are older releases of the cyrus-sasl libs that are pulled in by the 1.6.0_31 JDK that are on DVD2.   Not having RHEL repo and patch access is not reccomended for production environments however.   Todd ... View more

Per e-mail thread discussion 1.7 JDK was manually added before cluster upgrade, and was present within custom local yum repo at the site.   CM/CDH will pick the later JDK present to use.  Per Phillp L:   Yes. We ask yum to install jdk and if yum decides that it's appropriate to install a newer version of jdk than is already installed, that is what it will do.   If you want to keep a package on a particular version, even if a newer version is available in a repo, there is a way to make yum do that:   http://blog.serverbuddies.com/how-can-i-get-yum-to-keep-package-at-a-certain-version/       ... View more

Nothing is IPv6 only at this point, so you don't have to worry.  Its going to take a while before we (the internets) can do IPv6 only for things.   ... View more

In your hosts file; do not comment out the loopback interface (127.0.0.1) just let that be its normal values, you can allow the ipv6 value to be set as well, it is not necessary to comment either of those out. From your command line in the shell, do a "getent hosts cdh1.jsnewland.com" and "getent hosts 192.168.125.135" to verify name resolution is doing what you want.  If it comes back with unexpected values, verify in your vm's that /etc/nsswitch.conf is set for  "hosts   files dns" in that order, rather than "hosts   dns  files". What is the host OS you are using for the VM? If you had a 8GB system, you would be much better off running a single  3 to 4 GB VM.  You need to realize the parent OS (especially if GUI desktop is in use) is going to need memory, including overhead to run the actual vm servers and instances. At this scale of physical system (6GB RAM); attempting to emulate a cluster of 3 x 2GB nodes is going to get in the way of your attempt to use hadoop.  Take a look at our Example VM that is available for download, it's set up to run in a laptop/desktop configuration.  The sample vm uses 4GB as its base memory configuration. For the vmstat information you provided, here is the breakdown of what it is telling you: The attached file vmstat-test.txt is a test of making a path 12 times on a VM with 12GB RAM with 6GB swap configured, on a physical host with 128GB ram. Note the differences from your output. Note in the explanation of the column titles for vmstat, my tag of "<---" below indicate what you should focus on when evaluating vmstat output.  Compare your vmstat to the test I did in the attached file. You are heavily swapping.  It's not a question of being out of swap (it would crash at that point), its the volume of activity of paging back and forth that is literally choking the VM.  Below is your vmstat re-pasted: # vmstat 2 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st  2 21 913364  52192    640  20384  212  411   673   441  266  521  8  3 67 22  0  1 23 912360  60864    496  19840 1398  572  1912   576  356  578  8  3  0 90  0  0 17 911292  57136    504  19892 2200  304  2256   304  340  621  4  3  0 93  0  1 17 909904  50180    536  22032 1530   18  2600    18  376  630  7  5  0 88  0  1 15 908268  49372    536  23460 1906   22  2614    22  341  643  3  3  0 95  0  2 19 906812  49084    544  25304 1838    0  3014     0  328  778  3  5  0 92  0  0 15 906036  49152    532  26032 1582  220  2500   220  297  540  3  5  0 92  0  3 16 908180  62844    536  23092 1460 2220  2286  2220  477  591 18  8  0 74  0  2 12 906608  58860    536  25644 1830    0  3120     0  440  603 10 11  0 79  0  3 21 904808  53412    536  26244 2370    0  2668     0  578  767  5  9  0 86  0 Now to understand how to read the vmstat output. Procs        r: The number of processes waiting for run time.        b: The number of processes in uninterruptible sleep. <<---    Memory        swpd: the amount of virtual memory used. <<----        free: the amount of idle memory.        buff: the amount of memory used as buffers.        cache: the amount of memory used as cache.        inact: the amount of inactive memory. (-a option)        active: the amount of active memory. (-a option)    Swap        si: Amount of memory swapped in from disk (/s).  <<<----        so: Amount of memory swapped to disk (/s).         <<<----    IO        bi: Blocks received from a block device (blocks/s).        bo: Blocks sent to a block device (blocks/s).    System        in: The number of interrupts per second, including the clock.  <----        cs: The number of context switches per second.    <----    CPU        These are percentages of total CPU time.        us: Time spent running non-kernel code. (user time, including nice time)        sy: Time spent running kernel code. (system time)        id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.        wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.  <<<----        st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown. ... View more