Hi.  I am having a "What the heck" moment.  Could someone please explain the theory behind this.  I have always presumed that Sqoop - unlike other MR processes that might require the entire dataset to be in memory to work - should not ever have a OOM issue.  Afterall, it is using its memory as a buffer, copying the data from DB to the staging area in HDFS, and when complete, moving from staging to --target-dir.   So, we were moving a fairly large DB (500GB) but our client would only allow us to use 1 mapper (don't ask why...gulp).  About 90 minutes into the process, it terminated with:   Container is running beyond the 'PHYSICAL' memory limit. Current usage: 1.0Gib of 1 GB physical memory used; 2.7GB of 2.1 GB virtual memory used. Killing container   This is really confusing me.  I suppose I can solve the problem by a) increasing the vmem/pmem ratio (yarn.nodemanager.vmem-pmem-ratio = xyz) OR b) not checking for this error (yarn.nodemanager.vmem-check-enabled = false).   But WHY is this error coming up?     Thanks in advance and cheers. ... View more

Hi.  We have a client who has 2 clusters.  On the security cluster, they have sensitive data that they redact and copy to the analysis cluster.  For security reasons, they would like to minimize the number of open ports on the security cluster.  We have successfully tested using distcp from the shell to copy the data with port 8020 open.  They would now like to automate the process through oozie.  In testing, we have run into an error that port 8042 (Node Manager External Port) is not open.     We do not understand why distcp works fine without port 8042 available when run through the shell but fails when called through Oozie.   Any help would be appreciated.  Thanks.   Henry ... View more

We have a client who wants to setup a small separate special purpose cluster.  It will consist of 5 nodes and they want to run Kudu on it.  If we configure with 3 Master nodes and 3 Tablet Servers, at least one of the nodes will have both Master roles as well as Tablet Server role.  In any case, in this scenario, if hypothetically the node with both roles fail, and the replication is set to 3, neither the master tablet nor any of the table tablets will be able to satisfy the replication factor since there are now only 2 master roles and 2 table servers available.   Since this is still within the (n-1)/2 failures, will the service still run?  After 5 minutes, and when the nodes are determined to be permanent failures, a new follower needs to be created but since there are only 2 tablet servers still functioning, what will happen?  Will the system limp along, still functioning or will it fail?   Thanks in advance for any help. ... View more