Your OS seems to be running out of free port numbers in the ephemeral range. Typically on Linux this is in range 32k to 64k, which is quite a lot of ports. A common reason is abuse of software clients (due to excessive connections being created without use of shared connection pools, or a leak of connections due to non-closure in the code), or lower level problems with the socket closure (such as the FIN stage of TCP not being correctly processed, causing the OS to hold the port open for an extended period of time waiting for the final close to complete). Are you perhaps executing a lot of concurrent programs on your cluster, or use a multi-threaded app that builds a new network client (for HDFS, etc.) under each thread? When you experience this, you could run an lsof check on the host of the failing task to find which PID(s) are occupying most of the network client ephemeral ports and if there is a pattern to their destination(s). This can help figure out where the problem specifically lies, and what category (in the above) it may belong to. ... View more

There was a known issue in Cloudera Manager up until 5.14.4 and 5.15.1 that can cause the necessary container executor configuration files to not be deployed on new or recommissioned NodeManager hosts. The fix for this is noted at https://www.cloudera.com/documentation/enterprise/release-notes/topics/cm_rn_fixed_issues.html#OPSAPS-24398 A better workaround than changing ownership (which is a red herring because of missing symlinks causing the executor binary to look at the wrong path) is to simply add a YARN Gateway role to all NodeManager hosts and perform a 'Deploy Client Configuration' under YARN. Upgrading to 5.15.1 or higher when it comes out should help resolve this issue. ... View more

The error quotes a missing function that has been present in Oozie since CDH 5.5.0. It therefore appears that somehow your environment is keeping or passing around an older jar of 'oozie-sharelib-oozie' artifact that is without this added function. If its your sharelib that's carrying a bad file, you can inspect it via: # hadoop fs -ls -R /user/oozie/ | grep sharelib-oozie The above should return only a single jar file size and the version of the filename should match what you are running. If you get 3 or more files in the output, consider redeploying your ShareLib via https://www.cloudera.com/documentation/enterprise/latest/topics/admin_oozie_sharelib.html#concept_i2f_r5t_2r If you just get one version of the jar instead, then perhaps some application jar of your project(s) is assembling a fat jar that includes Oozie Sharelib dependencies in it, albeit from a non-CDH version, or a very old CDH version (< 5.5.0). You can inspect suspect jars by running: # jar tf filename.jar | grep LauncherMain Repack all such Oozie-including jars to exclude Oozie dependencies in them, as the system classpath will already provide the dependencies and of the right version. ... View more

Please see this prior post comment on AM ranges: http://community.cloudera.com/t5/Batch-Processing-and-Workflow/Where-is-the-setting-for-the-port-range-used-by-org-apache/m-p/38131/highlight/true#M2081 As to firewalls, the general practice I've observed is to setup rules at points of external access into the cluster (such as from user or other cluster networks) but leave the intra-cluster network open for the services within. Our port range has a classification of internal/external if that would help you build your rules: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_ports.html ... View more

> How many vCores allocated for Tasks within the Executors? Tasks run inside pre-allocated Executors, and do not cause further allocations to occur. Read on below to understand the relationship between tasks and executor from a resource and concurrency viewpoint: """ Every Spark executor in an application has the same fixed number of cores and same fixed heap size. The number of cores can be specified with the --executor-cores flag when invoking spark-submit, spark-shell, and pyspark from the command line, or by setting the spark.executor.cores property in the spark-defaults.conf file or on a SparkConf object. Similarly, the heap size can be controlled with the --executor-memory flag or the spark.executor.memory property. The cores property controls the number of concurrent tasks an executor can run. --executor-cores 5 means that each executor can run a maximum of five tasks at the same time. """ Read more at http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/ ... View more

@Harsh J   Thanks for reply. ... View more

Hi Harsh,   Could you please help me how to check /hbase connection in zkcli     ... View more

Got it! Thank you for the explanation.   ... View more

Hi, Any update on node label support for FairScheduler in some future CDH version? Thanks.   Regards, Iván ... View more