The Spark Jar Location (HDFS) (spark_jar_hdfs_path) parameter is set to /user/spark/share/lib/spark-assembly.jar However, the HDFS file /user/spark/share/lib/spark-assembly.jar is NOT there! The only HDFS folder/file for Spark that exists is /user/spark/applicationHistory   Although I have run via CM to 'Upload Spark Jar' (from drop-down Actions option) successfully (at least that's what CM tells me) when I check the spark HDFS folders/files the jar (spark-assembly.jar) is not there!!!   ... View more

I have created a new user (user2) and when running any MapReduce jobs the JobHistoryServer portal (port 19888) does NOT show any!!! I can see the job from witihin the ResourceManager Web UI (port 8088) and when I click on the 'History' link (under the Tracking UI column) I am getting the error: "No Found: job_xxxxxxxxxxxxx_xxxx). This happening for the MapReduce jobs only!!! When I run Spark Jobs (under the same userid, user2) I am able to see the 'History' logs!!!   An existing userid (user1) works fine! It seems to me that the issue is permissions-related.   Here are the HDFS permissions of the userids: [root@master ~]# hdfs dfs -ls /tmp/logs Found 5 items drwxrwxrwt   - user1 hadoop              0 2015-04-20 08:48 /tmp/logs/user1 drwxrwxrwt   - hdfs    supergroup          0 2015-04-09 14:59 /tmp/logs/hdfs drwxrwxrwx   - user2   hadoop              0 2015-05-06 16:23 /tmp/logs/user2 drwxrwxrwt   - root    hadoop              0 2015-04-09 16:46 /tmp/logs/root   Also, here are the persmissions of the HDFS /user/history/done and /user/history/done_intermediate folders: [root@master ~]# hdfs dfs -ls /user/history Found 2 items drwxrwx---   - mapred hadoop          0 2015-05-11 10:32 /user/history/done drwxrwxrwt   - mapred hadoop          0 2015-05-18 14:39 /user/history/done_intermediate     ... View more

Looking for Best Practices!   Having almost all CDH services (and CM) in the Master node and YARN's NodeManager, Spark's Workers, HDFS 's DataNodes, and HBase's RegionServers in the Data nodes, what type of CPU configuration should be suitable? For instance, should I provision the Master host with 20 cores with 3.00GHz of speed (see Intel's Xeon CPU E5-2690 v2 - Ivy Bridge processor, 2 CPU's with 10 cores per socket and 3.00GHz of speed )? Should I provision the Data hosts with 24 cores with 2.70GHz of speed (see Intel's Xeon CPU E5-2697 v2 - Ivy Bridge processor, 2 CPU's with 12 cores per socket, but with 2.70GHz of speed )?   Again, looking for the ultimate configuration and optimizing both cores and speed...   ... View more