Hi @R_SHETH, great to hear! Please mark the reply as the accepted answer if it solved your problem 🙂 ... View more

You can override the "spark-submit" association with alternatives, e.g.: /usr/sbin/alternatives --set spark-submit <path-to-spark2-submit> # <path-to-spark2-submit> could be like "/opt/cloudera/parcels/SPARK2-2.2.0-cloudera1-cdh5.13.3.p0.611179/bin/spark2-submit" # This is a path of CDH 5 to the Spark parcel directory. You need to adjust it to your path (the one that spark2-shell is pointing to) # You can find this path with: /usr/sbin/alternatives --display spark2-submit   However, from CDH 6.x, it is normal to use spark-submit instead of spark2-submit because there is only Spark2 included in CDH. Also, it is normal to use the packaged version inside of the CDH distribution, which seems to be 2.4.0 for your CDH 6.1.1. I am not sure, if using a different version is supported or recommended by Cloudera.   How did you install that Spark 2.2 version in your CDH 6 cluster?    Also see https://www.cloudera.com/documentation/enterprise/6/6.1/topics/spark.html ... View more

Hi @lsouvleros, as you already pointed out: this is influenced by a number of factors and widely influenced by your use case and existing organizational context.   Comparing to an HDFS in a classic compute/storage-coupled Hadoop cluster, some of the discussions from here do also apply: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sdx_vpc.html. This is, because Isilon is a network-attached storage and - similar to using Cloudera Virtual clusters - this has some implications on performance, especially for workloads with high-performance requirements. I have also seen environments where using Isilon instead of HDFS had impact on Impala performance.   In terms of reliability and stability, you can argue each way - depending on your architecture. However, a multi-datacenter-deployment is likely to be more easy to realize with Isilon, due to its enterprise-proof replication and failover capabilities.   In terms of efficiently using storage space, Isilon will have advantages. However, the higher cost compared to JBOD-based HDFS might make this point irrelevant.   For scalability, I guess it depends again on your organizational setup. You can easily scale up Isilon by buying more boxes from EMC. There are certainly really large Isilon deployments out there. On the other hand, scaling HDFS is also not hard and can help you to realize huge deployments.   In the end it will be a tradeoff of higher costs with Isilon but with more easy management vs. lower costs by higher efforts with HDFS.    This is my personal opinion and both EMC and Cloudera might have stronger arguments for their respective storage (e.g. [EMC link]). You can also look for the latest announcement for the blog.   Regards, Benjamin ... View more

Hi pollard, most services expose a "/conf" servlet in their WebUI which gives you the most complete set of actually used parameters. This should be the most promising source of thruth. Impala has a similar Servlet with path "/varz".   The instance process view of Cloudera Manager shows you the actually distributed config files - which often helps a lot but does not include default values. You can reach it from a service (e.g. Impala) by clicking on "Instances" -> (the instance you want to see, e.g. Impala Catalog Server on a node) -> Processes   Regards, Benjamin ... View more

I am using CDH/CM 6.2. Will update the cluster and test again. However, according to the docs, it should already work since 5.14. ... View more