I got confirmation from HW that HDP 2.3.2 is Spark 1.4.1 okay. The source of truth is the release notes. So it looks like a bug in Ambari (maybe just with redhat7) FYI here is the Ambari screenshot ... View more

@Artem Ervits Thanks for the quick reply 🙂 The HDP.repo is saying it is definitely 2.3.2 okay. Yum is also showing the packages are Spark 1.4.1 as expected. spark.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark-master.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark-python.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark-worker.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark_2_3_2_0_2950.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark_2_3_2_0_2950-master.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark_2_3_2_0_2950-python.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 spark_2_3_2_0_2950-worker.noarch 1.4.1.2.3.2.0-2950.el6 HDP-2.3 And spark-shell is also showing 1.4.1. Weird! So I am thinking Ambari is showing the wrong version somehow (not relating to the actual repos/packages on the cluster). We've had this problem before so maybe a bug? (the issue before was the version of Spark reported in HDP 2.2.4.2-2 in Stacks and Versions within the Admin menu was 1.2.0 and not 1.2.1 which was the actual version installed). btw, the reason we need a specific version is we are about to release to market our product and it is in solution validation. They are expecting Spark 1.4.1 and that is what we have tested with so this was a bit of a surprise. Thankfully it seems like a minor issue in Ambari? And going forward yes we definitely want keep up to date with Spark versions. thanks, Alan ... View more