@Ram Ghase You are trying to follow-up a demo with Spark 2.1, but your sandbox is at best at 2.0. You should follow tutorials that are supported by the version of Spark deployed on HDP 2.5 sandbox, Spark 1.6.2. Spark 2.0 is also possible, but I would wait for HDP 2.6 sandbox which is to be released probably next month. The error is self-explanatory. If you wish to address it, you could add those missing libraries. ... View more

@Subramaniyam KMV I assume you mean mosquito mqtt. Here is an example of installation on centos-7: https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-the-mosquitto-mqtt-messaging-broker-on-centos-7 You can probably skip the "secure" part. This is not specific to HDP 2.5, you can assume that the sandbox is just a Centos VM for your case. What it matters is the OS and availability of resources. ... View more

@som You would have to be more explicit about versions of Hive, Spark etc, also explain "failing". There is nothing different about accessing Hive views via Hive context from Spark as it is the same as with tables. Anyhow, check the following: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_spark-hive-access.html https://hortonworks.com/hadoop-tutorial/using-hive-with-orc-from-apache-spark/ Hopefully this helps. ... View more

@Ivan Majnaric HDP 2.6, probably next month ... View more

@Lior Hadaya CBO (cost based optimizer) and statistics collected on your tables. You may have the settings mentioned here set to true: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_performance_tuning/content/hive_perf_best_pract_use_col_stats_cost_base_opt.html As such, the behavior can change over time. You could also force stats on a specific table or even column. ... View more

@Sree Kupp The 15 additional seconds are outside of Hive. As such, the changes you make to Tez settings will only impact (positively or negatively) the Hive query, but it will not address your 15 seconds lag outside Hive. You delay is a combination of network latency, ODBC and Power BI rendering it in UI, The focus for tuning should be on ODBC tuning to chunk the data better to increase throughput, address the network latency (if any), caching on Power BI side. There are a few things that you could do on Hive side too, but it is a long shot to guess what you and you did not. For example, use ORC format, use Interactive Query (LLAP), increase the use of caching for map-side joins, improve parallelism by setting a few parameters that will allow a better chunking of the data for increased parallelism, etc. ... View more