Why are T-SQL queries from MS SQL Server 2016 via PolyBase on data in HDFS so slow? Doesn't PolyBase convert T-SQL to MapReduce?

[#1]

Cannot find any deep learning documentation on how exactly PolyBase works and nothing on performance tuning. Any help would be appreicated.

[#2 · Accepted solution]

Hi @Clay McDonald. One of the reason is because of MapReduce. Hive uses Tez but Polybase is not compatible yet with Tez. MapReduce is a batch data processing engine. You will also want to make sure your Hive tables are properly configured using best practices. Try implementing some of these rules where applicable http://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/.

Also be aware of your cluster size. MapReduce (as well as other data processing engines) use parallel processing but if you don't have many nodes than you are taking advantage of the design.

Note sure if its applicable in your case but you could use multiple SQL Servers to parallelize your Polybase query. https://msdn.microsoft.com/en-us/library/mt607030.aspx