Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why are T-SQL queries from MS SQL Server 2016 via PolyBase on data in HDFS so slow? Doesn't PolyBase convert T-SQL to MapReduce?

avatar
Contributor

Cannot find any deep learning documentation on how exactly PolyBase works and nothing on performance tuning. Any help would be appreicated.

1 ACCEPTED SOLUTION

avatar

Hi @Clay McDonald. One of the reason is because of MapReduce. Hive uses Tez but Polybase is not compatible yet with Tez. MapReduce is a batch data processing engine. You will also want to make sure your Hive tables are properly configured using best practices. Try implementing some of these rules where applicable http://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/.

Also be aware of your cluster size. MapReduce (as well as other data processing engines) use parallel processing but if you don't have many nodes than you are taking advantage of the design.

Note sure if its applicable in your case but you could use multiple SQL Servers to parallelize your Polybase query. https://msdn.microsoft.com/en-us/library/mt607030.aspx

View solution in original post

1 REPLY 1

avatar

Hi @Clay McDonald. One of the reason is because of MapReduce. Hive uses Tez but Polybase is not compatible yet with Tez. MapReduce is a batch data processing engine. You will also want to make sure your Hive tables are properly configured using best practices. Try implementing some of these rules where applicable http://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/.

Also be aware of your cluster size. MapReduce (as well as other data processing engines) use parallel processing but if you don't have many nodes than you are taking advantage of the design.

Note sure if its applicable in your case but you could use multiple SQL Servers to parallelize your Polybase query. https://msdn.microsoft.com/en-us/library/mt607030.aspx