Support Questions
Find answers, ask questions, and share your expertise

Why are T-SQL queries from MS SQL Server 2016 via PolyBase on data in HDFS so slow? Doesn't PolyBase convert T-SQL to MapReduce?

Solved Go to solution

Why are T-SQL queries from MS SQL Server 2016 via PolyBase on data in HDFS so slow? Doesn't PolyBase convert T-SQL to MapReduce?

Cannot find any deep learning documentation on how exactly PolyBase works and nothing on performance tuning. Any help would be appreicated.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Why are T-SQL queries from MS SQL Server 2016 via PolyBase on data in HDFS so slow? Doesn't PolyBase convert T-SQL to MapReduce?

Hi @Clay McDonald. One of the reason is because of MapReduce. Hive uses Tez but Polybase is not compatible yet with Tez. MapReduce is a batch data processing engine. You will also want to make sure your Hive tables are properly configured using best practices. Try implementing some of these rules where applicable http://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/.

Also be aware of your cluster size. MapReduce (as well as other data processing engines) use parallel processing but if you don't have many nodes than you are taking advantage of the design.

Note sure if its applicable in your case but you could use multiple SQL Servers to parallelize your Polybase query. https://msdn.microsoft.com/en-us/library/mt607030.aspx

View solution in original post

1 REPLY 1

Re: Why are T-SQL queries from MS SQL Server 2016 via PolyBase on data in HDFS so slow? Doesn't PolyBase convert T-SQL to MapReduce?

Hi @Clay McDonald. One of the reason is because of MapReduce. Hive uses Tez but Polybase is not compatible yet with Tez. MapReduce is a batch data processing engine. You will also want to make sure your Hive tables are properly configured using best practices. Try implementing some of these rules where applicable http://hortonworks.com/blog/5-ways-make-hive-queries-run-faster/.

Also be aware of your cluster size. MapReduce (as well as other data processing engines) use parallel processing but if you don't have many nodes than you are taking advantage of the design.

Note sure if its applicable in your case but you could use multiple SQL Servers to parallelize your Polybase query. https://msdn.microsoft.com/en-us/library/mt607030.aspx

View solution in original post