Yarn ClassPath Value for Polybase Pushdown

[#1]

I have Polybase setup and configured to work with my HDP Sandbox. I am able to Query my ORC table successfully but when I try to use Pushdown the query fails. I setup the Pushdown by copying the yarn.application.classpath from my yarn-site.xml in the path /etc/hadoop/conf on my HDP Sandbox to the yarn-site.xml file located in the path C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf.

<name>yarn.application.classpath</name> <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value> </property>

When I review the job log within Ambari I see the below error. Can anyone tell me what I am missing in my classpath?

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

[#2]

Tim - I do not see it as a setup issue because I am making it to the cluster and it is failing as it is trying to run the job on the cluster. Plus, I can run a query from the SQL Server 216 install on my ORC table successfully. The SQL 2016 instructions for setting up the pushdown is only to copy the classpath value so not a lot there as to How-To".

[#3 · Accepted solution]

Got it!! I added the below property to the mapred-site.xml file and the query ran successfully.

<property> <name>mapreduce.app-submission.cross-platform</name> <value>True</value> </property>

[#4]

@Timothy Spann I have the exact same issue. I need to modify the hdp.version and also add mapreduce to the yarn classpath. However as they are both in a different services, if I update one and restart the service, it overwrites the previously written one. How can I persist the configurations done on different services?

,

I am having this exact issue. If you configure one service, say YARN, you have to restart. This overwrites the configuration changes you have done in mapred-site.xml. Isn't there a way to have all changes to all configs persist? Or is this a bug?

[#5]

I have added mapreduce.app-submission.cross-platform property but got the same error.

SQL Server 2017, HDP 2.2

Does anyone have other ideas?