Yarn ClassPath Value for Polybase Pushdown

[#1]

I have Polybase setup and configured to work with my HDP Sandbox. I am able to Query my ORC table successfully but when I try to use Pushdown the query fails. I setup the Pushdown by copying the yarn.application.classpath from my yarn-site.xml in the path /etc/hadoop/conf on my HDP Sandbox to the yarn-site.xml file located in the path C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf.

<name>yarn.application.classpath</name> <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value> </property>

When I review the job log within Ambari I see the below error. Can anyone tell me what I am missing in my classpath?

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

[#2]

@Artem Ervits can you give me a hint as to what is wrong with my classpath?

[#3]

Go to ambari, click on download client configs, place mapred-site, core-site and hdfs-site into the same location as yarn-site.

[#4]

I am working on this suggestion.

[#5]

Artem - after adding the files into the config folder in the Polybase Path I got the below error. I checked the mapred-site.xml file and the path it is unable to parse is in the mapreduce.application.framework.path. Should I go though the file and hardcode current in place of where I see ${hdp.version}?

Java exception raised on call to JobSubmitter_SubmitJob: Error [Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path] occurred while accessing external file.'

[#6]

Ok - I got past the parse error by substituting the version number and the job was submitted to my sandbox and a different error showed up this time that is below.

Application application_1465576751567_0001 failed 2 times due to AM Container for appattempt_1465576751567_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0001Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e38_1465576751567_0001_02_000001 Exit code: 1 Exception message: /hadoop/yarn/local/usercache/pdw_user/appcache/application_1465576751567_0001/container_e38_1465576751567_0001_02_000001/launch_container.sh: line 11: %PWD%;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;job.jar/job.jar;job.jar/classes/;job.jar/lib/*;%PWD%/*: bad substitution

[#7]

It still says bad substitution, you check the /var/log/Hadoop, specifically yarn and mapreduce logs

[#8]

I got past the substitution error by adding the hdp version value to the map-site.xml file. I am now getting the below error. Any suggestions?

Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 2016-06-11 03:04:34,485 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1465576751567_0007 State change from FINAL_SAVING to FAILED 2016-06-11 03:04:34,485 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(323)) - USER=pdw_user OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1465576751567_0007 failed 2 times due to AM Container for appattempt_1465576751567_0007_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0007Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e38_1465576751567_0007_02_000001 Exit code: 1 Exception message: /bin/bash: line 0: fg: no job control Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. APPID=application_1465576751567_0007

[#9]

Here is the Property I added.

<property> <name>hdp.version</name> <value>2.4.0.0-169</value> </property>

[#10]

Make sure everything is setup properly:

https://realizeddesign.blogspot.com/2015/09/setting-up-polybase-in-sql-server-2016.html

Also engage Microsoft support if something appears to be SQL Server related.

https://msdn.microsoft.com/en-us/library/mt163689.aspx

[#11]

Tim - I do not see it as a setup issue because I am making it to the cluster and it is failing as it is trying to run the job on the cluster. Plus, I can run a query from the SQL Server 216 install on my ORC table successfully. The SQL 2016 instructions for setting up the pushdown is only to copy the classpath value so not a lot there as to How-To".

[#12 · Accepted solution]

Got it!! I added the below property to the mapred-site.xml file and the query ran successfully.

<property> <name>mapreduce.app-submission.cross-platform</name> <value>True</value> </property>

[#13]

@Timothy Spann I have the exact same issue. I need to modify the hdp.version and also add mapreduce to the yarn classpath. However as they are both in a different services, if I update one and restart the service, it overwrites the previously written one. How can I persist the configurations done on different services?

,

I am having this exact issue. If you configure one service, say YARN, you have to restart. This overwrites the configuration changes you have done in mapred-site.xml. Isn't there a way to have all changes to all configs persist? Or is this a bug?

[#14]

I have added mapreduce.app-submission.cross-platform property but got the same error.

SQL Server 2017, HDP 2.2

Does anyone have other ideas?