Created 06-09-2016 02:32 PM
I have Polybase setup and configured to work with my HDP Sandbox. I am able to Query my ORC table successfully but when I try to use Pushdown the query fails. I setup the Pushdown by copying the yarn.application.classpath from my yarn-site.xml in the path /etc/hadoop/conf on my HDP Sandbox to the yarn-site.xml file located in the path C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf.
<name>yarn.application.classpath</name> <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value> </property>
When I review the job log within Ambari I see the below error. Can anyone tell me what I am missing in my classpath?
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Created 06-11-2016 04:19 AM
Got it!! I added the below property to the mapred-site.xml file and the query ran successfully.
<property> <name>mapreduce.app-submission.cross-platform</name> <value>True</value> </property>
Created 06-10-2016 02:45 AM
@Artem Ervits can you give me a hint as to what is wrong with my classpath?
Created 06-10-2016 03:45 AM
Go to ambari, click on download client configs, place mapred-site, core-site and hdfs-site into the same location as yarn-site.
Created 06-10-2016 05:21 PM
I am working on this suggestion.
Created 06-10-2016 05:45 PM
Artem - after adding the files into the config folder in the Polybase Path I got the below error. I checked the mapred-site.xml file and the path it is unable to parse is in the mapreduce.application.framework.path. Should I go though the file and hardcode current in place of where I see ${hdp.version}?
Java exception raised on call to JobSubmitter_SubmitJob: Error [Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path] occurred while accessing external file.'
Created 06-10-2016 06:17 PM
Ok - I got past the parse error by substituting the version number and the job was submitted to my sandbox and a different error showed up this time that is below.
Application application_1465576751567_0001 failed 2 times due to AM Container for appattempt_1465576751567_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0001Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e38_1465576751567_0001_02_000001 Exit code: 1 Exception message: /hadoop/yarn/local/usercache/pdw_user/appcache/application_1465576751567_0001/container_e38_1465576751567_0001_02_000001/launch_container.sh: line 11: %PWD%;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;job.jar/job.jar;job.jar/classes/;job.jar/lib/*;%PWD%/*: bad substitution
Created 06-10-2016 11:54 PM
It still says bad substitution, you check the /var/log/Hadoop, specifically yarn and mapreduce logs
Created 06-11-2016 03:26 AM
I got past the substitution error by adding the hdp version value to the map-site.xml file. I am now getting the below error. Any suggestions?
Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 2016-06-11 03:04:34,485 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1465576751567_0007 State change from FINAL_SAVING to FAILED 2016-06-11 03:04:34,485 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(323)) - USER=pdw_user OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1465576751567_0007 failed 2 times due to AM Container for appattempt_1465576751567_0007_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0007Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e38_1465576751567_0007_02_000001 Exit code: 1 Exception message: /bin/bash: line 0: fg: no job control Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Created 06-11-2016 03:30 AM
Here is the Property I added.
<property> <name>hdp.version</name> <value>2.4.0.0-169</value> </property>
Created 06-10-2016 03:37 PM
Make sure everything is setup properly:
https://realizeddesign.blogspot.com/2015/09/setting-up-polybase-in-sql-server-2016.html
Also engage Microsoft support if something appears to be SQL Server related.
Created 06-10-2016 05:24 PM
Tim - I do not see it as a setup issue because I am making it to the cluster and it is failing as it is trying to run the job on the cluster. Plus, I can run a query from the SQL Server 216 install on my ORC table successfully. The SQL 2016 instructions for setting up the pushdown is only to copy the classpath value so not a lot there as to How-To".
Created 06-11-2016 04:19 AM
Got it!! I added the below property to the mapred-site.xml file and the query ran successfully.
<property> <name>mapreduce.app-submission.cross-platform</name> <value>True</value> </property>
Created 09-16-2016 08:39 PM
@Timothy Spann I have the exact same issue. I need to modify the hdp.version and also add mapreduce to the yarn classpath. However as they are both in a different services, if I update one and restart the service, it overwrites the previously written one. How can I persist the configurations done on different services?
,I am having this exact issue. If you configure one service, say YARN, you have to restart. This overwrites the configuration changes you have done in mapred-site.xml. Isn't there a way to have all changes to all configs persist? Or is this a bug?
Created 01-15-2018 02:01 AM
I have added mapreduce.app-submission.cross-platform property but got the same error.
SQL Server 2017, HDP 2.2
Does anyone have other ideas?