Member since
04-23-2017
30
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5567 | 06-11-2016 04:19 AM | |
5817 | 05-22-2016 05:51 AM |
06-11-2016
04:19 AM
1 Kudo
Got it!! I added the below property to the mapred-site.xml file and the query ran successfully. <property>
<name>mapreduce.app-submission.cross-platform</name>
<value>True</value>
</property>
... View more
06-11-2016
03:30 AM
Here is the Property I added. <property>
<name>hdp.version</name>
<value>2.4.0.0-169</value>
</property>
... View more
06-11-2016
03:26 AM
I got past the substitution error by adding the hdp version value to the map-site.xml file. I am now getting the below error. Any suggestions? Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
2016-06-11 03:04:34,485 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1465576751567_0007 State change from FINAL_SAVING to FAILED
2016-06-11 03:04:34,485 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(323)) - USER=pdw_user OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1465576751567_0007 failed 2 times due to AM Container for appattempt_1465576751567_0007_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0007Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e38_1465576751567_0007_02_000001
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control
Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
at org.apache.hadoop.util.Shell.run(Shell.java:487)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application. APPID=application_1465576751567_0007
... View more
06-10-2016
06:17 PM
Ok - I got past the parse error by substituting the version number and the job was submitted to my sandbox and a different error showed up this time that is below. Application application_1465576751567_0001 failed 2 times due to AM Container for appattempt_1465576751567_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0001Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e38_1465576751567_0001_02_000001
Exit code: 1
Exception message: /hadoop/yarn/local/usercache/pdw_user/appcache/application_1465576751567_0001/container_e38_1465576751567_0001_02_000001/launch_container.sh: line 11: %PWD%;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;job.jar/job.jar;job.jar/classes/;job.jar/lib/*;%PWD%/*: bad substitution
... View more
06-10-2016
05:45 PM
Artem - after adding the files into the config folder in the Polybase Path I got the below error. I checked the mapred-site.xml file and the path it is unable to parse is in the mapreduce.application.framework.path. Should I go though the file and hardcode current in place of where I see ${hdp.version}? Java exception raised on call to JobSubmitter_SubmitJob: Error [Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path] occurred while accessing external file.'
... View more
06-10-2016
05:24 PM
Tim - I do not see it as a setup issue because I am making it to the cluster and it is failing as it is trying to run the job on the cluster. Plus, I can run a query from the SQL Server 216 install on my ORC table successfully. The SQL 2016 instructions for setting up the pushdown is only to copy the classpath value so not a lot there as to How-To".
... View more
06-10-2016
05:21 PM
I am working on this suggestion.
... View more
06-10-2016
02:45 AM
@Artem Ervits can you give me a hint as to what is wrong with my classpath?
... View more
06-09-2016
02:32 PM
2 Kudos
I have Polybase setup and configured to work with my HDP Sandbox. I am able to Query my ORC table successfully but when I try to use Pushdown the query fails. I setup the Pushdown by copying the yarn.application.classpath from my yarn-site.xml in the path /etc/hadoop/conf on my HDP Sandbox to the yarn-site.xml file located in the path C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\Polybase\Hadoop\conf. <name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*</value>
</property> When I review the job log within Ambari I see the below error. Can anyone tell me what I am missing in my classpath? Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
... View more
Labels:
- Labels:
-
Apache YARN
05-22-2016
05:51 AM
After a lot of trial and error, reading documentation and beers. I was able to figure out that the issue was the name of my pig jar. I renamed it to pig.jar in the lib folder, backed out all changes to my pig-env,sh and my oozie job ran and succeeded. Thank you to everyone for your help and especially to Jitendra for taking the time to give suggestions.
... View more