Any help would be appreciated? I have researched this for the late 72 hours and nothing found has worked. I have checked security and the folders are owned by yarn and there appears(unless I am missing something) plenty of space on all the drives. All four nodemanagers are still marked as unhealthy and none of my jobs will complete. The below is still listed in the log.   Unable to create directory /data/yarn/nm error mkdir of /data/yarn failed, removing from the list of valid directories. Unable to create directory /data/yarn/container-logs error mkdir of /data/yarn failed, removing from the list of valid directories. Most of the disks failed. 1/1 local-dirs are bad: /data/yarn/nm; 1/1 log-dirs are bad: /data/yarn/container-logs     ... View more

Here is the Property I added. <property> <name>hdp.version</name> <value>2.4.0.0-169</value> </property> ... View more

I got past the substitution error by adding the hdp version value to the map-site.xml file. I am now getting the below error. Any suggestions? Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 2016-06-11 03:04:34,485 INFO rmapp.RMAppImpl (RMAppImpl.java:handle(767)) - application_1465576751567_0007 State change from FINAL_SAVING to FAILED 2016-06-11 03:04:34,485 WARN resourcemanager.RMAuditLogger (RMAuditLogger.java:logFailure(323)) - USER=pdw_user OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1465576751567_0007 failed 2 times due to AM Container for appattempt_1465576751567_0007_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0007Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e38_1465576751567_0007_02_000001 Exit code: 1 Exception message: /bin/bash: line 0: fg: no job control Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. APPID=application_1465576751567_0007 ... View more

Ok - I got past the parse error by substituting the version number and the job was submitted to my sandbox and a different error showed up this time that is below. Application application_1465576751567_0001 failed 2 times due to AM Container for appattempt_1465576751567_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1465576751567_0001Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e38_1465576751567_0001_02_000001 Exit code: 1 Exception message: /hadoop/yarn/local/usercache/pdw_user/appcache/application_1465576751567_0001/container_e38_1465576751567_0001_02_000001/launch_container.sh: line 11: %PWD%;$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure;job.jar/job.jar;job.jar/classes/;job.jar/lib/*;%PWD%/*: bad substitution ... View more

Artem - after adding the files into the config folder in the Polybase Path I got the below error. I checked the mapred-site.xml file and the path it is unable to parse is in the mapreduce.application.framework.path. Should I go though the file and hardcode current in place of where I see ${hdp.version}? Java exception raised on call to JobSubmitter_SubmitJob: Error [Unable to parse '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path] occurred while accessing external file.' ... View more

Tim - I do not see it as a setup issue because I am making it to the cluster and it is failing as it is trying to run the job on the cluster. Plus, I can run a query from the SQL Server 216 install on my ORC table successfully. The SQL 2016 instructions for setting up the pushdown is only to copy the classpath value so not a lot there as to How-To". ... View more

I am working on this suggestion. ... View more

@Artem Ervits can you give me a hint as to what is wrong with my classpath? ... View more

The job still fails with the same error in the log oozie log. Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.PigMain] ... View more

I have copied both files from /etc/hadoop/conf to /usr/hdp/2.4.0.0-169/pig/conf and chmod to 755. Is this correct? ... View more

I did not get any pig jars when I ran grep -iR core-site.xml /usr/hdp/2.4.0.0-169/pig. The only output was the below. Should I copy the xml files to the pig/conf as you instructed? /usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"core-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-4247: S3 properties are not picked up from core-site.xml in local mode (cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3145: Parameters in core-site.xml and mapred-site.xml are not correctly substituted (cheolsoo) ... View more

Here is what my new pig-env.sh looks like. JAVA_HOME=/usr/lib/jvm/java HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client} export HADOOP_CONF_DIR=$HADOOP_CONF_DIR:/etc/hadoop/conf export PIG_CLASSPATH=$PIG_CLASSPATH:$HADOOP_CONF_DIR if [ -d "/usr/lib/tez" ]; then PIG_OPTS="$PIG_OPTS -Dmapreduce.framework.name=yarn" fi ... View more

Here is the output Jitendra. Do you see anything wrong? /usr/hdp/2.4.0.0-169/pig/doc/api/constant-values.html:<td><code>"hadoop-site.xml"</code></td> /usr/hdp/2.4.0.0-169/pig/conf/pig.properties:# By default, Pig expects hadoop configs (hadoop-site.xml and core-site.xml) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-3200: MiniCluster should delete hadoop-site.xml on shutDown (prkommireddi via cheolsoo) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-2491: Pig docs still mention hadoop-site.xml (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-1791: System property mapred.output.compress, but pig-cluster-hadoop-site.xml doesn't (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt:PIG-1186: Pig do not take values in "pig-cluster-hadoop-site.xml" (daijy) /usr/hdp/2.4.0.0-169/pig/CHANGES.txt: 'pig-cluster-hadoop-site.xml' in the non HOD case just like it does in the /usr/hdp/2.4.0.0-169/pig/RELEASE_NOTES.txt:variable to point to the directory with your hadoop-site.xml file and then run ... View more

I am running the HDP Sandbox. My hadoop version command returns the below and I have not changed anything to my recollection about the environment. hadoop version Hadoop 2.7.1.2.4.0.0-169 Subversion git@github.com:hortonworks/hadoop.git -r 26104d8ac833884c8776473823007f176854f2eb Compiled by jenkins on 2016-02-10T06:18Z Compiled with protoc 2.5.0 From source with checksum cf48a4c63aaec76a714c1897e2ba8be6 This command was run using /usr/hdp/2.4.0.0-169/hadoop/hadoop-common-2.7.1.2.4.0.0-169.jar ... View more

Actually that is not the issue. When I looked down in the log I found the below. I will investigate but appreciate any guidance as well. ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath). ... View more

I have took a look at the stderr logs and got this as an output. Does this make sense? SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/23/mapreduce.tar.gz/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/44/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Details at logfile: /hadoop/yarn/local/usercache/mbharrel/appcache/application_1463627394303_0038/container_e16_1463627394303_0038_01_000002/pig-job_1463627394303_0038.log ... View more

The changing of the ulimit appears to have fixed that issue. I am now presented with the below issue but at least it is something different. Thank you Xi for the help. ... View more

I was using IE Edge on my new Windows 10 OS. I just tried it with Chrome and it worked!! Thank you sir. ... View more