Hi Jaimie,   Just did a little google and found below things to be set. This may resolve the issue:   set hive.tez.container.size=2048; set hive.tez.java.opts=-Xmx1700m; --set this 80% of hive.tez.container.size   Please give it a try if you feel confident on this.   Thanks snm1502 ... View more

Hello @bgoogley, Thank you for the reply. We have already set the swappiness to 1. Still getting these swappiness warnings. Regards, Snm ... View more

Hello Vogone, Thanks for the reply. Yes, swappiness is set to 1. Regards, snm ... View more

Thank you for the prompt response, Harsh.   Output of ls -l /tmp/training:   [training@localhost ~]$ ls -l /tmp/training/ total 36 -rw-rw-r-- 1 training training 3175 Jan 15 21:14 hive_job_log_training_201601152114_2081367348.txt -rw-rw-r-- 1 training training 3175 Jan 15 21:15 hive_job_log_training_201601152115_1734130449.txt -rw-rw-r-- 1 training training 9762 Jan 15 21:15 hive.log -rw-rw-r-- 1 training training 15704 Jan 15 21:05 training_20160115210505_a8632532-ad5d-40c8-8265-b0147c38655c.log Also, even after setting the HADOOP_CLIENT_OPTS before running the CLI and setting hadoop.tmp.dir=.; I am getting the same error:   [training@localhost ~]$ HADOOP_CLIENT_OPTS="-Djava.io.tmpdir=." [training@localhost ~]$ hive Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties Hive history file=/tmp/training/hive_job_log_training_201601152151_801169159.txt hive> use case_ipl; OK Time taken: 1.814 seconds hive> select * from bowling_no_partition limit 5; OK 1 RV Uthappa Kolkata 36.4 299 16 18.68 8.15 13.7 2 GJ Maxwell Punjab 14.2 113 3 37.66 7.88 28.6 4 DR Smith Chennai 13.0 135 4 33.75 10.38 19.5 7 SK Raina Chennai 8.0 43 4 10.75 5.37 12.0 8 JP Duminy Delhi 47.5 334 13 25.69 6.98 22.0 Time taken: 0.806 seconds hive> select name from bowling_no_partition; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/training/training_20160115215252_e2cea1ec-fc8e-46e4-ac82-6357d76ce6d1.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 1; number of reducers: 0 2016-01-15 21:52:25,188 null map = 0%, reduce = 0% 2016-01-15 21:52:41,173 null map = 100%, reduce = 0% Ended Job = job_1452912287088_0002 with errors Error during job, obtaining debugging information... Examining task ID: task_1452912287088_0002_m_000000 (and more) from job job_1452912287088_0002 Unable to retrieve URL for Hadoop Task logs. Does not contain a valid host:port authority: local Task with the most failures(4): ----- Task ID: task_1452912287088_0002_m_000000 URL: Unavailable ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-16_114_6229068466610181659/_task_tmp.-ext-10001 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529) ... 9 more Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-16_114_6229068466610181659/_task_tmp.-ext-10001 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:674) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:246) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:234) ... 20 more Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-1 Logs: /tmp/training/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> set hadoop.tmp.dir=.; hive> select name from bowling_no_partition; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/training/training_20160115215252_1aa2c785-f1ba-45db-b3c1-4a2f45a26be6.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 1; number of reducers: 0 2016-01-15 21:52:58,406 null map = 0%, reduce = 0% 2016-01-15 21:53:15,462 null map = 100%, reduce = 0% Ended Job = job_1452912287088_0003 with errors Error during job, obtaining debugging information... Examining task ID: task_1452912287088_0003_m_000000 (and more) from job job_1452912287088_0003 Unable to retrieve URL for Hadoop Task logs. Does not contain a valid host:port authority: local Task with the most failures(4): ----- Task ID: task_1452912287088_0003_m_000000 URL: Unavailable ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-49_840_1514077634964084555/_task_tmp.-ext-10001 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529) ... 9 more Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-49_840_1514077634964084555/_task_tmp.-ext-10001 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:674) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:246) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:234) ... 20 more Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-1 Logs: /tmp/training/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> Thanks snm1523 ... View more

Hello Harsh,   I am back with the same issue. Using a fresh copy of the VM.   1. Have verified that /tmp/training folder exists 2. Have cleaned the folder and tried the same query again its not working 3. Have checked the permissions of tmp and training folder and seems correct: drwxrwxrwt. 34 root root 4096 Jan 15 20:29 tmp drwxrwxrwx 2 training training 4096 Jan 15 20:28 training 4. NodeManager service is also running.   Still I get below error while executing select name from batting;   hive> select name from batting; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/training/training_20160115203535_f4fdd957-7680-48e3-b7d4-ac26a73b48b8.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 1; number of reducers: 0 2016-01-15 20:35:18,502 null map = 0%, reduce = 0% 2016-01-15 20:35:34,393 null map = 100%, reduce = 0% Ended Job = job_1452671681915_0035 with errors Error during job, obtaining debugging information... Examining task ID: task_1452671681915_0035_m_000000 (and more) from job job_1452671681915_0035 Unable to retrieve URL for Hadoop Task logs. Does not contain a valid host:port authority: local Task with the most failures(4): ----- Task ID: task_1452671681915_0035_m_000000 URL: Unavailable ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":29,"name":"PA Patel","runs":205,"high_score":57,"average":20.5,"strike_rate":110.81,"sixes":1,"team":"Bangalore"} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":29,"name":"PA Patel","runs":205,"high_score":57,"average":20.5,"strike_rate":110.81,"sixes":1,"team":"Bangalore"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_20-35-09_549_8437051062805036433/_task_tmp.-ext-10001 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531) ... 9 more Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_20-35-09_549_8437051062805036433/_task_tmp.-ext-10001 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:674) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:246) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:234) ... 20 more Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-1 Logs: /tmp/training/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask Request you to please help on this.   Thanks snm1523 ... View more

Thank you for the help, Harsh. Found NM service was down, restarted the service and its working now. ... View more

Thank you for the help, Harsh.   Have updated to version 1.8.0_65 which worked.   Thanks Snm1523 ... View more

Hello Harsh,   Apologies for very late reply. However, was stuck in some other issues.   I am using Cloudera-udacity 4.1.1 VM which has come preconfigured with the said version.   I am logged in with training as user.   Upon attempting the said commands, I found that folder training did not exist instead a folder 'hive-training' was present. So I renamed hive-training to training in HDFS and then was able to execute the said commands.   [training@localhost ~]$ hadoop fs -ls /tmp/training [training@localhost ~]$ hadoop fs -ls -d /tmp/training Found 1 items drwxr-xr-x - training supergroup 0 2015-12-09 01:59 /tmp/training [training@localhost ~]$ hadoop fs -ls -d /tmp/ Found 1 items drwxrwxrwt - hdfs supergroup 0 2015-12-28 09:52 /tmp [training@localhost ~]$ Also, I noticed Hive warehouse is stored under /user. Below is an output to few additional commands:   [training@localhost ~]$ hadoop fs -ls /user Found 5 items drwxrwxrwt - yarn supergroup 0 2014-07-27 09:35 /user/history drwxr-xr-x - hue supergroup 0 2013-09-05 20:08 /user/hive drwxr-xr-x - hue hue 0 2013-09-10 10:37 /user/hue drwxr-xr-x - training supergroup 0 2015-12-08 22:36 /user/training drwxr-xr-x - training supergroup 0 2014-07-27 03:52 /user/trial [training@localhost ~]$ hadoop fs -ls /user/training Found 19 items drwx------ - training supergroup 0 2015-12-28 09:53 /user/training/.staging -rw-r--r-- 1 training supergroup 2144 2015-12-08 22:12 /user/training/Batting.csv drwxr-xr-x - training supergroup 0 2015-12-08 22:36 /user/training/hive_case drwxr-xr-x - training supergroup 0 2014-07-27 07:35 /user/training/out drwxr-xr-x - training supergroup 0 2014-07-27 08:20 /user/training/out1 drwxr-xr-x - training supergroup 0 2014-07-27 09:01 /user/training/out10 drwxr-xr-x - training supergroup 0 2014-07-27 09:10 /user/training/out11 drwxr-xr-x - training supergroup 0 2014-07-27 09:11 /user/training/out12 drwxr-xr-x - training supergroup 0 2014-07-27 09:45 /user/training/out13 drwxr-xr-x - training supergroup 0 2014-07-27 09:35 /user/training/out19 drwxr-xr-x - training supergroup 0 2014-07-27 08:15 /user/training/out2 drwxr-xr-x - training supergroup 0 2014-07-27 09:37 /user/training/out20 drwxr-xr-x - training supergroup 0 2014-07-27 08:21 /user/training/out3 drwxr-xr-x - training supergroup 0 2014-07-27 08:25 /user/training/out4 drwxr-xr-x - training supergroup 0 2014-07-27 08:38 /user/training/out5 drwxr-xr-x - training supergroup 0 2014-07-27 08:43 /user/training/out6 drwxr-xr-x - training supergroup 0 2014-07-27 08:45 /user/training/out7 drwxr-xr-x - training supergroup 0 2014-07-27 08:58 /user/training/out8 drwxr-xr-x - training supergroup 0 2014-07-27 08:59 /user/training/out9 [training@localhost ~]$ hadoop fs -ls /user/hive Found 1 items drwxrwxrwx - hue supergroup 0 2015-12-09 02:00 /user/hive/warehouse In addition, I able to execute select * from batting; successfully however, upon attempting select name from batting; this time it is not returning any error instead the job is just going in pending state. I am also unable to kill the job as I get below error.   [training@localhost ~]$ hadoop job -kill application_1451313861984_0001 DEPRECATED: Use of this script to execute mapred command is deprecated. Instead use the mapred command for it. Exception in thread "main" java.lang.IllegalArgumentException: JobId string : application_1451313861984_0001 is not properly formed at org.apache.hadoop.mapreduce.JobID.forName(JobID.java:156) at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:273) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1250) Lastly, I have attached a screenshot of Applications Manager page which shows that the jobs are in pending state.     Please advise.   Thanks snm1523 ... View more