Is it possible to create partition like 01 from date like 2017-01-02' where 01 is month ? I have daily sales record and I need to do query like select * from sales where month = '01' . So it will be better if I could partition my daily sales by month.but my data has date of format 2017-01-01 and doing create table tl (columns ......) partitioned by (date <datatype> ) will create partition on daily basis which is the last thing I want . I need to create partition dynamically. ... View more

I wanted to access hive table from servlet. when I run the url I get javax.servlet.ServletException:Servlet execution threw an exception org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)RootCause java.lang.NoClassDefFoundError:Couldnot initialize class org.apache.hadoop.security.UserGroupInformation org.apache.spark.util.Utils$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) org.apache.spark.util.Utils$anonfun$getCurrentUserName$1.apply(Utils.scala:2391) scala.Option.getOrElse(Option.scala:121) org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2391) org.apache.spark.SparkContext.<init>(SparkContext.scala:295) org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320) org.apache.spark.sql.SparkSession$Builder$anonfun$6.apply(SparkSession.scala:868) org.apache.spark.sql.SparkSession$Builder$anonfun$6.apply(SparkSession.scala:860) scala.Option.getOrElse(Option.scala:121) org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)HiveRead.doGet(HiveRead.java:30) javax.servlet.http.HttpServlet.service(HttpServlet.java:635) javax.servlet.http.HttpServlet.service(HttpServlet.java:742) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) I am running this code SparkSession spark =SparkSession.builder().appName("Java Spark SQL basic example").enableHiveSupport().config("spark.sql.warehouse.dir","hdfs://saurab:9000/user/hive/warehouse").config("mapred.input.dir.recursive",true).config("hive.mapred.supports.subdirectories",true).config("hive.vectorized.execution.enabled",true).master("local").getOrCreate(); response.getWriter().println("olo"); I looked at this question and just to test added export SPARK_CLASSPATH=$CLASS_PATH:/home/saurab/hadoopec/spark/jars/hadoop-auth-2.7.3 since I was not using shark Though my aim is to access hive table and run sql using Spark, I am running this code to test. This code has nothing to do with hive table but If I run only this much code I get above exception. I guess it has something to do with authentication, but I can't figure out what. this is my pom.xml IS there a better way to access hive tables and run queries using servlets? ... View more

I encountered a weired behaviour of hive jdbc. when I create table like create table my_table (....other good stuffs..) it gets create inside database named default but when I do create table mydb.mytable (..stuffss..) it gets created inside mydb. I am using spark and hive . Previously I would do like Connection con = DriverManager.getConnection("jdbc:hive2://localhost:10000/", "hiveuser", "hivepassword"); Statement stmt = con.createStatement(); Here I could specify DriverManager.getConnection("jdbc:hive2://localhost:10000/mydb", "hiveuser", "hivepassword"); but now I am using spark , so I am doing SparkSession spark = SparkSession .builder() .appName("Java Spark SQL basic example") .enableHiveSupport() .config("spark.sql.warehouse.dir", "hdfs://saurab:9000/user/hive/warehouse") .config("hive.metastore.warehouse.dir", "hdfs://saurab:9000/user/hive/warehouse") .master("local") .getOrCreate(); I see no configuration to specify database name. That's why I need to explicitly address database if I want to do any CRUD So how does spark connect to hive ? ... View more

I wanted to run hive queries through jdbc, but I am getting <code>java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask Then I looked nodemanager log. Here are some key notes to consider <code>1)Container container_1499666177243_0001_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURERESULT=FAILURE 2)DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE And here is complete stack trace <code>2017-07-10 11:41:34,149 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1499666177243_0001_02_000001 and exit code: 1 ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) at org.apache.hadoop.util.Shell.run(Shell.java:869) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1499666177243_0001_02_000001 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:869) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2017-07-10 11:41:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2017-07-10 11:41:34,153 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2017-07-10 11:41:34,153 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:748) 2017-07-10 11:41:34,153 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1 2017-07-10 11:41:34,156 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0001_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2017-07-10 11:41:34,156 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1499666177243_0001_02_000001 2017-07-10 11:41:34,199 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=saurab OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1499666177243_0001 CONTAINERID=container_1499666177243_0001_02_000001 2017-07-10 11:41:34,200 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0001/container_1499666177243_0001_02_000001 2017-07-10 11:41:34,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0001_02_000001 transitioned from EXITED_WITH_FAILURE to DONE 2017-07-10 11:41:34,203 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1499666177243_0001_02_000001 from application application_1499666177243_0001 2017-07-10 11:41:34,204 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1499666177243_0001_02_000001 2017-07-10 11:41:34,204 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1499666177243_0001 2017-07-10 11:41:35,208 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1499666177243_0001_02_000001] 2017-07-10 11:41:35,209 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1499666177243_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2017-07-10 11:41:35,210 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0001 2017-07-10 11:41:35,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1499666177243_0001 2017-07-10 11:41:35,211 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1499666177243_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2017-07-10 11:41:35,211 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1499666177243_0001, with delay of 10800 seconds 2017-07-10 11:43:26,431 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1499666177243_0002_000002 (auth:SIMPLE) 2017-07-10 11:43:26,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1499666177243_0002_02_000001 by user saurab 2017-07-10 11:43:26,438 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1499666177243_0002 2017-07-10 11:43:26,439 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=saurab IP=10.10.10.149 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1499666177243_0002 CONTAINERID=container_1499666177243_0002_02_000001 2017-07-10 11:43:26,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1499666177243_0002 transitioned from NEW to INITING 2017-07-10 11:43:26,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1499666177243_0002_02_000001 to application application_1499666177243_0002 2017-07-10 11:43:26,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1499666177243_0002 transitioned from INITING to RUNNING 2017-07-10 11:43:26,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0002_02_000001 transitioned from NEW to LOCALIZING 2017-07-10 11:43:26,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1499666177243_0002 2017-07-10 11:43:26,441 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1499666177243_0002 2017-07-10 11:43:26,442 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle 2017-07-10 11:43:26,442 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1499666177243_0002 2017-07-10 11:43:26,444 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez-conf.pb transitioned from INIT to DOWNLOADING 2017-07-10 11:43:26,444 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez.session.local-resources.pb transitioned from INIT to DOWNLOADING 2017-07-10 11:43:26,446 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1499666177243_0002_02_000001 2017-07-10 11:43:26,448 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/nmPrivate/container_1499666177243_0002_02_000001.tokens. Credentials list: 2017-07-10 11:43:26,449 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user saurab 2017-07-10 11:43:26,450 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/nmPrivate/container_1499666177243_0002_02_000001.tokens to /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/container_1499666177243_0002_02_000001.tokens 2017-07-10 11:43:26,450 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002 = file:/home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002 2017-07-10 11:43:26,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez-conf.pb(->/home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/filecache/10/tez-conf.pb) transitioned from DOWNLOADING to LOCALIZED 2017-07-10 11:43:26,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://saurab:9000/tmp/hive/saurab/_tez_session_dir/fed51831-bf68-45b0-abea-11fb2b007c2f/.tez/application_1499666177243_0002/tez.session.local-resources.pb(->/home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/filecache/11/tez.session.local-resources.pb) transitioned from DOWNLOADING to LOCALIZED 2017-07-10 11:43:26,676 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0002_02_000001 transitioned from LOCALIZING to LOCALIZED 2017-07-10 11:43:26,715 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0002_02_000001 transitioned from LOCALIZED to RUNNING 2017-07-10 11:43:26,715 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1499666177243_0002_02_000001 2017-07-10 11:43:26,718 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/container_1499666177243_0002_02_000001/default_container_executor.sh] 2017-07-10 11:43:26,868 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1499666177243_0002_02_000001 is : 1 2017-07-10 11:43:26,868 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1499666177243_0002_02_000001 and exit code: 1 ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) at org.apache.hadoop.util.Shell.run(Shell.java:869) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1499666177243_0002_02_000001 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1: 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:869) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:748) 2017-07-10 11:43:26,868 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0002_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE 2017-07-10 11:43:26,868 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1499666177243_0002_02_000001 2017-07-10 11:43:26,898 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002/container_1499666177243_0002_02_000001 2017-07-10 11:43:26,899 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=saurab OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1499666177243_0002 CONTAINERID=container_1499666177243_0002_02_000001 2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1499666177243_0002_02_000001 transitioned from EXITED_WITH_FAILURE to DONE 2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Removing container_1499666177243_0002_02_000001 from application application_1499666177243_0002 2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1499666177243_0002_02_000001 2017-07-10 11:43:26,900 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1499666177243_0002 2017-07-10 11:43:27,904 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1499666177243_0002_02_000001] 2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1499666177243_0002 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /home/saurab/hadoopec/hadoop/tmp/hadoop-tmp-dir/nm-local-dir/usercache/saurab/appcache/application_1499666177243_0002 2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1499666177243_0002 2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1499666177243_0002 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2017-07-10 11:43:27,905 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1499666177243_0002, with delay of 10800 seconds Surprisingly this error only comes up when I SET hive.execution.engine=tez;, It works fine with SET hive.execution.engine=mr ... View more