Working on a very common problem of refreshing reference tables in hive using wipe and load methodology. Want to know if there is a way to refresh the data without killing all read locks (queries reading data) or waiting for all read to finish to acquire exclusive lock. Use case is, some queries may run for few hours and the refresh job needs to be run in between. Is there a way, all the running queries will use current snapshot of the data and refresh job should complete without waiting. New queries should get the new data set. These tables are not partitioned. I was thinking about using ACID but it doen't support merge yet or value from other table for UPDATE statement. ... View more

Assuming most of the users will be running hive queries to access the data. Is it possible to control resource Utilization by different queues in a multi-tenant environment when hive doAs is set to false. Since all the hive queries will be running as hive user and in one queue, does setting up multiple queues to control resource utilization make any sense? I read this http://hortonworks.com/blog/best-practices-for-hive-authorization-using-apache-ranger-in-hdp-2-2/, but this makes me believe set doAs=true and handle authorization through both Ranger Hive+Hdfs policies NOT only Hive policies. And we can't even use "SQL Standard Authorization" if need to setup multiple queues to control resource utilization. Am I missing anything here. ... View more

I am getting below error when trying to run Pig using tez. Any pointer will be appreciated. The same script run successfully if I am using MR as execution engine. I checked the tez.lib.uris and it is correctly set to hdfs tez.tar.gz file location. Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.VerifyError: class org.apache.tez.dag.api.records.DAGProtos$DAGPlan overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) ... View more

I am trying to run below query in hive using TEZ and it is failing with NullPointerException whereas same query is running fine using MR execution engine. We are using hdp 2.3.2 select count(*) from (select serial_number from hive_demo.gdwi_test group by serial_number) q; select serial_number from hive_demo.gdwi_test group by serial_number; Both these queries are throwing NullPointerException. Vertex failed, vertexName=Map 1, vertexId=vertex_1468116141308_1525_2_00, diagnostics=[Vertex vertex_1468116141308_1525_2_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: gdwi_test initializer failed, vertex=vertex_1468116141308_1525_2_00 [Map 1], java.lang.NullPointerException at org.apache.hadoop.io.Text.encode(Text.java:450) at org.apache.hadoop.io.Text.encode(Text.java:431) at org.apache.hadoop.io.Text.writeString(Text.java:480) at org.apache.hadoop.mapred.split.TezGroupedSplit.write(TezGroupedSplit.java:101) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.createSplitProto(MRInputHelpers.java:249) at org.apache.tez.mapreduce.hadoop.InputSplitInfoMem.createSplitsProto(InputSplitInfoMem.java:168) at org.apache.tez.mapreduce.hadoop.InputSplitInfoMem.getSplitsProto(InputSplitInfoMem.java:117) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.createEventList(HiveSplitGenerator.java:200) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:180) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:246) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:240) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:240) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:227) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more