Member since
02-16-2017
45
Posts
2
Kudos Received
0
Solutions
07-17-2019
11:05 AM
Which version of HDP are you using ?
... View more
11-14-2017
03:47 PM
1 Kudo
@Stefan Kupstaitis-Dunkler I never said I used hive CLI to connect to LLAP. I am connecting through beeline, we can give the username when we connect to LLAP through beeline and make it run through the (end)user through which we login to a beeline session. I have no issues regarding connecting to hiveserver2 or LLAP. I just want someone from hortonworks development team to acknowledge that we cannot have impersonation using LLAP.
... View more
11-10-2017
07:52 PM
@Stefan Kupstaitis-Dunkler If I do not set hive.server2.enable.doAs=true in hive-interactive section(even if I set hive.server2.enable.doAs=true in hive main config page), It runs all my queries as hive user in LLAP and end user in hive CLI. That means I clearly cannot have impersonation enabled in LLAP until I set hive.server2.enable.doAs=true in hive-interactive section. And as you said if I set it to true it wont make sense from resource point of view. Thus there is clearly a conflict between running LLAP and Impersonation, In other words we cannot have impersonation in LLAP.
... View more
11-09-2017
06:53 PM
Hello @Stefan Kupstaitis-Dunkler Well. It does not seem to simply ignore it. The property hive.server2.enable.doAs is there in two places in Ambari,One in config-settings page of hive and other in advanced page(which belongs to Hive-Interactive server or LLAP). Yes you are right it ignores it if you set this property for hive,but if you set this property for LLAP it definitely takes effect(I tested by creating a table through LLAP and it creates it with end user). I do not think LLAP daemons are down or crashing. I can monitor all my LLAP daemons and I also checked the status of LLAP,its in RUNNING_ALL state. If I just turn off this property everything runs normal on LLAP. We cannot run any query which starts mappers on LLAP if this property is set. I can connect to LLAP and run simple queries which doesn't start a mapper like select * or create. But if I start any query which requires a mapper it fails with the above error. I am assuming LLAP has a problem sharing its resources with multiple users. The error it is showing seems to be misdirecting. @Stefan Kupstaitis-Dunkler can you please confirm if you are able to run mappers under LLAP by setting this property hive.server2.enable.doAs=true in Advanced hive-interactive site section
... View more
11-06-2017
05:35 PM
I find that the jobs in LLAP are failing when I enable the property run as end user instead of hive user.I get the below error. [Code: 2, SQL
State: 08S01] Error while processing statement: FAILED: Execution Error,
return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
vertexName=Map 1, vertexId=vertex_1507231420401_0045_1_00, diagnostics=[Vertex
vertex_1507231420401_0045_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail
to create InputInitializerManager,
org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with
1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator Caused by:
java.lang.reflect.InvocationTargetException at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at
java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68) ... 25 more Caused by: java.lang.IllegalArgumentException:
No running LLAP daemons! Please check LLAP service status and zookeeper
configuration Does LLAP have any issue with this property being enabled? I could run the query on LLAP if I run it as hive user.
... View more
Labels:
- Labels:
-
Apache Hive
11-02-2017
04:42 PM
1 Kudo
Can I run multiple Hive Interactive servers running on different LLAP queues on a cluster(HDP 2.6.2). Right now I am only able to run single Hive interactive server through ambari by enabling hive interactive query. Is there any documentation which shows having multiple Hive LLAP connections on my cluster
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
10-25-2017
08:09 PM
Hi @Abdelkrim Hadjidj I have given 90% of my cluster to LLAP yet I can see the maximum value in the bar to be only 4. What is memory property which you want me to change so that I can have more sessions
... View more
10-25-2017
03:26 PM
Right now on a 10 node cluster I am able to run only 4 concurrent LLAP sessions. But I want to handle more than 20 concurrent users on LLAP. How can I achieve this ? I tried to increase the property - Max total concurrent queries from 4 to 5(which is not recommended, Ambari page does recommends not use more than 4 concurrent queries). My queries started failing when I set the property to 5 concurrent sessions. Is there any work around for this problem.
... View more
Labels:
- Labels:
-
Apache Hive
10-24-2017
07:12 PM
@Bala Vignesh N V I do see sort oder: + in Tez in explain plan. But on Tez it shows it as Map join operator(with sort oder:+ option seen under reduce output operator) and on MR it clearly shows it as Sort Merge Bucket map operator. Can you please confirm I cannot see Sort Merge Bucket map operator in Tez explain plan, the only way I can confirm its SMB join by looking for something like sort order: +. in tez explain plan
... View more
10-24-2017
02:31 PM
@thussain Yes I tried it on MR and I can clearly see Sort merge bucket map join in explain query. Are you suggesting me that SMB does not work in TEZ ?
... View more
10-23-2017
07:35 PM
@thussain I still cannot find sort merge join operator.Below are my queries set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat; set hive.enforce.sortmergebucketmapjoin=false; set hive.auto.convert.sortmerge.join=true; set hive.optimize.bucketmapjoin=true; set hive.optimize.bucketmapjoin.sortedmerge=true; set hive.enforce.bucketing=true; set hive.enforce.sorting=true; set hive.auto.convert.join=true; drop table key_value_large; drop table key_value_small; create table key_value_large ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 8 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; create table key_value_small ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 4 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; explain extended select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key It only shows map join
... View more
10-23-2017
07:26 PM
@icocio I can see SMB join only working on MR mode but not in TEZ mode.Could you see SMB join been shown in the explain plan with those properties ?
... View more
10-18-2017
03:07 PM
I have two bucketed tables on the same column, and I am trying to see if hive performs SMB join on these tables. But when I do an explain plan it only does a Map join. Does SMB join work in Hive 1.2.1?
... View more
Labels:
- Labels:
-
Apache Hive
10-16-2017
08:14 PM
One more thing.When I issue show locks <table> command. I can only see exclusive locks being shown for insert queries but it is not showing shared locks on select queries ?
... View more
10-16-2017
05:37 PM
I am running a select query with where condition picking the partition which is not been inserted. Now the select query is not supposed to run over all partitions. But still the select query waits for the insert(into static partition) to be finished.
... View more
10-16-2017
05:26 PM
I am not running the select query on the whole table. I am running select query with where clause filtering a single partition which is not been updated. It still waits for the insert to be finished even if the partition its inserting is not an existing one.
... View more
10-13-2017
03:28 PM
I am doing an insert overwrite on a partition(static) which takes a few minutes to finish. I want to still query the same table with select command when am doing the insert. But the select query does not start running until the insert query finishes . I know the insert query gets an exclusive lock on just the partition I am writing to .I am trying to figure out why the select query is not able to just read all other partitions on which it can acquire a lock on
... View more
Labels:
- Labels:
-
Apache Hive
10-12-2017
09:10 PM
Insert overwite query on LLAP is not working but it runs through normal Hive. The same query works through hive but fails through LLAP throwing this error Vertex failed, vertexName=Map 1, vertexId=vertex_1507670392589_0073_1_00, diagnostics=» Vertex vertex_1507670392589_0073_1_00» Map 1
killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:70)
at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:151)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4031)
at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:204)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2855)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2802)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2784)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1925)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2290)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2276)
at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
... 25 more
Caused by: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:47)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:121)
... 30 more
Vertex killed, vertexName=Reducer 2, vertexId=vertex_1507670392589_0073_1_01, diagnostics=» Vertex received Kill in NEW state., Vertex vertex_1507670392589_0073_1_01» Reducer 2
killed/failed due to:OTHER_VERTEX_FAILURE
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
... View more
Labels:
- Labels:
-
Apache Hive
10-02-2017
04:28 PM
I can actually see the query(ACID tables) running on LLAP, but it too slow compared to Hive query.Hive takes around 30sec and LLAP is taking around 14500secs to finish. Does LLAP have an higher penalty over hive
... View more
09-29-2017
05:37 PM
I am running a query which has an acid transaction table and I have also enabled acid transaction properties.When I run this query in the LLAP,The query does not run and it automatically is killing LLAP. Does LLAP does not support acid transactions ?
... View more
Labels:
- Labels:
-
Apache Hive
07-10-2017
05:17 PM
@Sindhu @Raja Sudhan I am not sure if you can perform SMB join in TEZ.I could clearly see SMB join in explain plan ,when I was running it on MR but it wasn't showing up in tez. You can find my query below https://community.hortonworks.com/questions/107180/is-smb-join-or-smb-map-join-enabled-in-tez.html#comment-108854
... View more
06-15-2017
02:19 PM
You can see the plan below
... View more
06-14-2017
02:55 PM
And this is the explain for TEZ
... View more
06-14-2017
02:54 PM
@Sindhu This is Explain for MR. The Query is explain select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key; And I also had to do set hive.enforce.sortmergebucketmapjoin=false; just for MR
... View more
06-11-2017
03:46 PM
The conversation of a join to SMB seems to be depending up on the execution engine. If I run the below commands on using MR set hive.execution.engine=mr; set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat; set hive.auto.convert.sortmerge.join=true; set hive.optimize.bucketmapjoin=true; set hive.optimize.bucketmapjoin.sortedmerge=true; set hive.enforce.bucketing=true; set hive.enforce.sorting=true; set hive.auto.convert.join=true; drop table key_value_large; drop table key_value_small; create table key_value_large ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 8 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; create table key_value_small ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 4 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; explain extended select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key I can see a 'Sorted Merge Bucket Map Join Operator' in the explain statement,But If I set the execution engine to TEZ. set hive.execution.engine=tez; And then run the same explain plan I get to see 'Map Join Operator' instead of SMB map join in the plan. I could see in some of JIRA pages that SMB is not implemented in TEZ http://mail-archives.apache.org/mod_mbox/hive-user/201508.mbox/%3c4D4BDAE9-F6A8-456F-A90A-A550D3C2898B@gmail.com%3e Can someone if TEZ can run SMB join.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez
06-11-2017
03:32 PM
Try set
hive.convert.join.bucket.mapjoin.tez=true; Then you cans see Map Join gets converted to Bucket Map join. If you set hive.enforce.sortmergebucketmapjoin=false; in MR then you will get through with that error
... View more
06-09-2017
03:11 PM
Does these configuration mentioned in this page work on TEZ engine .I could see SMB working only on MR
... View more
06-09-2017
02:40 PM
The conversation of a join to SMB seems to be depending up on the execution engine. If I run the below commands on using MR set hive.execution.engine=mr; set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
set hive.auto.convert.sortmerge.join=true;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.enforce.bucketing=true;
set hive.enforce.sorting=true;
set hive.auto.convert.join=true; drop table key_value_large;
drop table key_value_small; create table key_value_large (
key int,
value string
)
partitioned by (ds string)
CLUSTERED BY (key) SORTED BY (key ASC) INTO 8 BUCKETS
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; create table key_value_small (
key int,
value string
)
partitioned by (ds string)
CLUSTERED BY (key) SORTED BY (key ASC) INTO 4 BUCKETS
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; explain extended select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key I can see a 'Sorted Merge Bucket Map Join Operator' in the explain statement,But If I set the execution engine to TEZ. set hive.execution.engine=tez; And then run the same explain plan I get to see 'Map Join Operator' instead of SMB map join in the plan. Could anyone confirm if they could could get SMB map join in tez execution engine.
... View more
06-09-2017
02:34 PM
Yes, But the explain plan on the query shows that its just a 'Map Join Operator' instead of 'Sorted Merge Bucket Map Join Operator' . So effectively its neither converting SMB to map-join SMB, its just doing map join instead of SMB or SMB map join
... View more
06-07-2017
07:52 PM
When I run a query on LLAP the DAG execution time is less then sum of (Compile time+ Query Plan time + Submit DAG + Start DAG time). Is there any way to reduce the LLAP compilation time and query plan time INFO : Compile Query 2.10s
INFO : Prepare Plan 1.10s
INFO : Submit Plan 0.22s
INFO : Start DAG 0.22s
INFO : Run DAG 3.33s
... View more
Labels:
- Labels:
-
Apache Hive