About viswanath_kammu

viswanath_kammu · ‎07-17-2019

How can I move the data from my in premise HDFS cluster to the Azure datalake storage account.I was trying distcp but the command doesnt recognize the azure storage link

viswanath_kammu · ‎07-17-2019

Which version of HDP are you using ?

viswanath_kammu · ‎11-14-2017

@Stefan Kupstaitis-Dunkler I never said I used hive CLI to connect to LLAP. I am connecting through beeline, we can give the username when we connect to LLAP through beeline and make it run through the (end)user through which we login to a beeline session. I have no issues regarding connecting to hiveserver2 or LLAP. I just want someone from hortonworks development team to acknowledge that we cannot have impersonation using LLAP.

viswanath_kammu · ‎11-10-2017

@Stefan Kupstaitis-Dunkler If I do not set hive.server2.enable.doAs=true in hive-interactive section(even if I set hive.server2.enable.doAs=true in hive main config page), It runs all my queries as hive user in LLAP and end user in hive CLI. That means I clearly cannot have impersonation enabled in LLAP until I set hive.server2.enable.doAs=true in hive-interactive section. And as you said if I set it to true it wont make sense from resource point of view. Thus there is clearly a conflict between running LLAP and Impersonation, In other words we cannot have impersonation in LLAP.

viswanath_kammu · ‎11-09-2017

Hello @Stefan Kupstaitis-Dunkler Well. It does not seem to simply ignore it. The property hive.server2.enable.doAs is there in two places in Ambari,One in config-settings page of hive and other in advanced page(which belongs to Hive-Interactive server or LLAP). Yes you are right it ignores it if you set this property for hive,but if you set this property for LLAP it definitely takes effect(I tested by creating a table through LLAP and it creates it with end user). I do not think LLAP daemons are down or crashing. I can monitor all my LLAP daemons and I also checked the status of LLAP,its in RUNNING_ALL state. If I just turn off this property everything runs normal on LLAP. We cannot run any query which starts mappers on LLAP if this property is set. I can connect to LLAP and run simple queries which doesn't start a mapper like select * or create. But if I start any query which requires a mapper it fails with the above error. I am assuming LLAP has a problem sharing its resources with multiple users. The error it is showing seems to be misdirecting. @Stefan Kupstaitis-Dunkler can you please confirm if you are able to run mappers under LLAP by setting this property hive.server2.enable.doAs=true in Advanced hive-interactive site section

viswanath_kammu · ‎11-06-2017

I find that the jobs in LLAP are failing when I enable the property run as end user instead of hive user.I get the below error. [Code: 2, SQL State: 08S01] Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1507231420401_0045_1_00, diagnostics=[Vertex vertex_1507231420401_0045_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68) ... 25 more Caused by: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration Does LLAP have any issue with this property being enabled? I could run the query on LLAP if I run it as hive user.

viswanath_kammu · ‎11-02-2017

Can I run multiple Hive Interactive servers running on different LLAP queues on a cluster(HDP 2.6.2). Right now I am only able to run single Hive interactive server through ambari by enabling hive interactive query. Is there any documentation which shows having multiple Hive LLAP connections on my cluster

viswanath_kammu · ‎10-25-2017

Hi @Abdelkrim Hadjidj I have given 90% of my cluster to LLAP yet I can see the maximum value in the bar to be only 4. What is memory property which you want me to change so that I can have more sessions

viswanath_kammu · ‎10-25-2017

Right now on a 10 node cluster I am able to run only 4 concurrent LLAP sessions. But I want to handle more than 20 concurrent users on LLAP. How can I achieve this ? I tried to increase the property - Max total concurrent queries from 4 to 5(which is not recommended, Ambari page does recommends not use more than 4 concurrent queries). My queries started failing when I set the property to 5 concurrent sessions. Is there any work around for this problem.

viswanath_kammu · ‎10-24-2017

@Bala Vignesh N V I do see sort oder: + in Tez in explain plan. But on Tez it shows it as Map join operator(with sort oder:+ option seen under reduce output operator) and on MR it clearly shows it as Sort Merge Bucket map operator. Can you please confirm I cannot see Sort Merge Bucket map operator in Tez explain plan, the only way I can confirm its SMB join by looking for something like sort order: +. in tez explain plan

viswanath_kammu · ‎10-24-2017

@thussain Yes I tried it on MR and I can clearly see Sort merge bucket map join in explain query. Are you suggesting me that SMB does not work in TEZ ?

viswanath_kammu · ‎10-23-2017

@thussain I still cannot find sort merge join operator.Below are my queries set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat; set hive.enforce.sortmergebucketmapjoin=false; set hive.auto.convert.sortmerge.join=true; set hive.optimize.bucketmapjoin=true; set hive.optimize.bucketmapjoin.sortedmerge=true; set hive.enforce.bucketing=true; set hive.enforce.sorting=true; set hive.auto.convert.join=true; drop table key_value_large; drop table key_value_small; create table key_value_large ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 8 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; create table key_value_small ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 4 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; explain extended select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key It only shows map join

viswanath_kammu · ‎10-23-2017

@icocio I can see SMB join only working on MR mode but not in TEZ mode.Could you see SMB join been shown in the explain plan with those properties ?

viswanath_kammu · ‎10-18-2017

I have two bucketed tables on the same column, and I am trying to see if hive performs SMB join on these tables. But when I do an explain plan it only does a Map join. Does SMB join work in Hive 1.2.1?

viswanath_kammu · ‎10-16-2017

One more thing.When I issue show locks <table> command. I can only see exclusive locks being shown for insert queries but it is not showing shared locks on select queries ?

viswanath_kammu · ‎10-16-2017

I am running a select query with where condition picking the partition which is not been inserted. Now the select query is not supposed to run over all partitions. But still the select query waits for the insert(into static partition) to be finished.

viswanath_kammu · ‎10-16-2017

I am not running the select query on the whole table. I am running select query with where clause filtering a single partition which is not been updated. It still waits for the insert to be finished even if the partition its inserting is not an existing one.

viswanath_kammu · ‎10-13-2017

I am doing an insert overwrite on a partition(static) which takes a few minutes to finish. I want to still query the same table with select command when am doing the insert. But the select query does not start running until the insert query finishes . I know the insert query gets an exclusive lock on just the partition I am writing to .I am trying to figure out why the select query is not able to just read all other partitions on which it can acquire a lock on

viswanath_kammu · ‎10-12-2017

Insert overwite query on LLAP is not working but it runs through normal Hive. The same query works through hive but fails through LLAP throwing this error Vertex failed, vertexName=Map 1, vertexId=vertex_1507670392589_0073_1_00, diagnostics=» Vertex vertex_1507670392589_0073_1_00» Map 1 killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:70) at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89) at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:151) at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148) at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121) at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4031) at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:204) at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2855) at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2802) at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2784) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59) at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1925) at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203) at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2290) at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2276) at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184) at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68) ... 25 more Caused by: java.lang.IllegalArgumentException: No running LLAP daemons! Please check LLAP service status and zookeeper configuration at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92) at org.apache.hadoop.hive.ql.exec.tez.Utils.getSplitLocationProvider(Utils.java:47) at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.(HiveSplitGenerator.java:121) ... 30 more Vertex killed, vertexName=Reducer 2, vertexId=vertex_1507670392589_0073_1_01, diagnostics=» Vertex received Kill in NEW state., Vertex vertex_1507670392589_0073_1_01» Reducer 2 killed/failed due to:OTHER_VERTEX_FAILURE DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

viswanath_kammu · ‎10-02-2017

I can actually see the query(ACID tables) running on LLAP, but it too slow compared to Hive query.Hive takes around 30sec and LLAP is taking around 14500secs to finish. Does LLAP have an higher penalty over hive

viswanath_kammu · ‎09-29-2017

I am running a query which has an acid transaction table and I have also enabled acid transaction properties.When I run this query in the LLAP,The query does not run and it automatically is killing LLAP. Does LLAP does not support acid transactions ?

viswanath_kammu · ‎07-10-2017

@Sindhu @Raja Sudhan I am not sure if you can perform SMB join in TEZ.I could clearly see SMB join in explain plan ,when I was running it on MR but it wasn't showing up in tez. You can find my query below https://community.hortonworks.com/questions/107180/is-smb-join-or-smb-map-join-enabled-in-tez.html#comment-108854

viswanath_kammu · ‎06-28-2017

I want to store a table in hive sorted on a individual column. The table has got partitions, I know I can save the table sorted on a column by using clustered by and sort by. But that would create buckets. I still want to store the table sorted on the column with out bucketing. Is this possible in hive?

viswanath_kammu · ‎06-15-2017

You can see the plan below

viswanath_kammu · ‎06-14-2017

And this is the explain for TEZ

viswanath_kammu · ‎06-14-2017

@Sindhu This is Explain for MR. The Query is explain select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key; And I also had to do set hive.enforce.sortmergebucketmapjoin=false; just for MR

viswanath_kammu · ‎06-11-2017

The conversation of a join to SMB seems to be depending up on the execution engine. If I run the below commands on using MR set hive.execution.engine=mr; set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat; set hive.auto.convert.sortmerge.join=true; set hive.optimize.bucketmapjoin=true; set hive.optimize.bucketmapjoin.sortedmerge=true; set hive.enforce.bucketing=true; set hive.enforce.sorting=true; set hive.auto.convert.join=true; drop table key_value_large; drop table key_value_small; create table key_value_large ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 8 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; create table key_value_small ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 4 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; explain extended select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key I can see a 'Sorted Merge Bucket Map Join Operator' in the explain statement,But If I set the execution engine to TEZ. set hive.execution.engine=tez; And then run the same explain plan I get to see 'Map Join Operator' instead of SMB map join in the plan. I could see in some of JIRA pages that SMB is not implemented in TEZ http://mail-archives.apache.org/mod_mbox/hive-user/201508.mbox/%3c4D4BDAE9-F6A8-456F-A90A-A550D3C2898B@gmail.com%3e Can someone if TEZ can run SMB join.

viswanath_kammu · ‎06-11-2017

Try set hive.convert.join.bucket.mapjoin.tez=true; Then you cans see Map Join gets converted to Bucket Map join. If you set hive.enforce.sortmergebucketmapjoin=false; in MR then you will get through with that error

viswanath_kammu · ‎06-09-2017

Does these configuration mentioned in this page work on TEZ engine .I could see SMB working only on MR

viswanath_kammu · ‎06-09-2017

The conversation of a join to SMB seems to be depending up on the execution engine. If I run the below commands on using MR set hive.execution.engine=mr; set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat; set hive.auto.convert.sortmerge.join=true; set hive.optimize.bucketmapjoin=true; set hive.optimize.bucketmapjoin.sortedmerge=true; set hive.enforce.bucketing=true; set hive.enforce.sorting=true; set hive.auto.convert.join=true; drop table key_value_large; drop table key_value_small; create table key_value_large ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 8 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; create table key_value_small ( key int, value string ) partitioned by (ds string) CLUSTERED BY (key) SORTED BY (key ASC) INTO 4 BUCKETS ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; explain extended select count(*) from key_value_large a JOIN key_value_small b ON a.key = b.key I can see a 'Sorted Merge Bucket Map Join Operator' in the explain statement,But If I set the execution engine to TEZ. set hive.execution.engine=tez; And then run the same explain plan I get to see 'Map Join Operator' instead of SMB map join in the plan. Could anyone confirm if they could could get SMB map join in tez execution engine.

Online	Offline
Last Visited	‎07-17-2019 12:53 PM

Date Registered	‎02-16-2017 07:35 PM
Last Visited	‎07-17-2019 12:53 PM
Total Messages Posted	45
Total Kudos Received	2

Migrating data from local HDP HDFS to Azure HDInsi...

Re: Can we create multiple Hive LLAP instances or ...

Re: Hive LLAP has conflict with Impresonation (run...

Re: Hive LLAP has conflict with Impresonation (run...

Re: Hive LLAP has conflict with Impresonation (run...

Hive LLAP has conflict with Impresonation (running...

Can we create multiple Hive LLAP instances or Hive...

Re: How to increase LLAP concurent sessions

How to increase LLAP concurent sessions

Re: Can we see SMB join in explain query using Tez...

Re: Can we see SMB join in explain query using Tez...

Re: Can we see SMB join in explain query using Tez...

Re: Can we see SMB join in explain query using Tez...

Can we see SMB join in explain query using Tez and...

Re: Reading a Hive table while doing insert overwr...

Re: Reading a Hive table while doing insert overwr...

Re: Reading a Hive table while doing insert overwr...

Reading a Hive table while doing insert overwrite ...

Insert overwite query on LLAP is not working but i...

Re: Does Hive LLAP support acid transaction tables

Does Hive LLAP support acid transaction tables

Re: How to properly perform SMB (Sort Merge Bucket...

Can we create a sorted table in Hive without bucke...

Re: Is SMB Join or SMB Map join Enabled in TEZ

Re: Is SMB Join or SMB Map join Enabled in TEZ

Re: Is SMB Join or SMB Map join Enabled in TEZ

Is SMB Join or SMB Map join Enabled in TEZ

Re: Unable to see Sorted Merge Bucket Join Operato...

Re: What is Sort Merge Bucket (SMB) Join in Hive? ...

Re: Unable to see Sorted Merge Bucket Join Operato...