Member since
04-25-2016
579
Posts
609
Kudos Received
111
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2197 | 02-12-2020 03:17 PM | |
1537 | 08-10-2017 09:42 AM | |
10747 | 07-28-2017 03:57 AM | |
2460 | 07-19-2017 02:43 AM | |
1843 | 07-13-2017 11:42 AM |
02-14-2020
09:02 AM
@Ellyly the NPE you got is side effect of Exception you have in stage-3/4/5, can you please upload the Error stack you got before the NPE, I am assuming that you are running hive on tez? I think ReExecOverlayPlugin failing with NPE and shallowing the exception at client-side (fix for this has been identified as Hive-19204 but you should be able to see the real cause in HS2 logs
... View more
02-12-2020
03:17 PM
I think error log is pretty informative, getGroups call for $Superuser does not get any group associated with the user, you might need to run "hdfs groups" command on the HMS node for this user if that the case and fix the env issue.
... View more
02-12-2020
12:50 PM
@Ellyly can you please post the complete stack trace from the HiveServer2 logs?
... View more
12-14-2017
09:45 AM
1 Kudo
if you are running query on tez then this will be useful to follow https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html
... View more
08-10-2017
09:42 AM
add following in connection string tez.queue.name=<queue_name>
... View more
08-02-2017
08:15 AM
7 Kudos
Creating and running Temporary functions are discouraged while running a query on LLAP because of security reason since many users are sharing same instances of LLAP, it can create a conflict but still, you can create temp functions using add jar and hive.llap.execution.mode=auto. with exclusive llap execution mode(hive.llap.execution.mode=only) you will run into the ClassNotFoundException, hive.llap.execution.mode=auto will allow some part of query(map tasks) to run in the tez container. Here are steps to create a custom permanent function in LLAP(steps are tested on HDP-260) 1. create a jar for UDF function (in this case I am using simple udf): git clone https://github.com/rajkrrsingh/SampleCode
mvn clean package 2. upload the target/SampleCode.jar to the node where HSI is running(in my case I have copied it to /tmp directory) 3. add jar to hive_aux_jars (goto Ambari--> hive --> config --> hive-interactive-env template) export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:/tmp/SampleCode.jar
4. add the jar to Auxillary JAR list (goto Ambari--> hive --> config --> Auxillary JAR list) Auxillary JAR list=/tmp/SampleCode.jar 5. restart LLAP 6. create Permanent Custom function connect to HSI using beeline
create FUNCTION CustomLength as 'com.rajkrrsingh.hiveudf.CustomLength';
describe function CustomLength;
select CustomLength(description) from sample_07 limit 1;
7. check where the SampleCode.jar localized root@hdp26 container_e06_1501140901077_0019_01_000002]# pwd
/hadoop/yarn/local/usercache/hive/appcache/application_1501140901077_0019/container_e06_1501140901077_0019_01_000002
[root@hdp26 container_e06_1501140901077_0019_01_000002]# find . -iname sample*
./app/install/lib/SampleCode.jar
... View more
Labels:
07-28-2017
03:57 AM
@mungeol heo you can change the logging level of hiveserver2 using ambari, go to hive--> config --> Advanced hive-log4j --> replace hive.root.logger=INFO,DRFA with hive.root.logger=DEBUG,DRFA and restart hiveserver2
... View more
07-19-2017
02:43 AM
@Gagan Brahmi modify tez.am.resource.memory.mb in ambari tez configuration, restart tez and hive client.
... View more
07-13-2017
11:42 AM
1 Kudo
@Anurag Mishra you need to run following command, it will return json output,you need to parse json output and get the required details curl --compressed -X GET http://hdp26:8088/ws/v1/cluster/apps?state=RUNNING
{"apps":{"app":[{"id":"application_1499066924486_0010","user":"hive","name":"HIVE-971ad1b1-b8c8-425a-a0df-ad50f6cb4798","queue":"llap","state":"RUNNING","finalStatus":"UNDEFINED","progress":0.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://hdp26.hdp.local:8088/proxy/application_1499066924486_0010/","diagnostics":"","clusterId":1499066924486,"applicationType":"TEZ","applicationTags":"","priority":0,"startedTime":1499750778877,"finishedTime":0,"elapsedTime":190496298,"amContainerLogs":"http://hdp26.hdp.local:8042/node/containerlogs/container_e05_1499066924486_0010_01_000001/hive","amHostHttpAddress":"hdp26.hdp.local:8042","allocatedMB":1024,"allocatedVCores":1,"runningContainers":1,"memorySeconds":195067899,"vcoreSeconds":190495,"queueUsagePercentage":16.666668,"clusterUsagePercentage":10.0,"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"resourceRequests":[{"capability":{"memory":1024,"memorySize":1024,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":0},"relaxLocality":true,"resourceName":"*"}],"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":""},{"id":"application_1499066924486_0009","user":"hive","name":"llap0","queue":"llap","state":"RUNNING","finalStatus":"UNDEFINED","progress":100.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://hdp26.hdp.local:8088/proxy/application_1499066924486_0009/","diagnostics":"","clusterId":1499066924486,"applicationType":"org-apache-slider","applicationTags":"name: llap,version: 11jul2017,description: llap is a daemon service that works with a cache and works on sql constructs.","priority":1,"startedTime":1499750705675,"finishedTime":0,"elapsedTime":190569500,"amContainerLogs":"http://hdp26.hdp.local:8042/node/containerlogs/container_e05_1499066924486_0009_01_000001/hive","amHostHttpAddress":"hdp26.hdp.local:8042","allocatedMB":4096,"allocatedVCores":2,"runningContainers":2,"memorySeconds":780521363,"vcoreSeconds":381124,"queueUsagePercentage":66.66667,"clusterUsagePercentage":40.0,"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"resourceRequests":[{"capability":{"memory":512,"memorySize":512,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":0},"relaxLocality":true,"resourceName":"*"},{"capability":{"memory":3584,"memorySize":3584,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":1},"relaxLocality":true,"resourceName":"hdp26.hdp.local"},{"capability":{"memory":3584,"memorySize":3584,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":1},"relaxLocality":false,"resourceName":"/default-rack"},{"capability":{"memory":3584,"memorySize":3584,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":1},"relaxLocality":false,"resourceName":"*"}],"logAggregationStatus":"RUNNING","unmanagedApplication":false,"amNodeLabelExpression":""}]}}
... View more
07-11-2017
02:22 AM
np @Tomomichi Hirano feel free to accept best answer in this discussion thread so that other user can get benefit from it.
... View more