Member since
04-25-2016
579
Posts
609
Kudos Received
111
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1312 | 02-12-2020 03:17 PM | |
1001 | 08-10-2017 09:42 AM | |
8016 | 07-28-2017 03:57 AM | |
1502 | 07-19-2017 02:43 AM | |
1157 | 07-13-2017 11:42 AM |
02-14-2020
09:02 AM
@Ellyly the NPE you got is side effect of Exception you have in stage-3/4/5, can you please upload the Error stack you got before the NPE, I am assuming that you are running hive on tez? I think ReExecOverlayPlugin failing with NPE and shallowing the exception at client-side (fix for this has been identified as Hive-19204 but you should be able to see the real cause in HS2 logs
... View more
02-12-2020
03:17 PM
I think error log is pretty informative, getGroups call for $Superuser does not get any group associated with the user, you might need to run "hdfs groups" command on the HMS node for this user if that the case and fix the env issue.
... View more
02-12-2020
12:50 PM
@Ellyly can you please post the complete stack trace from the HiveServer2 logs?
... View more
12-14-2017
09:45 AM
1 Kudo
if you are running query on tez then this will be useful to follow https://community.hortonworks.com/articles/22419/hive-on-tez-performance-tuning-determining-reducer.html
... View more
08-10-2017
09:42 AM
add following in connection string tez.queue.name=<queue_name>
... View more
08-02-2017
08:15 AM
7 Kudos
Creating and running Temporary functions are discouraged while running a query on LLAP because of security reason since many users are sharing same instances of LLAP, it can create a conflict but still, you can create temp functions using add jar and hive.llap.execution.mode=auto. with exclusive llap execution mode(hive.llap.execution.mode=only) you will run into the ClassNotFoundException, hive.llap.execution.mode=auto will allow some part of query(map tasks) to run in the tez container. Here are steps to create a custom permanent function in LLAP(steps are tested on HDP-260) 1. create a jar for UDF function (in this case I am using simple udf): git clone https://github.com/rajkrrsingh/SampleCode
mvn clean package 2. upload the target/SampleCode.jar to the node where HSI is running(in my case I have copied it to /tmp directory) 3. add jar to hive_aux_jars (goto Ambari--> hive --> config --> hive-interactive-env template) export HIVE_AUX_JARS_PATH=$HIVE_AUX_JARS_PATH:/tmp/SampleCode.jar
4. add the jar to Auxillary JAR list (goto Ambari--> hive --> config --> Auxillary JAR list) Auxillary JAR list=/tmp/SampleCode.jar 5. restart LLAP 6. create Permanent Custom function connect to HSI using beeline
create FUNCTION CustomLength as 'com.rajkrrsingh.hiveudf.CustomLength';
describe function CustomLength;
select CustomLength(description) from sample_07 limit 1;
7. check where the SampleCode.jar localized root@hdp26 container_e06_1501140901077_0019_01_000002]# pwd
/hadoop/yarn/local/usercache/hive/appcache/application_1501140901077_0019/container_e06_1501140901077_0019_01_000002
[root@hdp26 container_e06_1501140901077_0019_01_000002]# find . -iname sample*
./app/install/lib/SampleCode.jar
... View more
- Find more articles tagged with:
- Data Processing
- Hive
- hive-udf
- hiveserver2
- How-ToTutorial
- llap
Labels:
07-28-2017
10:27 AM
@Anurag Mishra this is sample curl request to get information of zookeeper service, hope this will serve your purpose curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTERNAME>/services/ZOOKEEPER
... View more
07-28-2017
04:04 AM
@suresh krish check if HSI is up and running and make sure you have nothing wrong with zookeeper.
... View more
07-28-2017
03:57 AM
@mungeol heo you can change the logging level of hiveserver2 using ambari, go to hive--> config --> Advanced hive-log4j --> replace hive.root.logger=INFO,DRFA with hive.root.logger=DEBUG,DRFA and restart hiveserver2
... View more
07-19-2017
06:29 AM
hive locking will effect on non-transcational table as well while these setting turned on.
... View more
07-19-2017
05:44 AM
@sundaramoorthy muthusamy it will depend on whether you have enabled concurrency or not (hive.support.concurrency), with concurrency enabled and txn manager set as DbTxManager, while the query is running it will take a write lock on table and once you fire alter table set partition location, this DDL statement will wait for query to complete and release lock to perform alter operation or will time out (controlled by hive.txn.timeout)
... View more
07-19-2017
02:43 AM
@Gagan Brahmi modify tez.am.resource.memory.mb in ambari tez configuration, restart tez and hive client.
... View more
07-17-2017
01:38 PM
@suresh krish it seems that it is taking PtyCryptoCodec from some configuration, you need to check from where it is taking this configuration, run hive -e 'set -v;' > hive.out and grep codec in hive.out to know from where it is picking the PtyCryptoCodec
... View more
07-17-2017
10:50 AM
@sathish jeganathan can you get the output of following hive --hiveconf hive.root.logger=DEBUG,console > hivecli.out 2>&1
... View more
07-17-2017
08:49 AM
@suresh krish what the value for key io.compression.codecs in hdfs config using ambari?
... View more
07-17-2017
07:57 AM
@suresh krish can you add the output of following command hive --hiveconf hive.root.logger=DEBUG,console -e "show databases;"
... View more
07-17-2017
07:37 AM
@suresh krish then Atlas hook is not an issue here, do you find any exception/error in metastore logs?
... View more
07-17-2017
07:31 AM
@sathish jeganathan could you please post hivecli.out after running following command hive > hivecli.out 2>&1
... View more
07-17-2017
07:22 AM
@suresh krish can you check what the value for the hive.exec.post.hooks in hive config using ambari?
... View more
07-17-2017
07:15 AM
@wbu no, to achieve HA on the same protocol, you need to register similar protocol hiveserver2 with zookeeper.
... View more
07-16-2017
10:12 AM
@heta desai yes you can, query hive using hive-jdbc driver( sample - https://github.com/rajkrrsingh/HiveServer2JDBCSample) the same way what you do with normal RDBMS, get the resultset and display using javascript/jquery on the presentation layer.
... View more
07-16-2017
07:40 AM
@wbu
looking at the source code https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L141 https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L145 if you register 4 hiveserver2 instances (2 with binary while 2 with http mode) client can connect to any of this one hiveserver2 running with any protocol. so its not a HA between two similar protocol
... View more
07-13-2017
11:42 AM
1 Kudo
@Anurag Mishra you need to run following command, it will return json output,you need to parse json output and get the required details curl --compressed -X GET http://hdp26:8088/ws/v1/cluster/apps?state=RUNNING
{"apps":{"app":[{"id":"application_1499066924486_0010","user":"hive","name":"HIVE-971ad1b1-b8c8-425a-a0df-ad50f6cb4798","queue":"llap","state":"RUNNING","finalStatus":"UNDEFINED","progress":0.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://hdp26.hdp.local:8088/proxy/application_1499066924486_0010/","diagnostics":"","clusterId":1499066924486,"applicationType":"TEZ","applicationTags":"","priority":0,"startedTime":1499750778877,"finishedTime":0,"elapsedTime":190496298,"amContainerLogs":"http://hdp26.hdp.local:8042/node/containerlogs/container_e05_1499066924486_0010_01_000001/hive","amHostHttpAddress":"hdp26.hdp.local:8042","allocatedMB":1024,"allocatedVCores":1,"runningContainers":1,"memorySeconds":195067899,"vcoreSeconds":190495,"queueUsagePercentage":16.666668,"clusterUsagePercentage":10.0,"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"resourceRequests":[{"capability":{"memory":1024,"memorySize":1024,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":0},"relaxLocality":true,"resourceName":"*"}],"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":""},{"id":"application_1499066924486_0009","user":"hive","name":"llap0","queue":"llap","state":"RUNNING","finalStatus":"UNDEFINED","progress":100.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://hdp26.hdp.local:8088/proxy/application_1499066924486_0009/","diagnostics":"","clusterId":1499066924486,"applicationType":"org-apache-slider","applicationTags":"name: llap,version: 11jul2017,description: llap is a daemon service that works with a cache and works on sql constructs.","priority":1,"startedTime":1499750705675,"finishedTime":0,"elapsedTime":190569500,"amContainerLogs":"http://hdp26.hdp.local:8042/node/containerlogs/container_e05_1499066924486_0009_01_000001/hive","amHostHttpAddress":"hdp26.hdp.local:8042","allocatedMB":4096,"allocatedVCores":2,"runningContainers":2,"memorySeconds":780521363,"vcoreSeconds":381124,"queueUsagePercentage":66.66667,"clusterUsagePercentage":40.0,"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"resourceRequests":[{"capability":{"memory":512,"memorySize":512,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":0},"relaxLocality":true,"resourceName":"*"},{"capability":{"memory":3584,"memorySize":3584,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":1},"relaxLocality":true,"resourceName":"hdp26.hdp.local"},{"capability":{"memory":3584,"memorySize":3584,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":1},"relaxLocality":false,"resourceName":"/default-rack"},{"capability":{"memory":3584,"memorySize":3584,"virtualCores":1},"nodeLabelExpression":"","numContainers":0,"priority":{"priority":1},"relaxLocality":false,"resourceName":"*"}],"logAggregationStatus":"RUNNING","unmanagedApplication":false,"amNodeLabelExpression":""}]}}
... View more
07-13-2017
10:28 AM
No this will fetch you RUNNING application from single cluster where you execute it, if you want to monitor remote cluster for RUNNING job you can use YARN RM rest API curl --compressed -X GET http://<cluster1_RM_HOST>:8088/ws/v1/cluster/apps?state=RUNNING
curl --compressed -X GET http://<cluster2_RM_HOST>:8088/ws/v1/cluster/apps?state=RUNNING
... View more
07-13-2017
09:51 AM
try this on yarn cluster, it will fetch you all running application on cluster yarn application -list -appStates RUNNING
... View more
07-13-2017
09:49 AM
did you miss anything in stacktrace, do you have any error/exception around this exception stating why Initializer fail?
... View more
07-11-2017
02:22 AM
np @Tomomichi Hirano feel free to accept best answer in this discussion thread so that other user can get benefit from it.
... View more
07-10-2017
03:20 PM
Did I understand you right: is the NodeManager not used if there is no DataNode installed on the same host? this is not right understanding, NM will be used whether DN is on same host or not, as your installation has slave node where DN is running, if you install NM on these nodes then yarn will only spin container on these nodes, not the master node.
... View more
07-10-2017
03:11 PM
you can uninstall nodemanager from master node and colocate it with the datanodes, this way you can stop yarn to spin container on master node and since it is running on similar slave node where datanode is running so you need not to worry about the data locality.
... View more
07-10-2017
10:03 AM
@Tomomichi Hirano ya you can initially go with enabling ranger at hiveserver2 level for now,hiveserver2 will expect that you have read/write permission for hive user on file/folder on hdfs, Additionally, if you install ranger plugin for HDFS then it will be good for you so that you need not mess with hdfs acls and with ranger you can manage at a single place.
... View more