Member since
05-27-2016
22
Posts
1
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3116 | 12-23-2020 12:05 AM | |
| 3057 | 12-18-2020 05:57 AM |
12-23-2020
12:05 AM
LLAP internally uses MAP-REDUCE shuffle handler because inside TEZ code we have a hard-coded constant TEZ_SHUFFLE_HANDLER_SERVICE_ID = "mapreduce_shuffle". https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/dag/api/TezConstants.java So, when we are using tez_shuffle with LLAP its giving ‘NullpointerException’.
... View more
12-18-2020
05:57 AM
why don't you write 20 insert into query in append mode in the same sql file ';' separated, In this case you don't have to use union. And probably won't encounter heap space issue.
... View more
12-18-2020
05:49 AM
After Setup for the Tez Shuffle Handler following instructions on : https://tez.apache.org/shuffle-handler.html I'm getting below error in query execution: Vertex failed, vertexName=Map 2, vertexId=vertex_1608273679503_0002_2_01, diagnostics=[Task failed, taskId=task_1608273679503_0002_2_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1608273679503_0002_2_01_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:363) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266) ... 15 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Async Initialization failed. abortRequested=false at org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:461) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:395) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:568) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:520) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:381) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:335) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:118) at org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:143) ... 4 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:113) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:331) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:400) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:207) at org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:116) ... 5 more Caused by: java.lang.NullPointerException at org.apache.tez.runtime.api.impl.TezTaskContextImpl.getServiceConsumerMetaData(TezTaskContextImpl.java:190) at org.apache.tez.runtime.library.common.shuffle.impl.ShuffleManager.(ShuffleManager.java:264) at org.apache.tez.runtime.library.input.UnorderedKVInput.start(UnorderedKVInput.java:146) at org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:109) ... 9 more Can anyone help here please?
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez
10-30-2020
12:09 AM
Thank you! This was helpful.
... View more
10-28-2020
12:00 PM
I want to set logging level of hiveserver2Interactive.log to DEBUG in HDP-3.1. could anyone help here?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
01-24-2017
11:35 AM
@Kuldeep Kulkarni, Yes it was a problem with my rpm command, that's why yum hung. I rebuild rpmdb and everything working fine now. thanks a ton. 🙂
... View more
01-24-2017
06:00 AM
@Kuldeep Kulkarni
I encountered similar issue with HDFS_CLIENT, I've successfully deleted the client using curl call as mentioned in your post.
But, Installation of HDFS_CLIENT stuck at:
2017-01-24 08:44:43,873 - Package['unzip'] {}
And installation failed with following error message:
stderr: /var/lib/ambari-agent/data/errors-2631.txt
Python script has been killed due to timeout after waiting 1800 secs Could you please suggest me some pointers?
... View more
10-04-2016
09:34 AM
After performing the mentioned steps i'm still facing the same issue. The alert is same: “Connection failed: [Errno 111] Connection refused to localhost:6667” curl -u admin:admin -H 'X-Requested-By: ambari' -X GET "http://localhost:8080/api/v1/clusters/XXXX/alert_definitions/17"
{
"href" : "http://localhost:8080/api/v1/clusters/flumetest/alert_definitions/17",
"AlertDefinition" : {
"cluster_name" : "XXXX",
"component_name" : "KAFKA_BROKER",
"description" : "This host-level alert is triggered if the Kafka Broker cannot be determined to be up.",
"enabled" : true,
"id" : 17,
"ignore_host" : false,
"interval" : 1,
"label" : "Kafka Broker Process",
"name" : "kafka_broker_process",
"scope" : "HOST",
"service_name" : "KAFKA",
"source" : {
"default_port" : 9092.0,
"reporting" : {
"critical" : {
"value" : 5.0,
"text" : "Connection failed: {0} to {1}:{2}"
},
"warning" : {
"value" : 1.5,
"text" : "TCP OK - {0:.3f}s response on port {1}"
},
"ok" : {
"text" : "TCP OK - {0:.3f}s response on port {1}"
}
},
"type" : "PORT",
"uri" : "{{kafka-broker/listeners}}"
}
}
}
After changing the listener port in config to : 9092 I'm getting: Connection failed: [Errno 111] Connection refused to localhost:9092
... View more