Member since
02-02-2016
583
Posts
518
Kudos Received
98
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4896 | 09-16-2016 11:56 AM | |
| 1974 | 09-13-2016 08:47 PM | |
| 8007 | 09-06-2016 11:00 AM | |
| 4860 | 08-05-2016 11:51 AM | |
| 6736 | 08-03-2016 02:58 PM |
03-21-2016
10:46 AM
1 Kudo
Hoping you have completed all the pre-requisites to run spark on Mesos, however please follow below if you haven't done yet. http://spark.apache.org/docs/latest/running-on-mesos.html#connecting-spark-to-mesos Regarding spark + Mesos and Tableau connection, I believe you need a SparkSql thrift server so that Tableau can directly connect to the thrift port. Morever you can start your thrift server like below. $SPARK_HOME/sbin/start-thriftserver.sh --master mesos://host:port --deploy-mode cluster --executor-memory 5G Note: You also need spark ODBC driver at Tableau client side to connect to the Thrift server, you can download it from Here
... View more
03-17-2016
05:02 AM
1 Kudo
Yes, Please follow these steps and let me know if you still face same issue. Also kindly mention your HDP version. hive> set orc.compress.size=4096; hive> set hive.exec.orc.default.stripe.size=268435456; hive> your create table DDL; hive> load data query in orc table; hive> you select query;
... View more
03-16-2016
02:36 PM
4 Kudos
Looks like your table has ORC format so can you please try to set below properties and try it again? set orc.compress.size=4096 set hive.exec.orc.default.stripe.size=268435456
... View more
03-16-2016
01:30 PM
@David Tam The same conf's should work for local mode also, initially it made for YARN only then later it applicable for local mode also. As I said earlier that it's better to try it on spark 1.6 version. Please refer this Jira and it's Pull requests :- https://issues.apache.org/jira/browse/SPARK-11821
... View more
03-16-2016
07:06 AM
3 Kudos
In general, Zookeeper doesn't actually required huge drives because it will only store metadata information for many services, I have seen customer using 100G to 250G of partition size for zookeeper data directory and logs which is fine of many cluster deployment. Moreover administrator need to set configuration for automatic purging policy of snapshots and logs directories so that we don't end up by filling all the local storage. Please refer below doc for more info. http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
... View more
03-16-2016
05:30 AM
2 Kudos
Did you tired to use below properties as command line parameter while running spark-submit? Also there were few issues related to spark kerberos in spark 1.4 and 1.5 so its better to try this on spark 1.6 release. --principal <principle name> --keytab /etc/security/keytabs/spark.keytab
... View more
03-15-2016
10:36 AM
1 Kudo
In this case you need to configure two fencing methods and your last method should give success always so that automatic failover can happen successfully. Please refer below link. https://www.packtpub.com/books/content/setting-namenode-ha
... View more
03-14-2016
06:22 AM
1 Kudo
Can you please share your HDP version? Also it looks like this issue is similar to oozie bug https://issues.apache.org/jira/browse/OOZIE-2482.
... View more
03-12-2016
04:51 PM
2 Kudos
Facing this issue while running Oozie hive action, though same sql query works fine from hive shell. HDP version : 2.3.4.0 Vertex killed, vertexName=Reducer 2, vertexId=vertex_1457421589107_0056_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1457421589107_0056_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
Vertex killed, vertexName=Reducer 3, vertexId=vertex_1457421589107_0056_1_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1457421589107_0056_1_02 [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1457421589107_0056_1_00, diagnostics=[Vertex vertex_1457421589107_0056_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: employee initializer failed, vertex=vertex_1457421589107_0056_1_00 [Map 1], java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:208)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
at org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:88)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
at org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:502)
at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:434)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:305)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:407)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:155)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:255)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:248)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:248)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:235)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:399)
at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:552)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1191)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1158)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
... 28 more
]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1457421589107_0056_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1457421589107_0056_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 3, vertexId=vertex_1457421589107_0056_1_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1457421589107_0056_1_02 [Reducer 3] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:2
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [2]
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Oozie
-
Apache Tez
03-10-2016
05:21 PM
1 Kudo
Right now I can think of 3 possible scenario which might create this issue. 1. Check the datanode logs corresponding to that block id, If there is any issue with the disks then probably you will see some messages like below. In case you can't find these messages then you still need take a look at disks IO WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:200ms (threshold=15ms) OR WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write packet to mirror took 986ms (threshold=300ms) 2. check datanode logs and it's GC pattern because a long GC pause can also cause this issue. 3. If datanode doesn't show any disk and GC issues then the networks latency b/w client and datanode could be a culprit, you can use iperf or other network tool to check latency.
... View more
- « Previous
- Next »