Member since
07-18-2019
9
Posts
0
Kudos Received
0
Solutions
10-25-2019
11:05 AM
I am working on Big DATA using Hadoop. I want to work on Hadoop with Python. For this purpose, I randomly searched a lot of sites for books, articles and did not find any book or article. I am using anaconda for python and HDP 2.4 for Hadoop. Can anyone suggest me some books on Hive with python and Hbase with Python?
... View more
Labels:
10-22-2019
11:37 PM
I want to perform join operation on three long table in Hbase? explain with example.
... View more
Labels:
10-17-2019
10:35 AM
I want to store the hive query result to a file in JSON format. Through Brickhouse jar, I can get the query output in JSON format but am unable to store that in a file or table. The query I'm trying is given below. When the INSERT OVERWRITE query runs, it gives an error; how can I solve this error? Is there any way to store query results in JSON format through queries? Query: add jar hdfs:///mydir/brickhouse-0.7.1.jar;
INSERT OVERWRITE DIRECTORY '/mydir/textfile1'
stored as textfile
SELECT to_json( named_struct( "id",id,
"name",name))
FROM link_tbl; Error: INFO : Tez session hasn't been created yet. Opening session
INFO : Dag name: INSERT OVERWRITE DIRECTORY '/mydir/text...pl(Stage-1)
INFO :
INFO : Status: Running (Executing on YARN cluster with App id application_1571318954298_0001)
INFO : Map 1: -/-
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1571318954298_0001_1_00, diagnostics=[Vertex vertex_1571318954298_0001_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:70)
at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:151)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)
at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)
at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4536)
at org.apache.tez.dag.app.dag.impl.VertexImpl.access$4300(VertexImpl.java:202)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:3352)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3301)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3282)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1862)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:201)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1978)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1964)
at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
... 25 more
Caused by: java.lang.RuntimeException: Failed to load plan: hdfs://sandbox.hortonworks.com:8020/tmp/hive/hive/2eaf13cf-1f98-4a2d-8f76-4e9c839f355b/hive_2019-10-17_13-33-05_763_197979924455130156-2/hive/_tez_scratch_dir/d9d1df72-f68c-4c1f-b642-85a46f32a79f/map.xml: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 19963874, Size: 113
Serialization trace:
_mainHash (org.codehaus.jackson.sym.BytesToNameCanonicalizer)
_rootByteSymbols (org.codehaus.jackson.JsonFactory)
jsonFactory (brickhouse.udf.json.ToJsonUDF)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:472)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:311)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:101)
... 30 more
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 19963874, Size: 113
Serialization trace:
_mainHash (org.codehaus.jackson.sym.BytesToNameCanonicalizer)
_rootByteSymbols (org.codehaus.jackson.JsonFactory)
jsonFactory (brickhouse.udf.json.ToJsonUDF)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1173)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1062)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:1076)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:432)
... 32 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 19963874, Size: 113
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at org.apache.hive.com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:820)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:743)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
... 65 more
]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
... View more
07-23-2019
07:13 PM
@Predrag Minovic After running the PS this is the output. Is Zookeeeper running ? if not how can I run zookeeper?
... View more
07-22-2019
07:13 AM
@Vladimir Zlatkin form where we get these dependencies in jar files?
... View more
07-20-2019
04:55 AM
My Apachi Ambari 2.2.1.0 version and HBase client 2.2.0 are same now. but still getting same error Not a host:port pair: PBUF. Can you explain in detail how can I resolve it?
... View more
07-19-2019
04:19 PM
@Josh Elser How can we check HBase cluster version?
... View more
07-19-2019
12:17 PM
Please anybody guide ?
... View more
07-19-2019
05:44 AM
I am trying to connect Java api to Hbase, I am using Ambari for Hbase and netbean for java. I am facing an error Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF # sandbox.hortonworks.com�}�����-. libraries of Hbase-client-0.95.0.jar and Hbase-server.0.95.0.jar have same version. please guide how can I resolve ? this is a code that I am using for connection. Configuration conf = HBaseConfiguration.create(); conf.set("hbase.master", "192.168.23.143:16020"); conf.set("hbase.zookeeper.quorum","192.168.23.143"); conf.set("hbase.zookeeper.property.clientPort", "2181"); conf.set("zookeeper.znode.parent", "/hbase-unsecure"); System.out.println("Connecting ..."); HBaseAdmin admin = new HBaseAdmin(conf); HBaseAdmin.checkHBaseAvailable(conf); System.out.println("Connecting ..."); boolean bool = admin.tableExists("transport"); System.out.println( bool); } Error: run: Connecting ... 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.1-942149, built on 05/07/2010 17:14 GMT 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:host.name=DESKTOP-959ROGC 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_192 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.home=C:\Program Files\Java\jdk1.8.0_192\jre 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.class.path=E:\NetBeanProject\HBaseConnection\jar_file\commons-beanutils-1.7.0.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-beanutils-core-1.8.0.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-cli-1.2.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-codec-1.4.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-collections-3.2.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-configuration-1.6.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-digester-1.8.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-el-1.0.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-httpclient-3.0.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-io-2.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-lang-2.4.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-logging-1.0.3.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-logging-1.1.2.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-math-2.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\commons-net-1.4.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\core-3.1.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\hadoop-core-1.1.2.jar;E:\NetBeanProject\HBaseConnection\jar_file\hadoop-core-1.2.1.jar;E:\NetBeanProject\HBaseConnection\jar_file\hbase-0.94.10.jar;E:\NetBeanProject\HBaseConnection\jar_file\hbase-client-0.95.0.jar;E:\NetBeanProject\HBaseConnection\jar_file\hbase-server-0.95.0.jar;E:\NetBeanProject\HBaseConnection\jar_file\hsqldb-1.8.0.10.jar;E:\NetBeanProject\HBaseConnection\jar_file\log4j-1.2.16.jar;E:\NetBeanProject\HBaseConnection\jar_file\servlet-api-2.5-20081211.jar;E:\NetBeanProject\HBaseConnection\jar_file\servlet-api-2.5-6.1.14.jar;E:\NetBeanProject\HBaseConnection\jar_file\zookeeper.jar;E:\NetBeanProject\HBaseConnection\build\classes 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.library.path=C:\Program Files\Java\jdk1.8.0_192\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Java\bin;C:\Java;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Users\sidra\AppData\Local\Microsoft\WindowsApps;;. 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=C:\Users\sidra\AppData\Local\Temp\ 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:os.name=Windows 10 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:os.version=10.0 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:user.name=sidra 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:user.home=C:\Users\sidra 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Client environment:user.dir=E:\NetBeanProject\HBaseConnection 19/07/18 21:31:32 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.23.143:2181 sessionTimeout=180000 watcher=hconnection 19/07/18 21:31:32 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 17072@DESKTOP-959ROGC 19/07/18 21:31:32 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.23.143:2181 19/07/18 21:31:39 INFO zookeeper.ClientCnxn: Socket connection established to 192.168.23.143/192.168.23.143:2181, initiating session 19/07/18 21:31:39 INFO zookeeper.ClientCnxn: Session establishment complete on server 192.168.23.143/192.168.23.143:2181, sessionid = 0x16c04d6c5dc002e, negotiated timeout = 40000 Exception in thread "main" java.lang.IllegalArgumentException: Not a host:port pair: PBUF # sandbox.hortonworks.com�}�����-
... View more
Labels:
