Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pig Json import to HBase error -TableOutputFormat InvocationTargetException and ERROR 1002: Unable to store alias

Pig Json import to HBase error -TableOutputFormat InvocationTargetException and ERROR 1002: Unable to store alias

New Contributor

Hi

I am trying to process a XML file on pig and make an Json output, my initial idea to import json directly to HBase but seems its not supporting. Then with elephant bird, I converted the Json for HBase ready.

But I see below errors. Environment is single node and like;

Pig: 0.16.0

HBase: 1.1.5

Zookeeper: 3.4.6

Hadoop: 2.6.4

Ubuntu 12.04 32 bit

2016-07-27 15:47:29,606 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0xf057b40x0, quorum=localhost:2181, baseZNode=/hbase
2016-07-27 15:47:29,709 [main] ERROR org.apache.hadoop.hbase.mapreduce.TableOutputFormat - java.io.IOException: java.lang.reflect.InvocationTargetException
2016-07-27 15:47:29,741 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1002: Unable to store alias T
Details at logfile: /usr/local/Pig/bin/pig_1469627167519.log
grunt> 2016-07-27 15:47:29,772 [main-SendThread(localhost:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-27 15:47:29,774 [main-SendThread(localhost:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
2016-07-27 15:47:29,990 [main-SendThread(localhost:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x156277837ff000e, negotiated timeout = 40000
REGISTER /usr/local/Pig/lib/piggybank.jar
REGISTER /usr/local/Pig/lib/json-simple-1.1.jar
REGISTER /usr/local/Pig/lib/elephant-bird-pig-4.14.jar
REGISTER /usr/local/Pig/lib/elephant-bird-core-4.14.jar
REGISTER /usr/local/Pig/lib/elephant-bird-hadoop-compat-4.14.jar
REGISTER /usr/local/Pig/lib/google-collections-1.0.jar
REGISTER /usr/local/Hbase/lib/zookeeper-3.4.6.jar
REGISTER /usr/local/Hbase/lib/protobuf-java-2.5.0.jar
REGISTER /usr/local/Hbase/lib/hbase-common-1.1.5.jar
REGISTER /usr/local/Hbase/lib/guava-12.0.1.jar
DEFINE XPath org.apache.pig.piggybank.evaluation.xml.XPath();
S = LOAD '/usr/local/hadoop/data/untar/result/final/part-m-00000' USING com.twitter.elephantbird.pig.load.JsonLoader() AS (json: map[]);
T = FOREACH S GENERATE (int)json#'Oam_HardwareRfFailure_Sum' as Oam_HardwareRfFailure_Sum,(int)json#'Oam_HardwareRfTempOut_Sum' as Oam_HardwareRfTempOut_Sum............
STORE T INTO 'hbase://KPIKPI' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('info: Oam_HardwareRfFailure_Sum info: Oam_HardwareRfTempOut_Sum info: VS_NumFreqChanges info:........
4 REPLIES 4

Re: Pig Json import to HBase error -TableOutputFormat InvocationTargetException and ERROR 1002: Unable to store alias

New Contributor

The problem seems to be at step #2. Are you able to DUMP T? what does the /usr/local/Pig/bin/pig_1469627167519.log says?

Re: Pig Json import to HBase error -TableOutputFormat InvocationTargetException and ERROR 1002: Unable to store alias

New Contributor

Yes I am able to dump T (now P) the values are like below and the fields I stated are also present in the beginning of the dump log.

One of the duplicate lines below is not being sent as an info to Hbase and stated as the KPI_ID

(295_bsrGroupId=643_HNBId=001D4C-1211080200_Fsn=1211080200_bSRName=_manualPscUsed=false_20160701050000+0200,295_bsrGroupId=643_HNBId=001D4C-1211080200_Fsn=1211080200_bSRName=_manualPscUsed=false_20160701050000+0200,0,0,0,8810,24651,42,53,67090,66790,437,437,0,0,0,0,50,51,130,166,0,0,0,0,0,0,4000,0,0,0,161955,28792,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,564,32,0,0,0,35,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-0.0,0,0,0,0,0,0,-91.7,0,0,0,0,0,0,0,0,0,0,0.06895,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,56,0,0,0,0,0,0,5,0,0,0,0.018899,34,1,0.0,0,0,0.0,0,0,0.0,0,0,40,1,0,0,0,0,2,0,0,0,0,0,0.020567,0.0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,10,0,0,10,0,2,0,8,0,10,0,0,0,10,0,10,0,0,0)

From the log, there is not so much information. Both errors still exist.

================================================================================Pig Stack Trace---------------ERROR 1002: Unable to store alias Porg.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to store alias P   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1778)   at org.apache.pig.PigServer.registerQuery(PigServer.java:707)   at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1075)   at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:505)   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)   at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)   at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)   at org.apache.pig.Main.run(Main.java:564)   at org.apache.pig.Main.main(Main.java:176)   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)   at java.lang.reflect.Method.invoke(Method.java:498)   at org.apache.hadoop.util.RunJar.run(RunJar.java:221)   at org.apache.hadoop.util.RunJar.main(RunJar.java:136)Caused by: java.lang.RuntimeException: java.io.IOException: java.lang.reflect.InvocationTargetException   at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:211)   at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:934)   at org.apache.pig.newplan.logical.visitor.InputOutputFileValidatorVisitor.visit(InputOutputFileValidatorVisitor.java:69)   at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)   at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)   at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)   at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)   at org.apache.pig.newplan.logical.relational.LogicalPlan.validate(LogicalPlan.java:212)   at org.apache.pig.PigServer$Graph.compile(PigServer.java:1851)   at org.apache.pig.PigServer$Graph.access$300(PigServer.java:1527)   at org.apache.pig.PigServer.execute(PigServer.java:1440)   at org.apache.pig.PigServer.access$500(PigServer.java:118)   at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1773)   ... 14 moreCaused by: java.io.IOException: java.lang.reflect.InvocationTargetException   at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:459)   at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:436)   at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:317)   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:198)   at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:160)   at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)   ... 28 moreCaused by: java.lang.reflect.InvocationTargetException   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)   at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457)   ... 33 moreCaused by: java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace   at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)   at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)   at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)   at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83)   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:907)   at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:701)   ... 38 more================================================================================

Re: Pig Json import to HBase error -TableOutputFormat InvocationTargetException and ERROR 1002: Unable to store alias

Mentor
@Erdal Kucuk

looks like you're missing libraries in your Pig script java.lang.NoClassDefFoundError: org/cloudera/htrace/Trace

You have a custom stack that we do not support with HDP

HBase 1.1.2 vs. 1.1.5

Hadoop 2.7.1 vs Hadoop 2.6.4

Pig 0.15 vs Pig 0.16

and addition of ElephantBird. I would try to go through your error stack and try to satisfy all of your dependencies but in general you can achieve what you're trying to do w/out ElephantBird.

1. Use Nifi to read XML, convert to JSON and output to HBase.

2. Use Mapreduce to convert XML to JSON and output to HBase https://acadgild.com/blog/quick-way-convert-xml/ writing to HBase is well described in detail on the internet. Here's my non-MR example using HBase API https://github.com/dbist/workshops/blob/master/hbase/HBaseJsonLoad/src/main/java/com/hortonworks/hba...

3. Use built-in XMLLoader in Pig https://pig.apache.org/docs/r0.16.0/api/org/apache/pig/piggybank/storage/XMLLoader.html, convert to JSON and output to HBase using HBaseStorageHandler.

Highlighted

Re: Pig Json import to HBase error -TableOutputFormat InvocationTargetException and ERROR 1002: Unable to store alias

New Contributor

Hi Artem,

I used XML Loader easily and converted it to Json format using JsonStorage and made some changes. My data looks like this, but its longer (243 inputs)

{"SERIAL":"5450406299"},{"VS_MeanNumCSCall":"0"},{"VS_CPICHEcN0Dist1":"0"},....,{"KPI_DATE":"20160630"}

But at last part I wasn't successful to import this json to HBase anyhow and also with HBaseStorageHandler. I'm having similar errors as above. I don't know which jars are missing.

Don't have an account?
Coming from Hortonworks? Activate your account here