Hey all, I have a column in hive that xml data, however the xml format is not static, it changes based on a category column and there are alot. I wanted to know if anybody knows if there is a way to programmaitcally take out the tag and the value in the xml column and store it in map column. Best thing I found is xpath in hive however I need to declare what I am taking out. I can do it with a script via java/scala/python however I would just be connecting to Hive via Jdbc and pulling the data out and doing the work client side. And Unfortunately we do not have spark which I think would be able to do it in. Any ideas? ... View more

I’m trying to use pig to do a group by and count distinct on a dataset and I am getting a java error saying “java.lang.ClassCastException: org.apache.pig.data.SingleTupleBag cannot be cast to org.apache.pig.data.” Example Dataset A B ids flag status foo f 1001 1 K foo f 1001 1 K foo c 1002 1 H bar g 1001 1 J bar g 1002 P bar g 1003 1 L Here is an example of my code testtable = LOAD 'landing.testtable' USING org.apache.hive.hcatalog.pig.HCatLoader; filtertable = filter testtable by flag != ' ' AND status != ‘P'; grpcount = FOREACH (GROUP filtertable by (A, B)) { uniqueids = Distinct(filtertable.ids); GENERATE group.A As A_group, group.B As B_group, COUNT(uniqueids) AS id_count; } STORE grpcount INTO 'landing.grpcount USING org.apache.hive.hcatalog.pig.HCatStorer(); This is where I get the error “java.lang.ClassCastException: org.apache.pig.data.SingleTupleBag cannot be cast to org.apache.pig.data.” Not exactly sure what’s wrong here (hive table is properly setup with the right datatypes as well). I assume its erroring out on grpcount but Im not exactly sure why. But I am basically trying to duplicate this SQL Code in Pig Select A AS A_group, B AS B_group, count(distinct ids) As id_count From landing.testtable Where flag != ' ' And status not in (‘P') Group by A, B; I found this alternate solution https://issues.apache.org/jira/browse/PIG-4515 here but Im not really sure how to Implement it in my code :/. Using Pig .15/Hortonworks 2.2.0 ... View more