question Re: top function in pig/hive in Archives of Support Questions (Read Only)

top function in pig/hive

priyankavijayak — Fri, 05 Feb 2016 04:46:07 GMT

In a dataset (approx. 2 lakh records), there is coloumn named tags ( comma separated list of tags associated with question. examples of tags are "html","error" etc so on .

php,error,gd,image-processing

lisp,scheme,subjective,clojure

cocoa-touch,objective-c,design-patterns

core-animation

django,django-models

aspÃ»net

scala,pattern-matching,oop,object-oriented-design,design-principles

. . . . .

how to find top 10 most commonly used tags in dataset? in pig or hive

Re: top function in pig/hive

aervits — Fri, 05 Feb 2016 04:50:30 GMT

@priyanka vijayakumar good word count tutorial link. It uses Pig, Hcatalog and Hive, you will be better off with the combination of these.

Re: top function in pig/hive

pminovic — Fri, 05 Feb 2016 05:51:16 GMT

Here is a Pig word count with comments. Give the delimiter to TOKENIZE, in you case comma: TOKENIZE(line,','). You might have to select a different filter based on your input. You can start by commenting the filter out and adding it later if needed. Finally, to extract only 10 top entries you can use LIMIT: top10 = LIMIT ordered_word_count, 10. Be sure to inspect the stored file and make sure words (tags) have been properly tokenized. If not, add a filter mentioned above.

Re: top function in pig/hive

priyankavijayak — Sat, 06 Feb 2016 07:44:23 GMT

thanks a lot.

Re: top function in pig/hive

priyankavijayak — Sat, 06 Feb 2016 07:44:33 GMT

thanks a lot.