Hi All, We are trying to migrate our existing RDBMS(Sql Database) system to hadoop. We are planning to use hbase for the same. But we are not getting how to denormalize sql data to store it in hbase column format. Is it possible? If yes then what would be the best approach for that? Which hbase version is required for this? Any suggestions. ... View more

Hi All, How can we store the output of pig into multiple hbase tables. Hbase tables are already created, need to store the each specific value into specific table. For EX: I have got the output as (word1){data} (word2){data} (word3){data} (word4){data} So I need to store output into already created tables. Table Names are like word1 word2 word3 word4 Now output should be store in already created tables as word1 ----> (word1){data} word2 ----> (word2){data} word3 ----> (word3){data} Any suggestions. thank you. ... View more

Trying to load the json file which is having null values in it by using elephant-bird JsonLoader. sample.json {"created_at":"Mon Aug 22 10:48:23 +0000 2016","id":767674772662607873,"id_str":"767674772662607873","text":"KPIT Image Result for https:\/\/t.co\/Nas2ZnF1zZ... https:\/\/t.co\/9TnelwtIvm","source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":123,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"urls":[{"url":"https:\/\/t.co\/Nas2ZnF1zZ","expanded_url":"http:\/\/miltonious.com\/","display_url":"miltonious.com","indices":[24,47]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1471862903167"} script: REGISTER piggybank.jar REGISTER json-simple-1.1.1.jar REGISTER elephant-bird-pig-4.3.jar REGISTER elephant-bird-core-4.1.jar REGISTER elephant-bird-hadoop-compat-4.3.jar json = LOAD 'sample.json' USING JsonLoader('created_at:chararray, id:chararray, id_str:chararray, text:chararray, source:chararray, in_reply_to_status_id:chararray, in_reply_to_status_id_str:chararray, in_reply_to_user_id:chararray, in_reply_to_user_id_str:chararray, in_reply_to_screen_name:chararray, geo:chararray, coordinates:chararray, place:chararray, contributors:chararray, is_quote_status:bytearray, retweet_count:long, favorite_count:chararray, entities:map[], favorited:bytearray, retweeted:bytearray, possibly_sensitive:bytearray, lang:chararray'); describe json; dump json; When I dump json,I am getting the following output and the worning (Mon Aug 22 10:48:23 +0000 2016,767674772662607873,767674772662607873,google Image Result for Twitter Web Client,false,1234,12345,3214,43215,,,,,,,,,,,,,,) WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigHadoopLogger - org.apache.pig.builtin.JsonLoader(UDF_WARNING_1): Bad record, returning null for {complete json} By warning i guess it is getting NULL values. So how can we load a Json which is having null values in it. And I have tried in another way i.e json = LOAD 'sample.json' USING com.twitter.elephantbird.pig.load.JsonLoader('created_at:chararray, id:chararray, id_str:chararray, text:chararray, source:chararray, in_reply_to_status_id:chararray, in_reply_to_status_id_str:chararray, in_reply_to_user_id:chararray, in_reply_to_user_id_str:chararray, in_reply_to_screen_name:chararray, geo:chararray, coordinates:chararray, place:chararray, contributors:chararray, is_quote_status:bytearray, retweet_count:long, favorite_count:chararray, entities:map[], favorited:bytearray, retweeted:bytearray, possibly_sensitive:bytearray, lang:chararray'); describe json; Output Schema for json unknown. Please suggest me. Thanks. ... View more

Hi all, Sorry for the wrong phrasing of question. I have a scenario where to process the words.t file and data.txt file. words.txt word1 word2 word3 word4 data.txt {"created_at":"18:47:31,Sun Sep 30 2012","text":"RT @Joey7Barton: ..give a word1 about whether the americans wins a Ryder cup. I mean surely he has slightly more important matters. #fami ...","user_id":450990391,"id":252479809098223616} I need to get the output as (word1_epochtime){complete data which matched in text attribute} i.e (word1_1234567890){"created_at":"18:47:31,Sun Sep 30 2012","text":"RT @Joey7Barton: ..give a word1 about whether the americans wins a Ryder cup. I mean surely he has slightly more important matters. #fami ...","user_id":450990391,"id":252479809098223616} I have got the ouput as (word1){"created_at":"18:47:31,Sun Sep 30 2012","text":"RT @Joey7Barton: ..give a word1 about whether the americans wins a Ryder cup. I mean surely he has slightly more important matters. #fami ...","user_id":450990391,"id":252479809098223616} by using this script. 1.load words.txt 2.load data.txt c = cross words,data; d = FILTER c BY (data::text MATCHES CONCAT(CONCAT('.*',words::word),'.*')); e = foreach (group d BY word) {data); and I got the epochtime with the words as <code>time = FOREACH words GENERATE CONCAT(CONCAT(word,'_'),(chararray)ToUnixTime(CurrentTime(created_at))); But I am unable to CONCAT the words with time. How can i get the output as (word1_epochtime){data} Please feel free to suggest me for the above. Mohan.V ... View more

Hi all, I am new to pig and trying to learn on my own. I have written a script to get the epoch time with a word that is reading from words.txt file. Here is the script. words = LOAD 'words.txt' AS word:chararray; B = FOREACH A GENERATE CONCAT(CONCAT(A.word,'_'),(chararray)ToUnixTime(CurrentTime()); dump B; But the issue is, if words.txt file have only one word it is giving proper output. If it is having multiple words like word1 word2 word3 word4 then it is giving the following error ERROR 1066: Unable to open iterator for alias B java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Scalar has more than one row in the output. 1st : (word1 ), 2nd :(word2) (common cause: "JOIN" then "FOREACH ... GENERATE foo.bar" should be "foo::bar" ) at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Scalar has more than one row in the output. 1st : (word1 ), 2nd :(word2) (common cause: "JOIN" then "FOREACH ... GENERATE foo.bar" should be "foo::bar" ) at org.apache.pig.impl.builtin.ReadScalars.exec(ReadScalars.java:122) at o Please suggest me to solve this issue. Thank you. Mohan.V ... View more