I am trying to store the pig output by using
elephant bird LzoJsonStorage()
But it didn't worked.
sample:-
{"in_reply_to_user_id_str":null,"coordinates":null,"text":"\u0627\u0627\u0627\u0627\u0627\u062d \u0627\u0644\u0627\u062c\u0648\u0627\u0621 \u0628\u062a\u0627\u0639\u062a \u0633\u0643\u0633 \u062d\u0627\u0627\u0627\u0627\u0627\u0631\u0631 \u0645\u0646\u0648 \u0627\u0644\u0641\u062d\u0644 \u0627\u0644\u0644\u064a \u064a\u0628\u064a \u0627\u0633\u0648\u064a \u0644\u0647 \u0641\u0648\u0644\u0648 \u064a\u0633\u0648\u064a \u0631\u062a\u0648\u064a\u062a","created_at":"Thu Apr 12 17:38:47 +0000 2012","favorited":false,"contributors":null,"in_reply_to_screen_name":null,"source":"\u003Ca href=\"http:\/\/blackberry.com\/twitter\" rel=\"nofollow\"\u003ETwitter for BlackBerry\u00ae\u003C\/a\u003E","retweet_count":0,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"id_str":"190494185374220289","entities":{"hashtags":[],"user_mentions":[],"urls":[]},"geo":null,"retweeted":false,"place":null,"truncated":false,"in_reply_to_status_id_str":null,"user":{"created_at":"Tue Apr 10 11:43:10 +0000 2012","notifications":null,"profile_use_background_image":true,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","url":null,"contributors_enabled":false,"geo_enabled":false,"profile_text_color":"333333","followers_count":5,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2084863527\/Screen-120409-224940_normal.jpg","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2084863527\/Screen-120409-224940_normal.jpg","listed_count":0,"profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","description":"\u0627\u0628\u064a \u0633\u0643\u0633 \u0631\u0631\u0648\u0639\u0647 \u0645\u0639 \u0641\u062d\u0644 ","screen_name":"Ga7bah_sex","profile_link_color":"0084B4","location":"\u0627\u0631\u0636 \u0627\u0644\u0633\u0643\u0633","default_profile":true,"show_all_inline_media":false,"is_translator":false,"statuses_count":5,"profile_background_color":"C0DEED","id_str":"550121247","follow_request_sent":null,"lang":"ar","profile_background_tile":false,"protected":false,"profile_sidebar_fill_color":"DDEEF6","name":"\u0642\u062d\u0628\u0647 \u0648\u0627\u0628\u064a \u0632\u0628 ","default_profile_image":false,"time_zone":null,"friends_count":8,"id":550121247,"following":null,"verified":false,"utc_offset":null,"favourites_count":0,"profile_sidebar_border_color":"C0DEED"},"id":190494185374220289}Script:-
REGISTER elephant-bird-core-4.1.jar
REGISTER elephant-bird-hadoop-compat-4.1.jar
REGISTER elephant-bird-pig-4.1.jar
REGISTER json-simple-1.1.1.jar
REGISTER google-collections-1.0.jar
REGISTER hadoop-lzo-0.4.14.jar
REGISTER piggybank-0.12.0.jar
twitter = LOAD 'sample.json' USING com.twitter.elephantbird.pig.load.JsonLoader();
extracted = foreach twitter generate (chararray)$0#'created_at' as created_at,(chararray)$0#'id' as id,(chararray)$0#'id_str' as id_str,(chararray)$0#'text' as text,(chararray)$0#'source' as source;
STORE extracted into 'tweets' using com.twitter.elephantbird.pig.store.LzoJsonStorage();
Error While trying to store in tweets:-
java.lang.Exception: java.lang.RuntimeException: native-lzo library not available
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.RuntimeException: native-lzo library not available
at com.hadoop.compression.lzo.LzoCodec.createCompressor(LzoCodec.java:165)
at com.hadoop.compression.lzo.LzopCodec.createOutputStream(LzopCodec.java:50)
at com.twitter.elephantbird.util.LzoUtils.getIndexedLzoOutputStream(LzoUtils.java:75)
at com.twitter.elephantbird.mapreduce.output.LzoTextOutputFormat.getRecordWriter(LzoTextOutputFormat.java:24)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:81)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at
By the error I guess lzo library is not available.
Please suggest me how can I resolve this.
Mohan.V
