Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

mohan221213 — Mon, 12 Sep 2016 13:41:19 GMT

I am trying to store the pig output by using

elephant bird LzoJsonStorage()

But it didn't worked.

sample:-

{"in_reply_to_user_id_str":null,"coordinates":null,"text":"\u0627\u0627\u0627\u0627\u0627\u062d \u0627\u0644\u0627\u062c\u0648\u0627\u0621 \u0628\u062a\u0627\u0639\u062a \u0633\u0643\u0633 \u062d\u0627\u0627\u0627\u0627\u0627\u0631\u0631 \u0645\u0646\u0648 \u0627\u0644\u0641\u062d\u0644 \u0627\u0644\u0644\u064a \u064a\u0628\u064a \u0627\u0633\u0648\u064a \u0644\u0647  \u0641\u0648\u0644\u0648 \u064a\u0633\u0648\u064a \u0631\u062a\u0648\u064a\u062a","created_at":"Thu Apr 12 17:38:47 +0000 2012","favorited":false,"contributors":null,"in_reply_to_screen_name":null,"source":"\u003Ca href=\"http:\/\/blackberry.com\/twitter\" rel=\"nofollow\"\u003ETwitter for BlackBerry\u00ae\u003C\/a\u003E","retweet_count":0,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"id_str":"190494185374220289","entities":{"hashtags":[],"user_mentions":[],"urls":[]},"geo":null,"retweeted":false,"place":null,"truncated":false,"in_reply_to_status_id_str":null,"user":{"created_at":"Tue Apr 10 11:43:10 +0000 2012","notifications":null,"profile_use_background_image":true,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","url":null,"contributors_enabled":false,"geo_enabled":false,"profile_text_color":"333333","followers_count":5,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2084863527\/Screen-120409-224940_normal.jpg","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2084863527\/Screen-120409-224940_normal.jpg","listed_count":0,"profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","description":"\u0627\u0628\u064a \u0633\u0643\u0633 \u0631\u0631\u0648\u0639\u0647 \u0645\u0639 \u0641\u062d\u0644 ","screen_name":"Ga7bah_sex","profile_link_color":"0084B4","location":"\u0627\u0631\u0636 \u0627\u0644\u0633\u0643\u0633","default_profile":true,"show_all_inline_media":false,"is_translator":false,"statuses_count":5,"profile_background_color":"C0DEED","id_str":"550121247","follow_request_sent":null,"lang":"ar","profile_background_tile":false,"protected":false,"profile_sidebar_fill_color":"DDEEF6","name":"\u0642\u062d\u0628\u0647 \u0648\u0627\u0628\u064a \u0632\u0628 ","default_profile_image":false,"time_zone":null,"friends_count":8,"id":550121247,"following":null,"verified":false,"utc_offset":null,"favourites_count":0,"profile_sidebar_border_color":"C0DEED"},"id":190494185374220289}

Script:-

REGISTER elephant-bird-core-4.1.jar 

REGISTER elephant-bird-hadoop-compat-4.1.jar 

REGISTER elephant-bird-pig-4.1.jar 

REGISTER json-simple-1.1.1.jar 

REGISTER google-collections-1.0.jar 
REGISTER hadoop-lzo-0.4.14.jar 
REGISTER piggybank-0.12.0.jar
twitter = LOAD 'sample.json' USING com.twitter.elephantbird.pig.load.JsonLoader();
extracted = foreach twitter generate (chararray)$0#'created_at' as created_at,(chararray)$0#'id' as id,(chararray)$0#'id_str' as id_str,(chararray)$0#'text' as text,(chararray)$0#'source' as source;
STORE extracted into 'tweets' using com.twitter.elephantbird.pig.store.LzoJsonStorage();

Error While trying to store in tweets:-

java.lang.Exception: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.createCompressor(LzoCodec.java:165) at com.hadoop.compression.lzo.LzopCodec.createOutputStream(LzopCodec.java:50) at com.twitter.elephantbird.util.LzoUtils.getIndexedLzoOutputStream(LzoUtils.java:75) at com.twitter.elephantbird.mapreduce.output.LzoTextOutputFormat.getRecordWriter(LzoTextOutputFormat.java:24) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:81) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at

By the error I guess lzo library is not available.

Please suggest me how can I resolve this.

Mohan.V

Re: Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

gkeys — Mon, 12 Sep 2016 19:43:53 GMT

Please be sure to follow these instructions

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4-Win/bk_HDP_Install_Win/content/LZOCompression.html

You can do step 3 from the Ambari web UI.

Also, note that steps 1-2 need to be done on each node in the cluster.

Re: Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

mohan221213 — Mon, 12 Sep 2016 20:35:51 GMT

thanks @gkeys.

I have followed the doc, and it worked.

as i said you are the best.:):)

Re: Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

mohan221213 — Mon, 12 Sep 2016 20:38:09 GMT

@gkeys

please try to look in to this.and suggest me where am i missing.

https://community.hortonworks.com/questions/56017/pig-to-elasesticsearch-stringindexoutofboundsexcep.html

question Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR; in Archives of Support Questions (Read Only)

Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

Re: Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

Re: Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;

Re: Pig output STORE using elephantbird.pig.store.LzoJsonStorage ERROR;