Hi All I am trying to get the tweets using nifi and store those into a local file. I have used the following configurations to get the tweets using GetTwitter process Twitter Endpoint :- Filter Endpoint given all twitter keys Languages :- en Terms to Filter On :- facebook,wipro,google Trying to put the tweets into a file using PutFile Configurations that i have used are, Directory:- /root/tweets Conflict Resolution Strategy:- fail Create Missing Directories :-true Maximum File Count :- 1 Last Modified Time :- Permissions:-rw-r--r-- Owner:-root Group:- root I am able to get the tweet but that is ONE per ONE JSON FILE. If I increase the Maximum File Count to 20 then it creates 20 json files and each file contains only one tweet. But I want it to be store all the tweets in single json file. In this https://community.hortonworks.com/questions/42149/using-nifi-to-collect-tweets-into-one-large-file.html they have mentioned to use MergeContent processor. But I didnt get it how to use MergeContent exactly as I am completely new to NIFI. Please sugget me how to use. Do i have to use it after the Putfile or Before PutFile. please help. Mohan.V ... View more

I am trying to store the pig output by using elephant bird LzoJsonStorage() But it didn't worked. sample:- {"in_reply_to_user_id_str":null,"coordinates":null,"text":"\u0627\u0627\u0627\u0627\u0627\u062d \u0627\u0644\u0627\u062c\u0648\u0627\u0621 \u0628\u062a\u0627\u0639\u062a \u0633\u0643\u0633 \u062d\u0627\u0627\u0627\u0627\u0627\u0631\u0631 \u0645\u0646\u0648 \u0627\u0644\u0641\u062d\u0644 \u0627\u0644\u0644\u064a \u064a\u0628\u064a \u0627\u0633\u0648\u064a \u0644\u0647 \u0641\u0648\u0644\u0648 \u064a\u0633\u0648\u064a \u0631\u062a\u0648\u064a\u062a","created_at":"Thu Apr 12 17:38:47 +0000 2012","favorited":false,"contributors":null,"in_reply_to_screen_name":null,"source":"\u003Ca href=\"http:\/\/blackberry.com\/twitter\" rel=\"nofollow\"\u003ETwitter for BlackBerry\u00ae\u003C\/a\u003E","retweet_count":0,"in_reply_to_user_id":null,"in_reply_to_status_id":null,"id_str":"190494185374220289","entities":{"hashtags":[],"user_mentions":[],"urls":[]},"geo":null,"retweeted":false,"place":null,"truncated":false,"in_reply_to_status_id_str":null,"user":{"created_at":"Tue Apr 10 11:43:10 +0000 2012","notifications":null,"profile_use_background_image":true,"profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","url":null,"contributors_enabled":false,"geo_enabled":false,"profile_text_color":"333333","followers_count":5,"profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2084863527\/Screen-120409-224940_normal.jpg","profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2084863527\/Screen-120409-224940_normal.jpg","listed_count":0,"profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","description":"\u0627\u0628\u064a \u0633\u0643\u0633 \u0631\u0631\u0648\u0639\u0647 \u0645\u0639 \u0641\u062d\u0644 ","screen_name":"Ga7bah_sex","profile_link_color":"0084B4","location":"\u0627\u0631\u0636 \u0627\u0644\u0633\u0643\u0633","default_profile":true,"show_all_inline_media":false,"is_translator":false,"statuses_count":5,"profile_background_color":"C0DEED","id_str":"550121247","follow_request_sent":null,"lang":"ar","profile_background_tile":false,"protected":false,"profile_sidebar_fill_color":"DDEEF6","name":"\u0642\u062d\u0628\u0647 \u0648\u0627\u0628\u064a \u0632\u0628 ","default_profile_image":false,"time_zone":null,"friends_count":8,"id":550121247,"following":null,"verified":false,"utc_offset":null,"favourites_count":0,"profile_sidebar_border_color":"C0DEED"},"id":190494185374220289} Script:- REGISTER elephant-bird-core-4.1.jar REGISTER elephant-bird-hadoop-compat-4.1.jar REGISTER elephant-bird-pig-4.1.jar REGISTER json-simple-1.1.1.jar REGISTER google-collections-1.0.jar REGISTER hadoop-lzo-0.4.14.jar REGISTER piggybank-0.12.0.jar twitter = LOAD 'sample.json' USING com.twitter.elephantbird.pig.load.JsonLoader(); extracted = foreach twitter generate (chararray)$0#'created_at' as created_at,(chararray)$0#'id' as id,(chararray)$0#'id_str' as id_str,(chararray)$0#'text' as text,(chararray)$0#'source' as source; STORE extracted into 'tweets' using com.twitter.elephantbird.pig.store.LzoJsonStorage(); Error While trying to store in tweets:- java.lang.Exception: java.lang.RuntimeException: native-lzo library not available at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) Caused by: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.createCompressor(LzoCodec.java:165) at com.hadoop.compression.lzo.LzopCodec.createOutputStream(LzopCodec.java:50) at com.twitter.elephantbird.util.LzoUtils.getIndexedLzoOutputStream(LzoUtils.java:75) at com.twitter.elephantbird.mapreduce.output.LzoTextOutputFormat.getRecordWriter(LzoTextOutputFormat.java:24) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:81) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at By the error I guess lzo library is not available. Please suggest me how can I resolve this. Mohan.V ... View more