@Timothy Spann very effective answer, but similar information as @Randy Gelhausen and he was first in. ... View more

This article shows how to use a list of urls in an external file to iterate InvokeHttp https://community.hortonworks.com/content/kbentry/48816/nifi-to-ingest-and-transform-rss-feeds-to-hdfs-usi.html You can schedule GetFile to run once per day, week, etc. If errors at the end of the flow inserting into a db, you can configure to ignore failure. ... View more

Agree with @mqureshi @Constantin Stanca Would like to add the theme that compression is a strategy and usually not a universal yes or no, or this codec or that. Important questions to ask for your data are: Will it be processed frequently, rarely or never (cold storage)? How critical is performance when it is processed? Which leads to: Which file format/compression codec if any for each dataset? The following are good references for compression and file format strategies (takes some thinking and evaluating): http://www.slideshare.net/Hadoop_Summit/kamat-singh-june27425pmroom210cv2 http://comphadoop.weebly.com/ http://www.dummies.com/programming/big-data/hadoop/hadoop-for-dummies/ After formulating a strategy, think about dividing your hdfs filepaths into zones in accordance with your strategy. ... View more

@Saumitra Buragohain could you help out here? Could use your expertise 🙂 ... View more

I have heard that full-dev-platform is being deprecated and vagrant-multinode should be used for a development envt instead: https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/multinode-vagrant This is very resource intensive, so for dev the best option is: https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/quick-dev-platform Also, ansible should be installed latest version and downgraded as follows: https://cwiki.apache.org/confluence/display/METRON/Downgrade+Ansible ... View more

@Mohan V Very glad to see you solved it yourself by debugging -- it is the best way to learn and improve your skills 🙂 ... View more

There is a lot going on here -- when writing a complex script like this, the following approach is useful to build and debug: run locally against a small subset of records (pig -x local -f <scriptOnLocalFileSystem>.pig). This makes each instance of the script run faster. build each statement line by line until you get to the failure statement (run the first statement, add the second and run, etc until it fails). When it fails you need to focus on the last statement and fix it. These steps are good for finding grammar issues (which it looks like you have based on the error statement). If you also want to make sure your data is being processed correctly, put a DUMP statement after each line during each iteration. That way you can inspect the results of each statement If using inline statements like your grouped = statement, separate out at first until it works. This makes the issue easier to isolate. Let me know how that goes. ... View more