You could try to do a single INSERT INTO statement per partition and run as many of these simultaneously as your cluster has resources for. ... View more

Are you trying to replace the functionality of creating the host/date paths and files and/or are you trying to want NiFi to recursively parse the growing directories to get at the underlying syslog.log files? ... View more

Hey @Matt Clarke, if there is a better way to do this w/o RPG as you suggested in your answer over in https://community.hortonworks.com/questions/245373/nifi-cluster-listensmtp.html, would you have time to update this article to account for that? I point folks to this link all the time. Thanks! ... View more

Sounds like this should be running as an Isolated Processor and be configured to run on the Primary Node only instead of All Nodes. Then, to take full advantage of both of the NiFi nodes you have, you'll want to create a Remote Processor Group back on yourself much like explained in https://community.hortonworks.com/articles/97773/how-to-retrieve-files-from-a-sftp-server-using-nif.html. Good luck and happy Flowfiling! ... View more

I didn't see such a property when I looked at http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ListFTP/index.html, but some quick solutions to jumpstart this could be to simply replace this processor with a new one (which will have its own state management) or if you using "timestamps" for the "Listing Strategy" property then you could always do a linux "touch" command on the files on the FTP server which should trick the processors to grab them again. Good luck and happy Flowfiling! ... View more

I believe your biggest problem is that you are trying to use the HDP Sandbox for something of any decent size. That environment wasn't necessary built for you to runs 100's of GB datasets (which itself is surely not all that "big" of Big Data). The Sandbox also has a bunch of configuration settings focused on running a pseudo-cluster (all on one box) which is NOT idea for any job of any real size. You did go down the right path of changing the max amount of memory that YARN can use, but at the end of the day, your box only has two CPUs and you really can't run that many containers anyways. You'd probably need to go change the size of the TEZ containers, too, for Hive/TEZ to ask for more than whatever the tiny configuration from the Sandbox is granting you. I don't know the costing model, but I'm betting 4 boxes with 16GB each would be cheaper than this 64GB one you are using now and that would allow you to spread out the workload across multiple machines (and yes, you'd have to install HDP via Ambari, the the http://docs.hortonworks.com site can help a LOT). Good luck and happy Hadooping! ... View more