Are you trying to replace the functionality of creating the host/date paths and files and/or are you trying to want NiFi to recursively parse the growing directories to get at the underlying syslog.log files? ... View more

Hey @Matt Clarke, if there is a better way to do this w/o RPG as you suggested in your answer over in https://community.hortonworks.com/questions/245373/nifi-cluster-listensmtp.html, would you have time to update this article to account for that? I point folks to this link all the time. Thanks! ... View more

Missing the ambari server name before the ":8080"? ... View more

Sounds like this should be running as an Isolated Processor and be configured to run on the Primary Node only instead of All Nodes. Then, to take full advantage of both of the NiFi nodes you have, you'll want to create a Remote Processor Group back on yourself much like explained in https://community.hortonworks.com/articles/97773/how-to-retrieve-files-from-a-sftp-server-using-nif.html. Good luck and happy Flowfiling! ... View more

Sure... why not!?!? 🙂 I just installed HDP 3.1.0 via Ambari as barebones as I could on a small 5 nodes cluster (1 master and 4 workers). It did make me install Ambari Metrics, SmartSense and ZK, but I was able to delete those after everything was installed as shown in my screenshot. That said, I'd leave those in (and ZK will be required to do HDFS HA), but wanted to make the point that you could JUST have HDFS. Good luck and happy Hadooping! ... View more

The best answer to "will it work?" is "what did your testing show?". Give it a try and let us all know. I'm guessing you just have some old legacy code you can't change? I'm also guessing it will work! So... to be more precise, the 2.6.5 release notes component page, https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html, says Hadoop 2.7.3 is being used. The Apache Hadoop page for MRv1 compatibility with MRv2, https://hadoop.apache.org/docs/r2.7.3/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html, suggests you are probably in good shape for previously compiled MRv1 apps from running on MRv2, but doesn't quite guarantee if you need to compile them against MRv2 you'll be so lucky. Good luck and happy Hadooping! ... View more

I believe your biggest problem is that you are trying to use the HDP Sandbox for something of any decent size. That environment wasn't necessary built for you to runs 100's of GB datasets (which itself is surely not all that "big" of Big Data). The Sandbox also has a bunch of configuration settings focused on running a pseudo-cluster (all on one box) which is NOT idea for any job of any real size. You did go down the right path of changing the max amount of memory that YARN can use, but at the end of the day, your box only has two CPUs and you really can't run that many containers anyways. You'd probably need to go change the size of the TEZ containers, too, for Hive/TEZ to ask for more than whatever the tiny configuration from the Sandbox is granting you. I don't know the costing model, but I'm betting 4 boxes with 16GB each would be cheaper than this 64GB one you are using now and that would allow you to spread out the workload across multiple machines (and yes, you'd have to install HDP via Ambari, the the http://docs.hortonworks.com site can help a LOT). Good luck and happy Hadooping! ... View more

As the blueprints focus heavily on the host_groups concept which usually means some highly specialized master setups and then a more generic worker model, I feel that using blueprints beyond the initial cluster layout really works best for when you are adding more workers. My INITIAL recommendation (I'm arm-chair QBing this w/o all the details) would be to simple go to Ambari's UI and add a new host via the wizard process and then assign whatever master and worker processes you need. ... View more

could you provide some additional details? screenshots of the Ambari dashboard, HDFS service page, host list, etc.? ... View more

It is confusing what triggers this task to run. Do you have any additional info on that or know if there is any way to configure it more precisely? ... View more

I need to play with the S3 processors a bit to be more helpful, but wondering if there is any issue getting these files in a NiFi cluster and if you should be marking the pull processor to be an "isolated processor" to run only on the "primary node" as Brian describes in his answer to https://community.hortonworks.com/questions/99328/nifi-isolated-processors-in-a-clustered.html. Worth giving it a try first to see if that's part of the problem. ... View more

Yes, the collect() ACTION does require everything to come to the driver, but there is a better ACTION for you. Try using foreach() which is like an RDD's map() function in that it works on each partition of the underlying dataset independent of the other partitions (so you can run it wide!!). It returns nothing, which is probably what you want it to do anyways. Good luck and happy Sparking! ... View more

Pro: cheap. Con: not scalable. I know... you already knew that! 😉 Also, I'm not trying to be a smart @$$, but I do mean what I say at https://twitter.com/LesterMartinATL/status/504004795557236736 about these tiny clusters (yes, 3 worker nodes is a tiny cluster). A 3 node cluster might make sense for some problems, but you'll never be able to do anything at scale on it and it surely won't be performance-oriented for anything at the edge of what it can process. Regarding your 3TB capacity; does that mean each node has 3TB of capacity dedicated to itself (raw 9TB, but effective 3TB with replication factor of 3), or that each box has 1TB? I'm asking because for things like Terasort, we have to also consider the job's intermediate data (i.e. the info moving from the mappers to the reducers which is this case will be 500GB itself) as well as the final output back on HDFS; yes, another 500GB. The intermediary data isn't stored on HDFS, but if the input and output are then that 500GB is now 1TB (in + out) and will really be 3TB all by themselves if both of these are set with replication factor of 3. Even with replication factor of 1, this all smells problematic to me on this small cluster. If you only have 1TB of disk space on each node, then surely this will never run as just mentioned. Even if the space wasn't an issue, you'll need to run something like 3900 mappers just to process that (if my math is right of dividing 500GB by 128MB block size) plus a shed-load of reducers and that would take forever on three nodes. It has been many years since I was regularly running Terasort, but a very old heuristic I would use was a max of 30 minutes for a 10 worker node cluster comprised of boxes that had 128GB of ram and 10-12 disks. Clearly there are many variables at hand and I'm not sure looking at your specific output would immediately shed light on the exact problem, but what I would recommend is to start small and scale up. Run a 500MB gen and sort. Then double that to 1GB and then double to 2GB, 4GB, and so on to make sure your specific cluster's results make sense compared with the last run and I believe you'll see a good pattern. Eventually this will all run out of horsepower (aka nodes), but will give you a better benchmark. That is, that last good sized run will give you a number that should be cut in half when you double the size of the worker nodes! Good luck and happy Hadooping! ... View more

If the compressed file was of just one file, the Pig approach shown in https://stackoverflow.com/questions/34573279/how-to-unzip-gz-files-in-a-new-directory-in-hadoop might have been useful. No matter what you do, you'll have to do this in a single mapper from whatever data access framework you use this it won't be a parallelized job, but I understand your desire to save the time and network from the pull from HDFS and then put back in once extracted. The Java Map/Reduce example at http://cutler.io/2012/07/hadoop-processing-zip-files-in-mapreduce/ is also assuming the compressed file is a single file, but maybe it could be a start for some custom work you might be able to do. Good luck and happy Hadooping! ... View more

With Hive 3 pushing hard with fully managed tables with native file formats as transactional tables, see https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/managing-hive/content/hive_acid_operations.html for more info, this "direct from Spark to Hive" approach will get much harder do to the underlying "delta files" that get created when data is added/modified/removed from a Hive table. The Spark LLAP Connector will aid in this integration. That said, historically, the better answer is often to simple save your DF from Spark to HDFS, wrap it with an External Hive table and then do and INSERT INTO your existing Hive table with a SELECT * FROM your new external table. This lets Hive do all the heavy lifting and file conversions as needed and takes care of any partitioning and/or bucketing that you have in place. Good luck and happy Hadooping! ... View more

That pseudo-cluster itself is a scalability bottleneck. 😉 Storm likes to scale by having many Supervisor (worker) processes). As for the specific stats on your component, you could drill into the Storm UI and find your topology and then drill into it to find your bolt to see how it is doing and gauge for yourself it is working well enough. You'll get rolling statistics of how long it takes to process things such as shown below. You could also scale up the number of instances the bolt has, but again, the single-server pseudo-cluster is likely going to be your first bottleneck. ... View more

From the legacy Hortonworks Professional Services team, we would call this an Ambari Takeover. I'm not sure if there is a formal documented procedure available on the web, but our support and or consulting teams could help with this. Here's an article I found with a few seconds of googling about this concept in general; http://www.adaltas.com/en/2018/11/15/hadoop-cluster-takeover-with-apache-ambari/. Good luck & happy Hadooping! ... View more

I have never used it to do both at the same time and https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/using_sqoop_to_move_data_into_hive.html says "HDFS or Hive". Good luck and happy Hadooping! ... View more

Where the install instructions at https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/installing-upgrading-hdf.html not useful? Also, just a callout that NiFi itself is a master-less cluster config. I'm assuming that maybe you want to use the "master" for something like Ambari only? ... View more