You just need to align your LOCATION clause of EXTERNALS TABLE's DDL to point to your /FLIGHT folder. Hive will crawl all the subfolders. You might also consider using PARTITION BY and instead of having folders for year, month a day. This let's you do things like WHERE my_partition_col > '19991115' AND my_partition_col < '20010215' which would be much tougher if you partition by specific year, month, and day values. ... View more

While I'm doubtful these three directories are the very best answer to this problem, but the old "three directories for the NN metadata" came about long before a solid HA solution was available and as https://twitter.com/LesterMartinATL/status/527340416002453504 points out, it was (and actually still is) all about disaster recovery. The old adage was to configure the NN to write to three different disks (via the directories) -- two local and one off the box such as a remote mount point. Why? Well... as you know that darn metadata is the keys to the whole file system and if it ever gets lost then ALL of your data is non-recoverable!! I personally think this is still valuable even with HA as the JournalNodes are focused on the edits files and do a great job of having that information on multiple machines, but the checkpoint image files only exist on the two NN nodes in HA configuration and, well... I just like to sleep better at night. Good luck and happy Hadooping! ... View more

Welcome to Phoenix... where the cardinal rule is if you are going to use Phoenix, then for that table, don't look at it or use it directly from the HBase API. What you are seeing is pretty normal. I don't see your DDL, but I'll give you an example to compare against. Check out the DDL at https://github.com/apache/phoenix/blob/master/examples/WEB_STAT.sql and focus on the CORE column which is a BIGINT and the ACTIVE_VISITOR column which is INTEGER. Here's the data that gets loaded into it; https://github.com/apache/phoenix/blob/master/examples/WEB_STAT.csv. Here's what it looks like via Phoenix... Here's what it looks like through HBase shell (using the API)... Notice the CORE and ACTIVE_VISITOR values looking a lot like your example? Yep, welcome to Phoenix. Remember, use Phoenix only for Phoenix tables and you'll be all right. 🙂 Good luck and happy Hadooping/HBasing! ... View more

There are a TON of variables at play here. First up, the "big" dataset isn't really all that big for Hive or Spark and that will always play into the variables. My *hunch* (just a hunch) is that your Hive query from beeline is able to use an existing session and that it is able to get access to as many containers as it would like. Conversely, Zeppelin may have a SparkContext that has a smaller number of executors than your Hive query can get access to. Of course, the "flaw in my slaw" is that these datasets are relatively small anyways. Spark's "100x improvement" line is always related to reiterative (aka ML/AI) processing, but for traditional querying and data pipelining, Spark runs faster when there is a bunch of tasks (mappers and reducers) that need to run and it can transition between those milliseconds within the pre-allocated executor containers instead of seconds that Hive has to burn talking to YARN's RM to get the needed containers. I realize this isn't as much of an answer as you were looking for more than it was an opinion piece now that a I review it before hitting "post answer". 🙂 Either way, good luck and happy Hadooping/Sparking! ... View more

Looks like @Bryan Bende already answered this over on https://stackoverflow.com/questions/54791414/how-i-can-use-hbase-2-0-4-version-in-nifi ... View more

If their times are in sync now, I'm not sure of any inherent problems that would prevent you from starting them back up. Good luck & happy Hadooping! ... View more