1) You don't have to 2) Its relatively benign since HBase will have most data available on the local datanode ( because of HDFS local first write policy ) The biggest problem is that it makes your cluster config more complex. You will need two configs for hbase and non hbase nodes and potential performance implications when you run heavy workloads in the same cluster as hbase. 3) You don't HAVE to install Query server at all. Its optional normally the connection goes directly through the HBAse API and a lot of computation is client side. This is normally the better way to do things, the query server is like a proxy covering that client side functionality. In general I would prefer the normal client but there can be access issues for example ( normal api needs access to all datanodes ) 4) The PQS as said is like a proxy, normally the client side computations are quite light but that can be different if you return large amounts of data from distributed region servers. So you would need to keep an eye out for the CPU performance in the PQS. In addition to that you might use some CPU resources and compete with the other services on the node. 5) Same as 4 with the exception of the additional CPU competition on those nodes. Most of the time PQS is used because of firewall issues. If you cannot have access to all nodes it is convenient to put PQS on a couple edge nodes so you do not have to open up the full cluster to client connections. ... View more

"Let me put the question this way. If I have hive.execution.engine=tez; why do I need the property hive.server2.tez.initialize.default.sessions to set it to "True"? Whats the use-case for this property? I ran multiple tests but my hive.execution.engine property drives how the query works and not this default sessions property" The default session parameter has nothing to do with the way the query is executed. It is for pre-creating Tez sessions. If this is false the first query on an empty system will take at least 20seconds to create a session. Time for a Tez query: Hiveserver prepare, compilation, ...: ~1sec Not much you can do here however it continuously gets faster. Initialize Tez Application Master ( Session 😞 ~10 seconds To reduce that Hive can reuse Sessions, that are idle, AMs are kept for normally 120s after a query is run. Or you can instantiate default sessions if you cannot live with that delay. Initialize Containers : 3-10s The next step is to allocate the work containers to the Session, again Tez can reuse containers or you can preheat containers. ( pre allocate the containers ) The actual query That depends on your data. ... View more

The Plugin connects to Ranger to get the updated policy file and writes it to a local file in case it has to restart. It doesn't monitor the local files for changes to reload them. Because why would it. I assume if you stop ranger ( to make sure it cannot be contacted for an update ) then restart HDFS he would take your changes. ( Didn't try it but sounds like a reasonable assumption ). Now to exploit this you would need to force an hdfs restart and block access to ranger I suppose. Now how to stop people from tampering it? Just make sure random people cannot become root or users from the hadoop group on your system. the policy cache files can only be written by hive, ranger users etc. Once somebody is root on any node on the cluster you will have a hard time stopping anybody from doing things esp. if they can log in to the master servers. On the client you still might have a chance. ... View more

I see different issues here: a) Split up a line with multiple records If you have multiple communications by line you will need some pre processing. Hive provides maps and arrays but it is hard to use them in normal SQL. So tons of different ways but my suggestion would be to write a Pig UDF to split up a line into multiples potentially adding a column that adds the line information if you need to group them together somehow. http://stackoverflow.com/questions/11287362/splitting-a-tuple-into-multiple-tuples-in-pig b) Get date from Filename There are some ways to get at the filename in mapreduce but its difficult. MapReduce by definition abstracts filenames away. You have two options there: 1) Use a little python/java/shell whatever preprocessing script OUTSIDE hadoop that adds a field with the date to each row of each file taken from the filename. Easy but not that scalable 2) Write your own recordreader 3) Pig seems to provide some value called tagsource that can do the same http://stackoverflow.com/questions/9751480/how-can-i-incorporate-the-current-input-filename-into-my-pig-latin-script c) Do Graph analysis You can use Hive/pig/Spark for preprocessing and Spark provides a cool graph api. Tons of examples out there. http://spark.apache.org/graphx/ Good luck. ... View more

Bulk upload? In that case an edge node with hadoop client and run hadoop fs -put commands. You can expect about 300GB/h for each hadoop put into HDFS. However you can parallelize the commands. I.e. if you have multiple files you can run multiple puts in parallel ( Essentially till you saturate the internal network of the cluster or the reading of the files from the local/network storage. ) A little bash or python script will normally do the trick. Nifi will work too obviously and might provide some retry/error handling that you would otherwise have to code yourself so it depends a bit on your requirements. ... View more