There are a few cons to raising your block size: - Increased cost of recovery during write failures When a client is writing a new block into the DataNode pipeline and one of the DataNode fails, there is a enabled-by-default recovery feature that will attempt to refill the gap in the replicated pipeline by transferring the partially written block from one of the remaining good DataNodes to a new DataNode. When this happens, the client is blocked (the outstream,write(…) caller is blocked in the API code). With increased block size, the time waited will also increase greatly depending on how much of the partial block data was written before the failure occurred. A worst-case wait example would involve the time required for network-copying 1.99 GiB for a 2 GiB block size because an involved DN may have failed at that specific point. - Cost of replication caused by DataNode loss or decommission When a DataNode is lost or is being decommissioned, the system has to react by re-filling the gaps in replica counts it creates. With smaller block sizes this activity is easy to spread randomly across the cluster, as several different nodes overall can take part in the re-replicate process. With larger blocks, only a few DNs can participate, and another consequence could be more lopsided space usage across DNs. That said, use of 1-2 GiB is not unheard of and I've seen a few large clusters apply that as their default block size. Its just worth being aware of the cons, looking out for such impact and tuning accordingly as you go. HDFS certainly functions at its best for large sized files, and your usage seems in accordance with that. ... View more

Before anything else, I'd strongly recommend upgrading all your CDH packages to the same latest version. Your HBase service may be running on 4.7, while some client hosts are on 4.0. To get a definitive view on what version the service uses, please visit the HBase Master Web UI to see what the version value on its homepage shows. That's the version you can assume the cluster uses. Here's what I'd do, perhaps go over to see if it'd be safe for you to follow in your environment: - Check the # of rows on some critical, large table with a RowCounter job. Keep this info for data check after the operation below. - Login to a host with the most recent HBase version, run 'hbase shell' and then 'list_snapshots'. If any snapshots show up, and you do not need them, delete them away with 'delete_snapshot' commands. Once done, wait a few minutes and see if the used space begins to reduce due to HFiles from snapshots being cleaned away. If it does, no further actions are needed, and the rest of the points no longer apply to you. - If there are no snapshots, or there's no such command then stop HBase, MOVE (NOT DELETE, not yet) the .archive directory to /tmp. - Restart HBase, and if it comes up, run a RowCounter again on the same table to check if the counts are still the same/very close to the prior counting done above. - If HBase comes up and the counts on your critical tables are the same as before, then proceed with deleting the .archive directory you've moved. - If HBase does not come up, or the counts vary greatly, then place back the .archive directory in its previous path. This directory cannot be deleted as it is in use by HBase if this is the case, and you'll need to think of an alternative strategy of increasing space (deleting rows in HBase, dropping tables, expanding cluster, etc.) Does this help? ... View more

> I am little bit confused, so the WebHDFS REST API is listening on the same port as the NameNode's UI? Yes this is correct. The HTTP(S) serving port of the NameNode does multiple things: Serves the UI for browsers on / and a few other paths, serves a REST API on /webhdfs/*, etc. WebHDFS on HDFS service is used by contacting the currently configured web port of the NameNode and DataNode(s) (the latter by following redirects, not directly). In your case, the cluster is set to use HTTPS (TLS security) so you need to use the 50470 port, swebhdfs:// (notice the s-prefix for security) in place of webhdfs:// and https:// in place of http:// when following any WebHDFS tutorial. ... View more

This is a shell behaviour, not Sqoop. "When referencing a variable, it is generally advisable to enclose its name in double quotes." "Use double quotes to prevent word splitting. An argument enclosed in double quotes presents itself as a single word, even if it contains whitespace separators." - http://tldp.org/LDP/abs/html/quotingvar.html ... View more

The .archive directory support was certainly not present in our original CDH 4.0.x sources: https://github.com/cloudera/hbase/tree/cdh4.0.0-release I'm therefore unsure who/what created it in your environment. ... View more

If you are using Cloudera Manager and have re-added the host/role after deleting it, Cloudera Manager should've marked your HBase service with a 'Stale Configuration' icon, indicating a restart is required for your HBase service (and client configuration deployment for its gateways) to see the changes in the ZK client configuration. There's no way to live-refresh the configuration at runtime, but you can consider performing a rolling restart to eliminate availability issues. ... View more