The PID-file problem is actually quite common. I have seen this a couple of times already, not just zookeeper-related (e.g. services not starting because of existing pid file with invalid process id, ...). ... View more

Thanks @gopal. In this case we should definitely use ORC+(new)Zlib. I'll edit my answer 🙂 ... View more

You could try to restart the Ambari Agents on the Zookeeper nodes or the complete ambari server. What is the current state of the Zookeeper service and host_components? could you check via Rest API <ambari-server>/api/v1/clusters/<clustername>/services/ZOOKEEPER ... View more

I'd definitely open a feature enhancement, so that we can get engineering's input on that as well. Please keep me in the loop. ... View more

Makes sense, but this is going to be difficult to implement. Do you mean the % of available capacity or % of general network capacity? I assume you mean the % of currently available capacity, which is changing depending on the jobs that are running. We would need a way to predict the volume of files that are going to be transferred,..... The result would be an ever changing bandwidth...Maybe it makes sense to specify a minimum and maximum bandwidth, yarn gets priority and can use the full capacity. I need to think about that a bit more... 🙂 ... View more

Good question! The balancer has a configurable limit, which ensures that the balancer does not utilize too much network bandwidth. You'll find the parameter dfs.datanode.balance.bandwidthPerSec in the hdfs-site.xml, the default value is 1048576 bytes per second. https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml "Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second." ... View more

Great write-up.Thanks! ... View more

Thanks for sharing! How many datasets were in the Links table? Is the dataset in Links a subset from the ABC dataset? ... View more

ORC+ZLib seems to have the better performance. ZLib is also the default compression option, however there are definitely valid cases for Snappy. I like the comment from David (2014, before ZLib Update) "SNAPPY for time based performance, ZLIB for resource performance (Drive Space)." Make sure you checkout David's post: https://streever.atlassian.net/wiki/display/HADOOP/Optimizing+ORC+Files+for+Query+Performance As @gopal pointed out in the comment, we have switched to a new ZLib algorithm, hence the combination ORC + (new) ZLib is the way to go. The performance difference of ZLib and Snappy regarding disk writes is rather small. Btw. ZLib is not always the better option, when it comes to HBase, Snappy is usually better 🙂 ... View more