There are a few cons to raising your block size: - Increased cost of recovery during write failures When a client is writing a new block into the DataNode pipeline and one of the DataNode fails, there is a enabled-by-default recovery feature that will attempt to refill the gap in the replicated pipeline by transferring the partially written block from one of the remaining good DataNodes to a new DataNode. When this happens, the client is blocked (the outstream,write(…) caller is blocked in the API code). With increased block size, the time waited will also increase greatly depending on how much of the partial block data was written before the failure occurred. A worst-case wait example would involve the time required for network-copying 1.99 GiB for a 2 GiB block size because an involved DN may have failed at that specific point. - Cost of replication caused by DataNode loss or decommission When a DataNode is lost or is being decommissioned, the system has to react by re-filling the gaps in replica counts it creates. With smaller block sizes this activity is easy to spread randomly across the cluster, as several different nodes overall can take part in the re-replicate process. With larger blocks, only a few DNs can participate, and another consequence could be more lopsided space usage across DNs. That said, use of 1-2 GiB is not unheard of and I've seen a few large clusters apply that as their default block size. Its just worth being aware of the cons, looking out for such impact and tuning accordingly as you go. HDFS certainly functions at its best for large sized files, and your usage seems in accordance with that. ... View more

@bgooley Got it! I'm relatively new to Python development and was trying figure out what was an acceptable form of multiple criteria for a single POST request for the same key.  <== Maybe that sentence will help someone find this later on. I used tcpdump as well but could not see the POST data as it was encrypted with SSL.  I was not able to turn off SSL to see plain text content as I don't have access to the server.  I was exploring using a cert to decrypt but have never done that using tcpdump.  I would love to know how, as I have run into trying to debug traffic and could not see the content. Anyway, I was having trouble creating a dict for the form data as it would only accept a single key.  I didn't realize that an array(list) with a single dict key would work as well. This line was a problem in that Python would not let me add multiple key entries like "user_ids='4', user_ids='5'". I didn't realize all it needed was a list.   form_data = dict(csrfmiddlewaretoken=session.cookies['csrftoken'] ,next='/hue/useradmin/users/', user_ids=['4','5'])  I tried every combination I could think of but made the mistake of surrounding the arguments in quotes, making them a single text argument versus a list. What you did for me was verify a few things and put me on the right trail.   Thanks to everyone for their contribution! ... View more