Thanks that does show more information.   Though what i find weird is the same query has run with a large load earlier (with same config params) and now has failed (from the logs: java.lang.OutOfMemoryError: Java heap space).   Regards ... View more

Thanks for confirming that. We'll enable for Impala as well but after a week or so, but wanted to know if in the meantime it'd still work or not. ... View more

Thanks, to be on the same page taking help of below scenario:    hdfs snapshottable location /a/b/ has a file c which is snapshotted. Consider a scenario where c is deleted from hdfs using cli hdfs -rm -r -skipTrash (NN transaction happened and hdfs cli command doesn't show up the file anymore) and then a new file is created with same content/size and name.   - What gets stored in hdfs? whats the delta that snapshot add in hdfs in this case?    --> is it just that snapshot still holds c as block in hdfs in addition to the same file that was created in hdfs    --> NN resource used to maintain both of their metadata in heap?   is this all or there is more to it .   Regards   ... View more

Thanks Arpit,   To clarify what is the magnitude of the size we are talking about here when you mention - " directory will need to be tracked as deltas and that can result in both higher disk space and NameNode heap usage".   I'm assuming you mean just to store the metadata of the changed snapshot and which isn't significant given the actual size of data held(in reference to my example above) , if not please clarify   Regards ... View more