Option 1: reformat: you will need not only to "copyFromLocal" but also recreate the file system. See for example this for details. Option 2: Exit safe mode and find out where you are. I'd recommend this one. You can also find out what caused the trouble, maybe all corrupted blocks are on a bad disk or something like that. You can share the list of files you are uncertain whether to restore them or not. ... View more

Do you really need 3 masters on such a small cluster? How about 2 masters, putting 3rd Zookeeper on the Kafka node, or on one worker node, and having 5 worker nodes? I always prefer more computing power than "book-keeping" of master nodes. You can also put Knox and Flume on the Edge node, if you use them, of course. Then, distribute other master services on 2 masters, Ambari on one, AMS collector on the other; NN on one RM on the other (or on both if you want HA). And, yes, you can move the majority of master services later, using Ambari. ... View more

No, I mean, after hdfs is back, you detect corrupted files in hdfs, and restore important ones, like those in /hdp and /user/oozie/share/lib. You can also reformat and start afresh, but wait to see what files and how many of them are damaged. Regarding your worries about machines rebooting, theoretically yes, a thunderbolt can strike from a blue skies, but usually it doesn't 🙂 and when machines reboot they do so one at a time. That's why you do NN HA, backup NN metadata, avoid chipo servers, use UPS, etc. ... View more

Glad to hear it works! By the way, what type is "shape"? According to docs: ST_AsGeoJson(geometry) return GeoJson representation of geometry. ... View more