Sqoop metastore on MySQL is not out but there are jiras open. There are tutorials outthere but i was never successful to make it work. Yes metastore is for metadata about jobs, storing last record procesaed, job name, etc. There really aren't any backup strategies for hsqldb. It's a file on disk, while some in memory data is not flushed to disk you have potential Dara loss. Maybe putting it on RAID 1? Use ECC memory chips, etc. ... View more

You can try the following, cat your large file, grab a few lines output to new file on local fs. Ill be curious to know if that works with avro serialization. http://stackoverflow.com/questions/22852063/how-to-copy-first-few-lines-of-a-large-file-in-hadoop-to-a-new-file#22852192 Then use avro-tools to extract schema. ... View more

Please use this guide to package your topology, do no use hive 2.0 http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_storm-user-guide/content/ch_storm-pkg-topologies.html read the rest of the guide as it has storm-hive examples ... View more