This is a case of corrupt pig.tar.gz in hdfs (/hdp/apps/<version>/pig) folder. I am not sure how it ended up with a corrupt version there on a fresh install based on ambari. But once I manually updated with pig.tar.gz from /usr/hdp/<version>/pig/, the error got resolved. However, confusing part is pig view throwing a completed unrelated error (File does not exist at /user/rmutyal/pig/jobs/test_23-05-2016-14-46-54/stdout) ... View more

Hadoop is distributed filesystem and distributed compute, so you can store and process any kind of data. I know that a lot of examples point of csv and DB imports since they are the most common use cases. I will give a list of ways of how the data that you listed can be used and processed in hadoop. You can see some blogs and public repos for examples. 1. csv Like you said you will see a lot of examples including in our sandbox tutorials. 2. doc You can put raw 'doc' documents into hdfs and use tika or tesseract to do OCR from these documents. 3. audio and video. You can put raw data again in hdfs. Processing depends on what you want to do with this data. You can extract metadata out of this data using yarn. 4. relational DB. You can take a look at sqoop examples on how you can ingest relations DB into HDFS and use hive/hcatalog to access this data. ... View more

If there are no HA components (Namenode HA and 2 hiveserver2 instances), then there is no depending with zookeeper. Check the hiveserver2.log from /var/log/hiveserver2/hiveserver2.log to see if you see any errors. If you have 2 hiveserver2 instances, they will register with zookeeper which may be when they are running into issues. ... View more

1. NN data lost. Is it that the disk with NN directories crashed or have you deleted them? Is this with HA or non-HA. With non-HA, if both data directories of NN have no data, then you will run into data loss issues. You can revive and get to some state from secondary NN data directories but can not guarantee no data loss. If there is no useful data, you can always issue a NN format and start fresh. (You will need to manually update tez and mapreduce apps, you can get the information from manual install documentation) 2. Ambari not starting. Clean up /var/log and start it back. 3. Most likely HDFS services are not up. Filled up disk can kill processes. Once ambari is up, see which one is running and which one is not. Againt if NN data is lost, then NN will not start up. ... View more