This is a case of corrupt pig.tar.gz in hdfs (/hdp/apps/<version>/pig) folder. I am not sure how it ended up with a corrupt version there on a fresh install based on ambari. But once I manually updated with pig.tar.gz from /usr/hdp/<version>/pig/, the error got resolved. However, confusing part is pig view throwing a completed unrelated error (File does not exist at /user/rmutyal/pig/jobs/test_23-05-2016-14-46-54/stdout) ... View more

You don't need to restart HDP 2.4 cluster. But it is recommended to decommission the node with dead disk, change the disk and add it back to the cluster. This will ensure that data is evenly distributed on all the data disks on this node. 1. To decommission, go to ambari -> Host -> Datanode. That has an option to decommission. 2. Go to nodemanager to decommission it as well. 3. Once it goes to decommissioned state, stop Datanode and nodemanager on that host and replace the disk. 4. Start datanode and nodemanager back. 5. You will see an option to recommission at the same place. You can click on it to take it out of decommissioned state. No other services across the cluster need to be stopped and if you have more than 3 datanodes and your default rep. factor is 3, all services will continue. ... View more

Hadoop is distributed filesystem and distributed compute, so you can store and process any kind of data. I know that a lot of examples point of csv and DB imports since they are the most common use cases. I will give a list of ways of how the data that you listed can be used and processed in hadoop. You can see some blogs and public repos for examples. 1. csv Like you said you will see a lot of examples including in our sandbox tutorials. 2. doc You can put raw 'doc' documents into hdfs and use tika or tesseract to do OCR from these documents. 3. audio and video. You can put raw data again in hdfs. Processing depends on what you want to do with this data. You can extract metadata out of this data using yarn. 4. relational DB. You can take a look at sqoop examples on how you can ingest relations DB into HDFS and use hive/hcatalog to access this data. ... View more

If you are looking for opensource volume level encryption tools, we have seen LUKS being used. There will be some overhead from LUKS. You can take a look at LUKS at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Encryption.html ... View more

You can increase memory on your mappers. Take a look at mapreduce.map.memory.mb, mapreduce.reduce.memory.mb, mapreduce.map.java.opts and mapreduce.reduce.java.opts. I think your mapreduce.map.memory.mb is set to 256MB based on the error. I don't know what else is running on your 3GB node and what heap is given, but you maybe able to allocate 1GB of it to yarn (container memory). It is also possible to get it to run on 15GB node by using node labels. You can also switch off nodemanager on 3GB node if other processes are running no this, so it uses 15GB node. ... View more