Are you sure that dump behaves the same? If I do ( using your data 😞 a = load '/tmp/test' using PigStorage(',') as (year,month,day); dump a; (2015,,08)(2015,,09)... And if I do b = foreach a generate month;and dump b; ()()() Looks to me pigstorage works perfectly fine with dump. If I use illustrate everything goes wrong though. After using illustrate even the dump command fails with a nullpointer exception. So not only does it not work correctly it breaks the grunt shell until I restart it. I think the problem is the illustrate command: Which is not too surprising since this is the warning on top of it in the pig docs: Illustrate: (Note! This feature is NOT maintained at the moment. We are looking for someone to adopt it.) ... View more

I assume you are using HDP? In that case PIG_HOME is set when executing the pig command. If you cat /usr/bin/pig you can find the line export PIG_HOME=${PIG_HOME:-/usr/hdp/2.3.2.0-2950/pig}. So you could run this manually. ... View more

If you have 4 nodes he will not be able to replicate 8 copies.It looks like some tmp files from accumulo. While I do not know accumulo too well some small files like jars normally have a high replication level so they are locally available on most nodes. You can check the filesystem with: hadoop fsck / -files -blocks -locations Normally programs honor a parameter called max replication which in your case should be 4 but it seems like accumulo doesnt always do that. https://issues.apache.org/jira/browse/ACCUMULO-683 Is this causing any problems? Or are you just worried about the errors in the logs. ... View more

And you should be able to remove it in the hive.exec.pre/post/failure.hooks parameter in Ambari/Hive/Config/AdvancedConfig as a workaround if this is really resulting in the error. Perhaps a bug? ... View more

I would think this is not enough information to give you an answer. Tez will normally be faster even on big amounts of data but your setup is pretty unusual. Huge amounts of memory ( Hive is normally CPU or IO bound so more nodes are in general a good idea ). And very big tasks which could lead to some Garbage Collection issues ( tez would reuse tasks which would not happen in mapreduce ) . Also you don't change the sort memory for example so that could still be small. The biggest question I have is cluster utilization and CPU utilization when you run both jobs. I.e. is the cluster fully utilized (run top on a node ) when you run the mapred job but not in tez?. Or is he waiting in a specific node? Tez dynamically adjusts the number of reducers running so it is possible that it decided on less tasks. Running set hive.tez.exec.print.summary=true; can help you figure out which part of your query took longest. The second question would be query complexity. Tez allows the reuse of tasks during execution so complex queries work better. It is always possible that there is a query that will run better on MapReduce. ... View more